Customize the HTTP POST payload
Set the payload format, log line separator, maximum size, and custom headers of a Standard HTTP/HTTPS POST endpoint, in Azion Console or with the Azion API.
You can customize the payload that a Standard HTTP/HTTPS POST endpoint receives from a stream, from Azion Console or with the Azion API. To create a stream that sends to an HTTP endpoint, with its data source and template, refer to Send logs to an HTTP endpoint.
The payload is the body of each POST request that Data Stream sends. Four fields of the endpoint shape it: Payload Format, Payload Log Line Separator, Payload Max Size, and Custom Headers. No other endpoint type has them. In the stream form, the endpoint is set in the field labeled Connector. The template decides which variables each log line carries. For every payload field and its bounds, refer to Templates and payload.
Prerequisites
Console
- An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
- A workload on the account that receives requests.
- An HTTP or HTTPS URL that accepts
POSTrequests, and the headers it requires, such as an authentication token. - Access to Azion Console. To sign in, refer to How to access Azion Console.
- An account with fewer than 3,000 workloads. At 3,000 workloads or more, the Console blocks the stream forms, and you manage streams through the API only.
Set the payload fields
The fields are in the Output section of the stream form, on a stream you create or one you edit. With the defaults, the endpoint receives NDJSON: one log line per line, with no brackets and no comma between lines.
Console
To set the payload fields in Azion Console:
Access Azion Console > Data Stream.
Select + Stream to create a stream, or select the row of an existing stream to edit it.
In the Output section, set Connector to Standard HTTP/HTTPS POST. In URL, enter the address of your endpoint, such as https://logs.example.com/ingest.
In Payload Format, keep $dataset, or enter up to 250 characters around it. Data Stream replaces $dataset with the log lines of the batch, joined by the separator.
In Payload Log Line Separator, keep \n to put each log line on its own line. You can enter up to 100 characters instead.
In Payload Max Size, enter the maximum size of a data packet in bytes, from 1000000 to 2147483647. The default is 1000000.
In Custom Headers, enter one header in each Header row, as header-name:value. For a token, enter Authorization:Bearer <your-token>.
To add a row, select Header. The Console holds up to five headers and requires at least one.
If the stream uses sampling, the Attention dialog opens. Saving then deactivates every other stream on the account. Select Confirm.
The Console shows Your data stream has been created for a stream you create, or Your data stream has been updated for an edit.
A change to the payload of an existing stream takes a few minutes to propagate. Saving checks the format of the fields, not the endpoint, so a wrong URL or header shows up later as failed sends.
Send the log lines as one JSON array
A receiver that reads each request body as a single JSON document needs the log lines in a JSON array, with brackets around them and a comma between them. Two fields change: Payload Format wraps $dataset in brackets, and the separator becomes a comma. For the NDJSON and JSON array bodies side by side, refer to Templates and payload.
Console
To send a JSON array in Azion Console:
Access Azion Console > Data Stream.
In the list, select the row of the stream that sends to the HTTP endpoint.
In the Output section, set Payload Format to [$dataset].
Set Payload Log Line Separator to ,.
If the Attention dialog opens, select Confirm.
The Console shows Your data stream has been updated. Each request body is then one JSON array of log lines.
Confirm the delivery
Real-Time Events records every send of a stream, accepted or not, with the status code your endpoint returned. Send a few requests to the workload, then wait about a minute. A batch leaves after 60 seconds, or sooner at 2,000 log lines or at the Payload Max Size.
Console
To find the sends in Azion Console:
Access Azion Console > Real-Time Events.
Each row is one send. Find the rows with HTTP_POST in Endpoint Type, and read their Status Code.
A Status Code of 200 means your endpoint accepted the batch. Streamed Lines gives the number of log lines in the batch.
A status other than 200, such as 405, is the answer of your endpoint. A 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.