Send logs to an HTTP endpoint

You can send the logs of a stream to any URL that receives HTTP or HTTPS POST requests from Azion Console or with the Azion API. To change the body of each request, such as to send a JSON array, refer to Customize the HTTP POST payload.

Data Stream sends the log lines of each batch in the body of POST requests to your URL. In the stream form, the endpoint is set in the field labeled Connector, and the HTTP endpoint is its Standard HTTP/HTTPS POST option. Use it for a platform that has no option of its own in that list. For every field and its bounds, refer to Endpoints.

The example collects the requests of one workload with the Applications data source.

Prerequisites

Console

An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
A workload on the account that receives requests.
The URL that receives the logs. It must start with http:// or https:// and accept POST requests.
The headers your platform requires to accept the requests, such as an access key. Data Stream sends them with every request.
An endpoint that receives each batch within 20 seconds. After 20 seconds, the send ends with status 504. For the timeout, refer to Data Stream limits.
Access to Azion Console. To sign in, refer to How to access Azion Console.

Create the stream

The stream collects from the workload you choose, through a workload filter. Unlike sampling, a workload filter leaves your other streams active.

Console

To create the stream in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Select + Stream

Name the stream

In the General section, enter a Name. For example: logs-to-http.

Select the data source

In the Input section, select Applications in Data Source.

Turn off Sampling

In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.

Choose the workload

In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow. If your Console names domains where this guide names workloads, the same control reads Filter Domains. The pick list then reads Available Domain and Chosen Domain.

Select the template

In the Render Template section, select Applications Event Collector in Template.

Select the HTTP endpoint

In the Output section, select Standard HTTP/HTTPS POST in Connector. It is the default option.

Enter the URL

In URL, enter the URL that receives the logs. For example: https://logs.example.com/ingest.

Keep the payload settings

Keep $dataset in Payload Format, \n in Payload Log Line Separator, and 1000000 in Payload Max Size. With these values, the endpoint receives one log line per line, in NDJSON format.

Enter the headers

In Custom Headers, enter one header in the Header field, written header-name:value. For example: Authorization:Bearer [API KEY]. The Console requires at least one header. To add another one, select Header. The Console takes up to five headers.

Keep the stream active

In the Status section, keep Active turned on.

Select Save

The Console shows Your data stream has been created. The stream appears in the Data Stream list with HTTP in the Connector column and the Active status.

The API and the Console save the stream without contacting the URL. A wrong URL or header surfaces only when the stream sends. An activation takes effect after one to two minutes.

Confirm the delivery

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.

Console

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source

Read the latest sends

Each row is one send. Find the rows with HTTP_POST in Endpoint Type, and read their Status Code.

A Status Code of 200 means your endpoint accepted the batch. Streamed Lines gives the number of log lines in the batch.

Any status other than 200 is the answer of your endpoint, with two exceptions. 503 means Data Stream found the endpoint unavailable, and 504 means the endpoint did not receive the batch within 20 seconds. A URL that accepts no POST request answers each send with its own error, such as 405. For the causes, refer to Troubleshoot Data Stream.

Next steps

Customize the HTTP POST payloadSend the log lines as a JSON array, or change the separator and the request size.EndpointsEvery field of the HTTP endpoint, with its type, bounds, and API name.Edit, stop, or delete a streamChange the URL or the headers, pause the stream, or remove it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.