Send logs to Apache Kafka
Create a stream that sends the logs of your applications as messages to an Apache Kafka topic, in Azion Console or with the Azion API.
You can send the logs of a stream to a topic of an Apache Kafka cluster from Azion Console or with the Azion API. To send the WAF events of a firewall to a SIEM, refer to Stream WAF events to a SIEM.
Data Stream sends the log lines as messages to one topic of the cluster. In the stream form, the endpoint is set in the field labeled Connector, and Apache Kafka is one of its options. For every field and its bounds, refer to Endpoints.
The example collects the requests of one workload with the Applications data source.
Prerequisites
Console
- An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
- A workload on the account that receives requests.
- An Apache Kafka cluster and the host and port of the servers for the initial connection. You need only these servers, not every server of the cluster.
- The name of the topic that receives the messages. The stream sends to one topic.
- To encrypt the data with Transport Layer Security (TLS), a digital certificate on the receiving servers, issued by a trusted certificate authority (CA) such as IdenTrust, DigiCert, Sectigo, GoDaddy, GlobalSign, or Let’s Encrypt.
The Apache Kafka endpoint has no credential field: the stream connects with the server addresses, the topic, and the TLS setting.
- Access to Azion Console. To sign in, refer to How to access Azion Console.
Create the stream
The stream collects from the workload you choose, through a workload filter. Unlike sampling, a workload filter leaves your other streams active.
Console
To create the stream in Azion Console:
Access Azion Console > Data Stream.
In the General section, enter a Name. For example: logs-to-kafka.
In the Input section, select Applications in Data Source.
In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.
In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow. If your Console names domains where this guide names workloads, the same control reads Filter Domains. The pick list then reads Available Domain and Chosen Domain.
In the Render Template section, select Applications Event Collector in Template.
In the Output section, select Apache Kafka in Connector.
In Bootstrap Servers, enter each server as host:port, separated by a comma and no space. For example: kafka1.example.com:9092,kafka2.example.com:9092.
In Kafka Topic, enter the name of one topic. For example: azion.logs.
Turn on Enable Transport Layer Security (TLS) to send the data encrypted. The receiving servers need a certificate from a trusted CA.
In the Status section, keep Active turned on.
The Console shows Your data stream has been created. The stream appears in the Data Stream list with Kafka in the Connector column and the Active status.
The API and the Console save the stream without contacting the cluster. A wrong server address or topic surfaces only when the stream sends. An activation takes effect after one to two minutes.
Confirm the delivery
Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute. A stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.
Console
To find the sends in Azion Console:
Access Azion Console > Real-Time Events.
Each row is one send. Find the rows with KAFKA in Endpoint Type, and read their Status Code.
A Status Code of 200 means the cluster accepted the batch. Streamed Lines gives the number of log lines in the batch.
A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.