Send logs to AWS Kinesis Data Firehose
Create a stream that sends the logs of your applications to an Amazon Kinesis Data Firehose delivery stream, in Azion Console or with the Azion API.
You can send the logs of a stream to an Amazon Kinesis Data Firehose delivery stream from Azion Console or with the Azion API. To write the logs straight to an Amazon S3 bucket, with no delivery stream in between, refer to Send logs to Amazon S3.
Data Stream sends the log lines to the delivery stream in batches. In the stream form, the endpoint is set in the field labeled Connector, and Kinesis is its AWS Kinesis Data Firehose option. For every field and its bounds, refer to Endpoints.
The example collects the requests of one workload with the Applications data source.
Prerequisites
Console
- An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
- A workload on the account that receives requests.
- An AWS account with an Amazon Kinesis Data Firehose delivery stream. The delivery stream must use Direct PUT as its source, and it needs a destination, such as an Amazon S3 bucket.
- The name of the delivery stream, up to 64 characters.
- The region code where the delivery stream runs, such as
us-east-1. - An AWS access key: an access key ID and its secret access key, created in AWS Identity and Access Management (IAM).
- Access to Azion Console. To sign in, refer to How to access Azion Console.
Create the stream
A workload filter limits the stream to the workload you choose. Sampling would deactivate your other streams, and a workload filter keeps them active.
Console
To create the stream in Azion Console:
Access Azion Console > Data Stream.
In the General section, enter a Name. For example: logs-to-kinesis.
In the Input section, select Applications in Data Source.
In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.
In the Transform section, set Option to Filter Workloads. Select your workload in Available Workload, then move it to Chosen Workload with the arrow. If your Console names domains where this guide names workloads, the same control reads Filter Domains. The pick list then reads Available Domain and Chosen Domain.
In the Render Template section, select Applications Event Collector in Template.
In the Output section, select AWS Kinesis Data Firehose in Connector.
Enter the name of the delivery stream in Stream Name. Enter its region code in Region, such as us-east-1.
Enter the access key ID in Access Key and the secret access key in Secret Key. The Console masks both fields.
In the Status section, keep Active turned on.
The Console shows Your data stream has been created. The Data Stream list shows the stream with AWS Kinesis Firehose in the Connector column and the Active status.
The API and the Console save the stream without contacting AWS. A wrong stream name, region, or access key surfaces only when the stream sends. An activation takes effect after one to two minutes.
Confirm the delivery
Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute. A stream sends a batch to Kinesis every 60 seconds, or sooner when it reaches 500 log lines.
Console
To find the sends in Azion Console:
Access Azion Console > Real-Time Events.
Each row is one send. Find the rows with AWS_KINESIS_FIREHOSE in Endpoint Type, and read their Status Code.
A Status Code of 200 means the delivery stream accepted the batch. Streamed Lines gives the number of log lines in the batch.
A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.