Send Data Stream data to Object Storage
Point a Data Stream stream at an Object Storage bucket through the S3 endpoint, in Azion Console or with the Azion API, and confirm the delivery.
You can send the logs of a stream to an Object Storage bucket from Azion Console or with the Azion API. To create a stream that writes to a bucket from the start, refer to the Data Stream quickstart. To send the logs to an Amazon S3 bucket, refer to Send logs to Amazon S3.
Data Stream reaches the bucket through the S3 endpoint and writes each batch of log lines as one object. In the stream form, the endpoint is set in the field labeled Connector, and its option is Simple Storage Service (S3). For every field and its bounds, refer to Endpoints.
Prerequisites
Console
- The Edit Data Stream permission. For the permissions, refer to Stream settings.
- A stream on the account. To create one, refer to the Data Stream quickstart.
- An Object Storage bucket. To create one, refer to the Object Storage quickstart.
- An Object Storage credential that reaches the bucket, with four capabilities: List Files, Write Files, List All Bucket Names, and List Buckets. In the API, these are
listFiles,writeFiles,listAllBucketNames, andlistBuckets. Without the two bucket-list capabilities, every send fails with status503, and no object reaches the bucket. The secret key shows only once, when the credential is created. To create the credential, refer to S3 protocol compatibility credentials. - Access to Azion Console. To sign in, refer to How to access Azion Console.
Point the stream at the bucket
The stream keeps its data source, its filters, and its template. Only the endpoint changes, with the bucket’s S3 address, its region, and the key pair of the credential.
Console
To point the stream at the bucket in Azion Console:
Access Azion Console > Data Stream.
In the list, select the row of the stream.
In the Output section, select Simple Storage Service (S3) in Connector.
Enter the values of your bucket:
- URL:
https://s3.us-east-005.azionstorage.net - Bucket Name: the name of your bucket
- Region:
us-east-005
Enter the access key in Access Key and the secret key in Secret Key. The Console masks both fields. On an account with View Data Stream only, a lock icon replaces the reveal icon.
On an edit, the Console does not require Secret Key.
In Object Key Prefix, enter the start of each object name. For example: activity. Without a prefix, the objects go to the root of the bucket.
In Content Type, select plain/text or application/gzip. With plain/text, each object holds one log line per line.
In the Status section, keep Active turned on.
If the Attention dialog opens, select Confirm.
The Console shows Your data stream has been updated. In the Data Stream list, the Connector column shows Amazon S3 for the stream.
Saving checks the format of each field and does not contact the bucket. A wrong key or a missing capability shows only as a failed send. An activation takes effect after one to two minutes. If the stream uses sampling, saving it as active deactivates every other stream on the account; a workload filter leaves them active.
Confirm the delivery
A stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines. Two checks confirm the delivery: the object in the bucket, and the send that Real-Time Events records.
Find the object in the bucket
Each object name is the Object Key Prefix, a /, the date and time of the send in the YYYY/MM/DD/hh/mm/ format, and a UUID. Data Stream adds the / after the prefix.
Console
To find the object in Azion Console:
Access Azion Console > Object Storage > Buckets.
Select the bucket the stream sends to.
With the prefix activity, the object is listed under a key that starts with activity/.
The object is in the bucket, which confirms that the stream delivered a batch.
Read the send in Real-Time Events
Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned.
Console
To find the sends in Azion Console:
Access Azion Console > Real-Time Events.
Each row is one send. Find the rows with S3 in Endpoint Type, and read their Status Code.
A Status Code of 200 means the bucket accepted the batch. Streamed Lines gives the number of log lines in the batch.
An object under the prefix and a send with status 200 together confirm the delivery. A status of 503 means Data Stream found the endpoint unavailable. With Object Storage, check first that the credential carries listAllBucketNames and listBuckets. For the causes, refer to Troubleshoot Data Stream.