Send logs to Google BigQuery

Create a stream that writes the logs of your applications as rows in a Google BigQuery table, in Azion Console or with the Azion API, and confirm the delivery.

Last updated

View as Markdown Agent setup

You can send the logs of a stream to a table in Google BigQuery from Azion Console or with the Azion API.

Data Stream sends the log lines as rows to a table of a BigQuery dataset. In the stream form, the endpoint is set in the field labeled Connector, and BigQuery is its Google BigQuery option. For every field and its bounds, refer to Endpoints.

The stream signs in to Google with a service account key that you provide. Data Stream performs the Google OAuth 2.0 authentication and generates the JSON Web Tokens, so the key is the only credential you set. The example collects the requests of one workload with the Applications data source.

Prerequisites

Console

  • An Azion account with the Edit Data Stream permission. For the permissions, refer to Stream settings.
  • A workload on the account that receives requests.
  • A Google Cloud project with the BigQuery API enabled, and the ID of the project.
  • Billing enabled on the project. The free tier does not accept rows streamed into a table. Google Cloud lists the fees for these inserts under Streaming Inserts in the BigQuery price table.
  • A dataset in the project, and its dataset ID. The dataset ID is case sensitive.
  • A table in the dataset, with a schema for the data the stream sends, and its table ID. The table ID is the name you give the table.
  • A Google Cloud service account with the BigQuery Admin role, and a key of the service account in JSON format. Google Cloud downloads the key as a JSON file when you create it.

The key file holds the keys below. Data Stream takes the whole content of the file:

JSON
01{
02 "type": "service_account",
03 "project_id": "<project-id>",
04 "private_key_id": "<private-key-id>",
05 "private_key": "<private-key>",
06 "client_email": "<client-email>",
07 "client_id": "<client-id>",
08 "auth_uri": "<auth-uri>",
09 "token_uri": "<token-uri>",
10 "auth_provider_x509_cert_url": "<auth-provider-cert-url>",
11 "client_x509_cert_url": "<client-cert-url>"
12}

Create the stream

The stream collects from the workload you choose, through a workload filter. A workload filter keeps your other streams active, which a sampled stream does not.

Console

To create the stream in Azion Console:

Open Data Stream

Access Azion Console > Data Stream.

Select + Stream
Name the stream

In the General section, enter a Name. For example: logs-to-bigquery.

Select the data source

In the Input section, select Applications in Data Source.

Turn off Sampling

In the Transform section, turn off Sampling while Option is still All Current and Future Workloads, its starting value. A stream cannot carry sampling and a workload filter together.

Choose the workload

In the Transform section, set Option to Filter Workloads. In Available Workload, select your workload and move it to Chosen Workload with the arrow. If your Console names domains where this guide names workloads, the same control reads Filter Domains. The pick list then reads Available Domain and Chosen Domain.

Select the template

In the Render Template section, select Applications Event Collector in Template.

Select the BigQuery endpoint

In the Output section, select Google BigQuery in Connector.

Enter the table location

Enter the ID of your Google Cloud project in Project ID, the dataset ID in Dataset ID, and the table ID in Table ID. For example: my-project, azion_logs, and requests.

Paste the service account key

In Service Account Key, paste the whole content of the JSON key file. The field is a JSON editor.

Keep the stream active

In the Status section, keep Active turned on.

Select Save

The Console shows Your data stream has been created. The stream appears in the Data Stream list with BigQuery in the Connector column and the Active status.

The API and the Console save the stream without contacting Google Cloud. A wrong project, dataset, table, or key surfaces only when the stream sends. An activation takes effect after one to two minutes.

Confirm the delivery

Real-Time Events records every send of a stream, delivered or not, with the status code the endpoint returned. Send a few requests to the workload, then wait about a minute: a stream sends a batch every 60 seconds, or sooner when it reaches 2,000 log lines.

Console

To find the sends in Azion Console:

Open Real-Time Events

Access Azion Console > Real-Time Events.

Select the Data Stream data source
Read the latest sends

Each row is one send. Find the rows with BIG_QUERY in Endpoint Type, and read their Status Code.

A Status Code of 200 means BigQuery accepted the batch. Streamed Lines gives the number of log lines in the batch.

A status other than 200 is the answer of the endpoint, and 503 means Data Stream found the endpoint unavailable. For the causes, refer to Troubleshoot Data Stream.

Next steps

EndpointsEvery field of the BigQuery endpoint, with its type, bounds, and API name.Create a custom templateChoose the variables of each log line, and so the data the table receives.Edit, stop, or delete a streamChange the table or the key, pause the stream, or remove it.Troubleshoot Data StreamFind what to change when a send returns a status other than 200.