Skip to content

out_azure_kusto: added workload identity #1631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

out_azure_kusto: added workload identity #1631

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e2f15be
Update azure_kusto.md for buffering commits
tanmaya-panda1 Jan 13, 2025
bcf5944
Merge pull request #1 from tanmaya-panda1/out_azure_kusto-docs-update…
tanmaya-panda1 Jan 14, 2025
37abfb2
Update pipeline/outputs/azure_kusto.md
esmerel Jan 14, 2025
412b444
Merge branch 'master' into master
tanmaya-panda1 Apr 29, 2025
2f336ba
out_azure_kusto: added workload identity auth mode
tanmaya-panda1 May 6, 2025
3e533aa
Merge branch 'master' into out_azure_kusto_workload_identity
tanmaya-panda1 May 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 83 additions & 27 deletions pipeline/outputs/azure_kusto.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,40 @@ description: Send logs to Azure Data Explorer (Kusto)

The Kusto output plugin allows to ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, via the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism. This output plugin can also be used to ingest logs into an [Eventhouse](https://blog.fabric.microsoft.com/en-us/blog/eventhouse-overview-handling-real-time-data-with-microsoft-fabric/) cluster in Microsoft Fabric Real Time Analytics.

## Authentication Methods

Fluent-Bit can use various authentication methods to connect to your Azure Data Explorer cluster:

### Service Principal Authentication (Default)

For service principal authentication, you'll need to create an Azure AD application:

- [Register an Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
- [Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret)
- [Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants)

Configure Fluent Bit with your application's `tenant_id`, `client_id`, and `client_secret`.

### Managed Identity Authentication

When running on Azure services that support Managed Identities (such as Azure VMs, AKS, or App Service):

1. [Assign the managed identity appropriate permissions to your Kusto database](https://learn.microsoft.com/en-us/azure/data-explorer/configure-managed-identities-cluster)
2. Configure Fluent Bit with `auth_type` set to `managed_identity`
3. For system-assigned identity, set `client_id` to `system`
4. For user-assigned identity, set `client_id` to the managed identity's client ID (GUID)

### Workload Identity Authentication

For Kubernetes environments using Azure Workload Identity:

1. [Set up Azure Workload Identity in your Kubernetes cluster](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster)
2. Configure your pod to use a service account with Workload Identity Federation
3. Configure Fluent Bit with:
- `auth_type` set to `workload_identity`
- `tenant_id` and `client_id` of your Azure AD application
- `workload_identity_token_file` pointing to your token file path (typically `/var/run/secrets/azure/tokens/azure-identity-token`)

## For ingesting into Azure Data Explorer: Creating a Kusto Cluster and Database

You can create an Azure Data Explorer cluster in one of the following ways:
Expand All @@ -20,15 +54,6 @@ You can create an Eventhouse cluster and a KQL database follow the following ste
- [Create an Eventhouse cluster](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-eventhouse-cluster)
- [Create a KQL database](https://docs.microsoft.com/en-us/azure/data-explorer/eventhouse/create-database)


## Creating an Azure Registered Application

Fluent-Bit will use the application's credentials, to ingest data into your cluster.

- [Register an Application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
- [Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret)
- [Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants)

## Creating a Table

Fluent-Bit ingests the event data into Kusto in a JSON format, that by default will include 3 properties:
Expand All @@ -51,11 +76,12 @@ By default, Kusto will insert incoming ingestions into a table by inferring the

| Key | Description | Default |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| tenant_id | _Required if `managed_identity_client_id` is not set_ - The tenant/domain ID of the AAD registered application. | |
| client_id | _Required if `managed_identity_client_id` is not set_ - The client ID of the AAD registered application. | |
| client_secret | _Required if `managed_identity_client_id` is not set_ - The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). |
| managed_identity_client_id | _Required if `tenant_id`, `client_id`, and `client_secret` are not set_ - The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (GUID) for user-assigned managed identity. | |
| ingestion_endpoint | _Required_ - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net |
| tenant_id | _Required for service principal and workload identity auth_ - The tenant/domain ID of the AAD registered application. | |
| client_id | _Required for service principal and workload identity auth_ - The client ID of the AAD registered application. When using managed identity authentication, set this to 'system' for system-assigned identity or provide the managed identity's client ID. | |
| client_secret | _Required for service principal auth_ - The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). | |
| workload_identity_token_file | _Required for workload identity auth_ - The file path containing the workload identity token when using Azure Workload Identity authentication in Kubernetes. | /var/run/secrets/azure/tokens/azure-identity-token |
| auth_type | Authentication type to use. Supported values: `service_principal` (default), `managed_identity`, `workload_identity`. | `service_principal` |
| ingestion_endpoint | _Required_ - The cluster's ingestion endpoint, usually in the form `https://ingest-cluster_name.region.kusto.windows.net` | |
| database_name | _Required_ - The database name. | |
| table_name | _Required_ - The table name. | |
| ingestion_mapping_reference | _Optional_ - The name of a [JSON ingestion mapping](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/mappings#json-mapping) that will be used to map the ingested payload into the table columns. | |
Expand Down Expand Up @@ -83,7 +109,9 @@ By default, Kusto will insert incoming ingestions into a table by inferring the

### Configuration File

Get started quickly with this configuration file:
Get started quickly with these configuration examples:

#### Service Principal Authentication (Default)

```
[OUTPUT]
Expand All @@ -99,18 +127,46 @@ Get started quickly with this configuration file:
ingestion_endpoint_connect_timeout <ingestion_endpoint_connect_timeout>
compression_enabled <compression_enabled>
ingestion_resources_refresh_interval <ingestion_resources_refresh_interval>
buffering_enabled On
upload_timeout 2m
upload_file_size 125M
azure_kusto_buffer_key kusto1
buffer_file_delete_early Off
unify_tag On
buffer_dir /var/log/
store_dir_limit_size 16GB
blob_uri_length 128
scheduler_max_retries 3
delete_on_max_upload_error Off
io_timeout 60s
buffering_enabled <buffering_enabled>
upload_timeout <upload_timeout>
upload_file_size <upload_file_size>
azure_kusto_buffer_key <azure_kusto_buffer_key>
buffer_file_delete_early <buffer_file_delete_early>
unify_tag <unify_tag>
buffer_dir <buffer_dir>
blob_uri_length <blob_uri_length>
scheduler_max_retries <scheduler_max_retries>
delete_on_max_upload_error <delete_on_max_upload_error>
```

#### Managed Identity Authentication

```
[OUTPUT]
Match *
Name azure_kusto
Auth_Type managed_identity
Client_Id <managed_identity_client_id> # Use 'system' for system-assigned managed identity
Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
Database_Name <database_name>
Table_Name <table_name>
# Additional parameters as needed
```

#### Workload Identity Authentication

```
[OUTPUT]
Match *
Name azure_kusto
Auth_Type workload_identity
Tenant_Id <tenant_id>
Client_Id <client_id>
Workload_Identity_Token_File /var/run/secrets/azure/tokens/azure-identity-token
Ingestion_Endpoint https://ingest-<cluster>.<region>.kusto.windows.net
Database_Name <database_name>
Table_Name <table_name>
# Additional parameters as needed
```

## Troubleshooting
Expand Down