Custom Ingestion into multiple tables

[cherish-inampudi]

Component(s)

exporter/clickhouse

Describe the issue you're reporting

Can we have a provision to customize insert query or insert into multiple tables while exporting to clickhouse.

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

Hi @cherish-inampudi, have you tried using multiple clickhouse exporter? like:

receivers:
  examplereceiver:
processors:
  batch:
    timeout: 5s
    send_batch_size: 100000
exporters:
  clickhouse/one:
    endpoint: tcp://127.0.0.1:9000?dial_timeout=10s&compress=lz4
    database: otel
    ttl_days: 3
    logs_table_name: otel_logs_one
    timeout: 5s
  clickhouse/two:
    endpoint: tcp://127.0.0.2:9000?dial_timeout=10s&compress=lz4
    database: otel
    ttl_days: 3
    logs_table_name: otel_logs_two
    timeout: 5s
service:
  pipelines:
    logs:
      receivers: [ examplereceiver ]
      processors: [ batch ]
      exporters: [ clickhouse/one,clickhouse/two ]

you can also combine a routingprocessor.

[cherish-inampudi]

Thank you @Frapschen, This helps. If I understand this correctly, this exports a whole log based on the conditions we put in the processor. Is it also possible to export some fields to one table and the remaining fields to a different table? For example, the resource fields to a resource table and the log records to the logs table.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.