Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow setting of storage policy for clickhouse exporter #32816

Closed
sdejong629 opened this issue May 2, 2024 · 5 comments
Closed

Allow setting of storage policy for clickhouse exporter #32816

sdejong629 opened this issue May 2, 2024 · 5 comments
Labels
enhancement New feature or request exporter/clickhouse needs triage New item requiring triage Stale

Comments

@sdejong629
Copy link

Component(s)

exporter/clickhouse

Is your feature request related to a problem? Please describe.

The exporter creates a new database and table, but when a specific storage policy is required for the table, this can not be set on table creation by the exporter.

Describe the solution you'd like

Add an option in de clickhouse_exporter config for allowing to set a custom storage policy on table create

Describe alternatives you've considered

It can be set afterwards in clickhouse, but this requires you to include the current storage policy and move data to the new storage before disabling the 'old' storage policy

Additional context

See the final SETTINGS line for table creation

CREATE TABLE otel_logs ON CLUSTER clickhouse_cluster
(
    `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
    `TraceId` String CODEC(ZSTD(1)),
    `SpanId` String CODEC(ZSTD(1)),
    `TraceFlags` UInt32 CODEC(ZSTD(1)),
    `SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
    `SeverityNumber` Int32 CODEC(ZSTD(1)),
    `ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
    `Body` String CODEC(ZSTD(1)),
    `ResourceSchemaUrl` String CODEC(ZSTD(1)),
    `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `ScopeSchemaUrl` String CODEC(ZSTD(1)),
    `ScopeName` String CODEC(ZSTD(1)),
    `ScopeVersion` String CODEC(ZSTD(1)),
    `ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    `LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
    INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
    INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
    INDEX idx_body Body TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 1
)
ENGINE = MergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (ServiceName, SeverityText, toUnixTimestamp(Timestamp), TraceId)
TTL toDateTime(Timestamp) + toIntervalHour(8)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1, storage_policy = 'custom_storage_policy';
@sdejong629 sdejong629 added enhancement New feature or request needs triage New item requiring triage labels May 2, 2024
Copy link
Contributor

github-actions bot commented May 2, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@hanjm
Copy link
Member

hanjm commented May 10, 2024

Looks goods to add new config field support extra setting. hi, @SpencerTorres what is your ideal and could have time to implement it?

@SpencerTorres
Copy link
Member

ClickHouse has a lot of options when configuring a table, too many for us to reasonably keep up with and add to the exporter config.

Instead, I recommend using the new create_schema: false option in the config and manually creating the required tables yourself. In a production environment, it's not good to rely on multiple instances of the exporter racing to auto-create the schema.

When you set create_schema to false, it will not create the tables for you. This lets you create the tables manually and have full control over the schema/settings. It also lets you know WHEN the table is created, instead of questioning whether the exporter did it.

If you need an example of what DDL the exporter would run, you can check the default_ddl/ folder. (This is a permalink, be sure to use the the version that matches your exporter.)

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@SpencerTorres
Copy link
Member

See my above response on controlling these specific properties. Table creation can now be fully managed by the user and is the recommended way to run the DDL.

We can close this @crobert-1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/clickhouse needs triage New item requiring triage Stale
Projects
None yet
Development

No branches or pull requests

4 participants