Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/azuredataexplorer] Accept exporterhelper config for azure data explorer exporter #24329

Closed
david-esteves-sh opened this issue Jul 17, 2023 · 6 comments
Labels

Comments

@david-esteves-sh
Copy link

david-esteves-sh commented Jul 17, 2023

Component(s)

exporter/azuredataexplorer

Is your feature request related to a problem? Please describe.

I am currently using the AzureDataExplorer exporter from the opentelemetry-collector-contrib and do not know of a way to configure the throughput or persistence with the current configuration. The current implementation already uses the ExportHelper but the queue_settings values are not configurable. I would like to be able to tweak the configuration to maximize throughput and also use the storage queue options to prevent data loss on shutdown.

Describe the solution you'd like

Allow to pass in sending_queue, retry_on_failure and timeout config values from the export helper as supported in many other exporters (e.g. otlpexporter, kafkaexporter, etc). Either for the entire exporter or per telemetry type, we could have it configurable by for metrics, logs, traces all separately as a lot of the config today seems to be

exporters:
  azuredataexplorer:
    cluster_uri: "https://CLUSTER.kusto.windows.net"
    application_id: "f80da32c-108c-415c-a19e-643f461a677a"
    application_key: "xx-xx-xx-xx"
    db_name: "oteldb"
    traces_table_name: "OTELTraces"
    # Newly acceptable config values
    retry_on_failure:
      enabled: false
    sending_queue: 
      storage: file_storage/otc
      num_consumers: 25
      queue_size: 30000    

or

exporters:
  azuredataexplorer:
    cluster_uri: "https://CLUSTER.kusto.windows.net"
    application_id: "f80da32c-108c-415c-a19e-643f461a677a"
    application_key: "xx-xx-xx-xx"
    db_name: "oteldb"
    traces_table_name: "OTELTraces"
    # Newly acceptable config values
    retry_on_failure:
      enabled: false
    traces_sending_queue: 
      storage: file_storage/otc
      num_consumers: 25
      queue_size: 30000    
    metrics_sending_queue: 
      num_consumers: 5
      queue_size: 300  

Describe alternatives you've considered

No response

Additional context

No response

@david-esteves-sh david-esteves-sh added enhancement New feature or request needs triage New item requiring triage labels Jul 17, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ag-ramachandran
Copy link
Contributor

ag-ramachandran commented Jul 26, 2023

Hello @david-esteves-sh
We'll evaluate this and see how we can prioritize this for one of the upcoming releases

cc: @asaharn @kustonaut

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

  • needs: Github issue template generation code needs this to generate the corresponding labels.
  • exporter/azuredataexplorer: @asaharn @ag-ramachandran

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@crobert-1
Copy link
Member

crobert-1 commented Sep 29, 2023

Removing needs triage and stale as a code owner has opened a PR for this. and assigning to @asaharn as a PR has been opened.

For some reason I can't assign it, but #27272 is open for this.

@crobert-1 crobert-1 removed Stale needs triage New item requiring triage labels Sep 29, 2023
TylerHelmuth pushed a commit that referenced this issue Oct 12, 2023
…#27272)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Added support for  Exporter Helper configuration. 
**Link to tracking Issue:** #24329 

**Testing:** <Describe what testing was performed and which tests were
added.> Added tests and manually tested with e2e scenarios

---------

Co-authored-by: Ramachandran A G <ramacg@microsoft.com>
Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Co-authored-by: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com>
@ag-ramachandran
Copy link
Contributor

ag-ramachandran commented Oct 13, 2023

Hi @crobert-1 , @david-esteves-sh , This PR is now merged. Can this issue be closed!

@crobert-1
Copy link
Member

Absolutely, thanks for letting me know!

Fixed by #27272

JaredTan95 pushed a commit to openinsight-proj/opentelemetry-collector-contrib that referenced this issue Oct 18, 2023
…open-telemetry#27272)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Added support for  Exporter Helper configuration. 
**Link to tracking Issue:** open-telemetry#24329 

**Testing:** <Describe what testing was performed and which tests were
added.> Added tests and manually tested with e2e scenarios

---------

Co-authored-by: Ramachandran A G <ramacg@microsoft.com>
Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Co-authored-by: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com>
jmsnll pushed a commit to jmsnll/opentelemetry-collector-contrib that referenced this issue Nov 12, 2023
…open-telemetry#27272)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
Added support for  Exporter Helper configuration. 
**Link to tracking Issue:** open-telemetry#24329 

**Testing:** <Describe what testing was performed and which tests were
added.> Added tests and manually tested with e2e scenarios

---------

Co-authored-by: Ramachandran A G <ramacg@microsoft.com>
Co-authored-by: Ziqi Zhao <zhaoziqi9146@gmail.com>
Co-authored-by: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants