Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Auto create table in BigQuery #13033

Open
Hipska opened this issue Apr 4, 2023 · 2 comments
Open

Auto create table in BigQuery #13033

Hipska opened this issue Apr 4, 2023 · 2 comments
Labels
area/gcp Google Cloud plugins including cloud_pubsub, cloud_pubsub_push, stackdriver feature request Requests for new plugin and for new features to existing plugins help wanted Request for community participation, code, contribution plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins

Comments

@Hipska
Copy link
Contributor

Hipska commented Apr 4, 2023

Use Case

With some input plugins the metric names and formats are unknown on beforehand, the number of tags and fields can vary over time. Therefore it would be nice if the outputs.bigquery plugin could handle this in a way similar to the outputs.sql plugin.

Expected behavior

Telegraf creates the required table in BigQuery with all the corresponding metric fields and tags just as the sql

Actual behavior

Tables for metrics need to be created on beforehand in BigQuery.

Additional info

Here is a PoC which adds a new BigQuery field on the fly, proving it should be possible.
@Hipska Hipska added feature request Requests for new plugin and for new features to existing plugins area/gcp Google Cloud plugins including cloud_pubsub, cloud_pubsub_push, stackdriver plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins labels Apr 4, 2023
@powersj
Copy link
Contributor

powersj commented Apr 4, 2023

Hi,

Are you planning on creating a PR with that PoC? Is this something you are able to test as well?

@powersj powersj added the waiting for response waiting for response from contributor label Apr 4, 2023
@Hipska
Copy link
Contributor Author

Hipska commented Apr 12, 2023

If my customer wants this feature, I will try to create a PR for it.

@telegraf-tiger telegraf-tiger bot removed the waiting for response waiting for response from contributor label Apr 12, 2023
@Hipska Hipska added the help wanted Request for community participation, code, contribution label Nov 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/gcp Google Cloud plugins including cloud_pubsub, cloud_pubsub_push, stackdriver feature request Requests for new plugin and for new features to existing plugins help wanted Request for community participation, code, contribution plugin/output 1. Request for new output plugins 2. Issues/PRs that are related to out plugins
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Hipska @powersj