Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[extension/storage/dbstorage] pgx support not wokring because db queries are sqlite specific #35692

Closed
danelson opened this issue Oct 8, 2024 · 2 comments · Fixed by #35757
Closed
Assignees
Labels
bug Something isn't working extension/storage/dbstorage

Comments

@danelson
Copy link
Contributor

danelson commented Oct 8, 2024

Component(s)

extension/storage/dbstorage

What happened?

Description

The dbstorage extension says that both sqlite3 and pgx are supported.

driver: the name of the database driver to use. By default, the storage client supports "sqlite3" and "pgx".

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.110.0/extension/storage/dbstorage/README.md?plain=1#L21

However when using the pgx driver, the create table statement uses syntax specific to sqlite3 which causes a failure

error graph/graph.go:425 Failed to start component {"error": "error connecting to storage: ERROR: type "blob" does not exist (SQLSTATE 42704)", "type": "Receiver", "id": "sqlquery"}

The queries in https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/storage/dbstorage/client.go#L20-L23 look specific to sqlite3.

Steps to Reproduce

See collector configuration below

Expected Result

I would expect there not to be an error and that the tracking table be created

Actual Result

error graph/graph.go:425 Failed to start component {"error": "error connecting to storage: ERROR: type "blob" does not exist (SQLSTATE 42704)", "type": "Receiver", "id": "sqlquery"}

Collector version

v0.110.0
Custom collector distro running on alpine compiled with go 1.22

OpenTelemetry Collector configuration

receivers:
  sqlquery:
    driver: postgres
    datasource: "host=localhost port=5432 user=postgres password=s3cr3t sslmode=disable"
    queries:
      - sql: "select * from my_logs where log_id > $$1"
        tracking_start_value: "1" # Not sure if this is correct yet but it should not matter for this issue
        tracking_column: log_id
        logs:
          - body_column: log_body

exporters:
  debug:
    verbosity: detailed

extensions:
  db_storage:
    driver: pgx
    datasource: "host=localhost port=5432 user=postgres password=s3cr3t sslmode=disable"

service:
  extensions: [db_storage]
  pipelines:
    logs:
      receivers: [sqlquery]
      exporters: [debug]

Log output

error   graph/graph.go:425      Failed to start component       {"error": "error connecting to storage: ERROR: type \"blob\" does not exist (SQLSTATE 42704)", "type": "Receiver", "id": "sqlquery"}

Additional context

Once this is fixed I think there may also be an issue with the tracking table name as I think it will include invalid characters (at least when using sqlqueryreceiver). This is inferring the table name from the code which I might not being doing right. I can't test with sqlite3 due to #35280

@danelson danelson added bug Something isn't working needs triage New item requiring triage labels Oct 8, 2024
Copy link
Contributor

github-actions bot commented Oct 8, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@danelson danelson changed the title [extension/dbstorage] pgx support not wokring because db queries are sqlite specific [extension/storage/dbstorage] pgx support not wokring because db queries are sqlite specific Oct 8, 2024
@atoulme atoulme removed the needs triage New item requiring triage label Oct 12, 2024
@atoulme
Copy link
Contributor

atoulme commented Oct 12, 2024

Thanks for the report. We can look into it.

@atoulme atoulme self-assigned this Oct 12, 2024
michael-burt pushed a commit to michael-burt/opentelemetry-collector-contrib that referenced this issue Nov 7, 2024
#### Description
Adds support for Postgres, using a test running a Postgresql docker
image.

#### Link to tracking issue
Fixes open-telemetry#35692

#### Testing
Run a postgresql image and test with it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working extension/storage/dbstorage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants