Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot access locally deployed minio using pg_duckdb #207

Open
transparent1998 opened this issue Sep 24, 2024 · 2 comments
Open

Cannot access locally deployed minio using pg_duckdb #207

transparent1998 opened this issue Sep 24, 2024 · 2 comments
Labels
enhancement New feature or request
Milestone

Comments

@transparent1998
Copy link

transparent1998 commented Sep 24, 2024

pg version:16.4

postgres=# select * from duckdb.secrets ;
 type |          id          |                  secret                  | region | session_token |  endpoint  | r2_account_id | use_ssl 
------+----------------------+------------------------------------------+--------+---------------+------------+---------------+---------
 S3   | xxxxxxxxxxxxxxxxxxxxx| xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx|        |               | minio:9000 |               | f
(1 row)

postgres=# select duckdb.install_extension('httpfs');
 install_extension 
-------------------
 t
(1 row)

postgres=# select * from read_csv('s3://jyt/test.csv') as t (a int,b int);
WARNING:  (PGDuckDB/CreatePlan) Prepared query returned an error: 'HTTP Error: HTTP GET error on 'https://jyt.s3.amazonaws.com/test.csv' (HTTP 403)
WARNING:  (PGDuckDB/CreatePlan) Prepared query returned an error: 'Conversion Error: Type VARCHAR with value '{}' can't be cast to the destination type LIST
LINE 1: ...est.csv'::text, false, true, true, '{}'::text[], 'auto'::character varying, ''...
                                                  ^
ERROR:  Function `read_csv(TEXT)` only works with Duckdb execution.
CONTEXT:  PL/pgSQL function read_csv(text,boolean,boolean,boolean,text[],character varying,character varying,character varying,character varying,character varying,boolean,text[],boolean,boolean,boolean,bigint,text[],character varying,boolean,boolean,text[],boolean,character varying,bigint,character varying,bigint,character varying,text[],boolean) line 3 at RAISE
postgres=# select * from read_csv('/tmp/a.csv') as t (a text);
 a 
---
 1
(1 row)

The key and endpoint are set, but the corresponding address is not modified when accessing, and there is no place to set s3_url_style

D load httpfs;
D set s3_use_ssl='false';
D set s3_url_style='path';
D set s3_endpoint='minio:9000';
D set s3_access_key_id ='xxxxxxxxxxxxxxxxxx';
D set s3_secret_access_key ='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx';
D select * from read_csv('s3://jyt/test.csv');
┌───────┬───────┐
│   a   │   b   │
│ int64 │ int64 │
├───────┼───────┤
│     12 │
└───────┴───────┘
D 

It can be accessed normally in duckdb

@wuputah wuputah changed the title 用pg_duckdb无法访问本地部署的minio Cannot access locally deployed minio using pg_duckdb Sep 24, 2024
@wuputah
Copy link
Collaborator

wuputah commented Sep 24, 2024

Correct, s3_url_style is not currently settable.

As discussed a bit in #197, we might want to move towards a more flexible way of supporting all the various options. For instance it would also be nice to support the scope option… and there's many others.

@JelteF JelteF added this to the 0.2.0 milestone Sep 30, 2024
@JelteF
Copy link
Collaborator

JelteF commented Sep 30, 2024

I guess this will be fixed by #151, once we implement it.

@JelteF JelteF added the enhancement New feature or request label Sep 30, 2024
@JelteF JelteF modified the milestones: 0.2.0, 0.3.0 Dec 10, 2024
@JelteF JelteF modified the milestones: 0.3.0, 0.4.0 Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants