Support cloud storage as the source/destination of SELECT ... INTO OUTFILE & LOAD DATA

[SunRunAway]

Feature Request

Is your feature request related to a problem? Please describe:

In the cloud environment, it is impossible to load files locally.

Describe the feature you'd like:

Syntax:

SELECT * FROM t INTO OUTFILE S3 's3://bucket-name/data.txt?region=us-west-2';
--- or
SELECT * FROM t INTO OUTFILE S3 's3-us-west-2://bucket-name/data.txt';

LOAD DATA FROM S3 's3://bucket-name/data.txt?region=us-west-2' INTO TABLE t;

The scheme and the parameters of S3 file can be similar with BR's, along with the credential policy.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[SunRunAway]

cc @zz-jason @nullnotnil

[ghost]

I think this is a very useful FR. In terms of syntax, I suggest we follow AWS Aurora, which is slightly different:

SELECT * FROM employees INTO OUTFILE S3 's3-us-west-2://aurora-select-into-s3-pdx/sample_employee_data'
    FIELDS TERMINATED BY ','
    LINES TERMINATED BY '\n';

LOAD DATA FROM S3 's3://mybucket/data.txt'
    INTO TABLE table1
    (column1, @var1)
    SET table_column2 = @var1/100;

Docs:

• SELECT .. INTO OUTFILE
• LOAD DATA

[SunRunAway]

I suggest we follow AWS Aurora

I've updated @nullnotnil's suggestion into the issue description.

---

The current URI in BR places the region information as a URL query like s3://bucket-name/data.txt?region=us-west-2, and AWS Aurora places it in the Schema.
Is it better to support them both for consistency?

[zz-jason]

@IANTHEREAL PTAL

[yufan022]

Hi I'm happy to work for this issue.

[lance6716]

LOAD DATA implements this feature in #40499 . Please notice that some syntax is reverted in later commits than #40499