Add a hint about expected extension in error message in register_csv, register_parquet, register_json

[cj-zhukov]

Describe the bug

When attempting to register an existing file with different format using the register_csv, register_json, register_parquet, instead of receiving an error, the operation silently returns an empty dataframe.

To Reproduce

let file = "foo.parquet"; // foo.parquet does exist
let ctx = SessionContext::new();
ctx.register_csv("foo", file, CsvReadOptions::default()).await?; // trying to register parquet file with register_csv
// ctx.register_json("foo", file, NdJsonReadOptions::default()).await?; // or same using register_json
let res = ctx.sql("select * from foo").await?;
res.show().await?;
// Observe that no error is raised, but an empty DataFrame is returned.
// ++
// ++

Expected behavior

Introduce a validation step in register_csv, register_json, register_parquet to check the file extension or validate the file's format. If the file does not match the expected format, return a clear error message:

Error: Execution("File path 'path/to/file' does not match the expected extension '.extension'")

Additional context

No response

[cj-zhukov]

take

[devhprl]

@cj-zhukov @alamb hi, I've just hit this one. judging file content by an extension doesn't feel fully right to me (as you can put there basically anything). yes if there is an content mismatch (expected vs actual) then there will be an undesired result (although I would expect some parsing error or something). nevertheless enforcing untrustworthy extensions doesn't seem to bring much value in that regard. in our case we intentionally omit the extensions completely in the storage and have other ways how the content info gets propagated (to the point where register_xyz gets picked). thanks for the great lib

[cj-zhukov]

@devhprl hi, thanks for the comment! When I was working on this, I took in consideration how read_xyz methods work. And they check if the file extension matches the expected extension. So it was one of the reason why I chose this approach for register_xyz methods as well. Actually, I don't see any good reason not to relay on file extension in this regard. For example, if I have some foo.parquet file, I expect to read (or register) it into dataframe with read_parquet method (but not read_csv).

[alamb]

@cj-zhukov @alamb hi, I've just hit this one. judging file content by an extension doesn't feel fully right to me (as you can put there basically anything). yes if there is an content mismatch (expected vs actual) then there will be an undesired result (although I would expect some parsing error or something). nevertheless enforcing untrustworthy extensions doesn't seem to bring much value in that regard. in our case we intentionally omit the extensions completely in the storage and have other ways how the content info gets propagated (to the point where register_xyz gets picked). thanks for the great lib

I agree that ensuring there is some way to override the format used to interpret files (e.g. reading a .dat file as parquet) is a good feature. @devhprl if there is something preventing you from doing that, please do file an issue with a reproducer and we can help make sure the APIs support your usecase

[devhprl]

thanks guys. will dig into that a bit. in the meantime I am slightly adjusting our case to either fit in to the current validations or shape up the need properly.