Road to v0.2

[matthewmturner]

This issue for setting the specific issues we would like completed before a v0.2 release. My hope is that there can be a release by end of Sept / Beginning of October.

• Querys that return large # of results can freeze / crash the app #137
• To investigate: Is DDL working correctly? #138
• Add --flightsql paramter for -c and -f #128
• (REQUIRED) Add tests for configuration system #144
• (REQUIRED) Implement select * from 'foo.parquet' support #125
• Add back DDL mode #153
• (REQUIRED) Create / add duckdb metadata functions #148
• (REQUIRED) Update to DataFusion 42.0.0 #158
• Add syntax highlighting to config #159
• Update from datafusionrc to ddl in config dir #166
• (REQUIRED) Figure out ddl mode executed / error UI #167
• (REQUIRED) DDL should optionally be available from CLI #170
• Add serve subcommand for flightsql feature #168
• Setup localstack for testing s3 #173
• Add ui to flightsql tab when unable to connect #175
• Test flightsql error cases #177
• (REQUIRED) Benchmark from CLI
• (REQUIRED) Analyze from CLI
• DDL mode isnt loading DDL file correctly now #213
• Add select / copy / paste / run selection to tui editor #224

[matthewmturner]

@alamb let me know if there is anything you would like to add

[alamb]

I think we should consider adding #148 and #158 as well

[matthewmturner]

So I think it turns out that the DDL issue I have is a bug with datafusion registering external tables to a schema.

I have this in my ~/.datafusion/.datafusionrc`

CREATE SCHEMA staging;

CREATE TABLE staging.foo AS VALUES (1);
CREATE EXTERNAL TABLE staging.min_aggs STORED AS PARQUET LOCATION 'ny2://atlas/sip/aggregates/minute_by_ticker_monthly_v2/year_month=2015-08/data.01.parquet';

But when I run SHOW TABLES I get the following:

Notice how the schema isnt picked up for the the external table. I dont see anything in the docs for CREATE EXTERNAL TABLE that would suggest this shouldnt work but maybe im misunderstanding.

@alamb do you have a view on this? If you agree its a bug in datafusion then i can create a ticket there.

[alamb]

I agree it is a datafusion bug. I will file a ticket

[alamb]

Filed apache/datafusion#12607

[Kinrany]

Is there anything that the current trunk version does worse than v0.1?

[matthewmturner]

@Kinrany i dont think so although i havent used v0.1 in a long time (years). i took a brief pause for december to work on some fun not work related project but im about to get back into dft in the beginning of the new year. im going to review where things are again and potentially do a release early on before proceeding to the next round of features i have in mind.

[matthewmturner]

I am going to work on #257 and then will do 0.2 release

[matthewmturner]

I was actually just poking around OpenDAL (specifically it's ObjectStore implementation) and it may actually be very easy to integrate hugging face - so i might see if it's as easy as im expecting and if so i will include that as well (basically the code in the example i linked but use the services-huggingface instead of services-s3).

[matthewmturner]

I was able to add the huggingface integration and ive started prepping for 0.2 release. however, im currently pinned to a specific git commit for the hudi feature and you cant publish versions using that. it looks like the hudi team is close to a new version but im not sure how long that will take.

i might see if i can use the FFI table provider interface instead so i prevent these version issues from coming up in the future.

[matthewmturner]

I ended up removing the hudi feature for now so i could move forward with the 0.2 release - there are several other features other than the FFI table provider i would like to work on although i do plan to get back to that.