sqllogictests failing locally due to "No such file or directory"

[andygrove]

Describe the bug

I am running cargo test on main branch at commit 974d49c and hitting this error. A second run worked fine.

     Running bin/sqllogictests.rs (target/debug/deps/sqllogictests-32de638f765421ed)
Running "pg_compat/pg_compat_types.slt"
Running "pg_compat/pg_compat_union.slt"
Running "pg_compat/pg_compat_window.slt"
Running "pg_compat/pg_compat_simple.slt"
Running "pg_compat/pg_compat_null.slt"
Running "pg_compat/pg_compat_type_coercion.slt"
Running "wildcard.slt"
Running "information_schema_columns.slt"
Running "projection.slt"
Running "nullif.slt"
Running "subquery.slt"
Running "options.slt"
Running "arrow_files.slt"
Running "dates.slt"
Running "misc.slt"
Running "prepare.slt"
Running "avro.slt"
Running "struct.slt"
Running "functions.slt"
Running "predicates.slt"
Running "intersection.slt"
Running "timestamps.slt"
Running "limit.slt"
Running "window.slt"
Running "ddl.slt"
Running "union.slt"
Running "insert.slt"
Running "distinct_on.slt"
Running "strings.slt"
Running "join.slt"
Running "describe.slt"
Running "explain.slt"
Running "decimal.slt"
Running "aggregate.slt"
Running "join_disable_repartition_joins.slt"
Running "clickbench.slt"
Running "csv_files.slt"
Running "information_schema_multiple_catalogs.slt"
Running "math.slt"
Running "arrow_typeof.slt"
Running "json.slt"
Running "information_schema_table_types.slt"
Running "copy.slt"
Running "set_variable.slt"
Running "scalar.slt"
Running "map.slt"
Running "cte.slt"
Running "schema_evolution.slt"
Running "binary.slt"
Running "cast.slt"
Running "order.slt"
Running "interval.slt"
Running "groupby.slt"
Running "metadata.slt"
Running "insert_to_external.slt"
Running "encoding.slt"
Running "identifiers.slt"
Running "group.slt"
thread 'main' panicked at datafusion/sqllogictest/bin/sqllogictests.rs:264:14:
Readable directory: Os { code: 2, kind: NotFound, message: "No such file or directory" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at /home/andy/git/apache/arrow-datafusion/datafusion/physical-plan/src/lib.rs:540:21:
internal error: entered unreachable code

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[alamb]

Here is the line that is panic'ing: https://github.com/apache/arrow-datafusion/blob/974d49c907c5ddb250895efaef0952ed063f1831/datafusion/sqllogictest/bin/sqllogictests.rs#L264

[alamb]

Looking into this

[andygrove]

What command are you using to run this?

cargo test

[alamb]

🤔 it passed for me locally. I am now running the test on a fresh checkout...

[alamb]

It also worked fine on a fresh checkout

I can reproduce the error if I delete the ./datafusion/sqllogictest/test_files directory 🤔

andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion$ rm -rf ./datafusion/sqllogictest/test_files
andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion$ cargo test --test sqllogictests
    Finished test [unoptimized + debuginfo] target(s) in 0.13s
     Running bin/sqllogictests.rs (target/debug/deps/sqllogictests-a4acaa02cc91a4e3)
thread 'main' panicked at datafusion/sqllogictest/bin/sqllogictests.rs:264:14:
Readable directory: Os { code: 2, kind: NotFound, message: "No such file or directory" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: test failed, to rerun pass `-p datafusion-sqllogictest --test sqllogictests`

[andygrove]

It worked for me on a second or third run, so definitely seems to be an intermittent issue

[tustvold]

FWIW #8539 might help with this

[alamb]

FWIW #8539 might help with this

The error reported in this PR seems to be related to reading the slt files themselves, rather than with something they are doing during execution (which is what #8539 is related to)

I did make #8544 to at least make the error messages better