perf(duckdb): enable parquet metadata cache and prefetch

[khalid244]

Summary

• Enable parquet_metadata_cache=true to cache Parquet schema and row group info
• Enable prefetch_all_parquet_files=true to prefetch row groups asynchronously

Test plan

• Tested with cold and warm query benchmarks
• Verified settings are applied via DuckDB logs

[xe-nvdk]

Thanks for the PR — good find on parquet_metadata_cache. We confirmed that the existing enable_object_cache=true on line 140 is actually a legacy no-op on DuckDB 1.4.3 (it's marked as [PLACEHOLDER] Legacy setting - does nothing), so this has real value.

Two changes needed:

1. Move parquet_metadata_cache to general init (not S3-only)

This setting caches Parquet footer/schema/row group metadata and benefits all storage backends (local, S3, Azure). Currently it's inside configureS3Access(), so local-disk deployments won't get it.

Move it to the general DuckDB init section (around line 140) and replace the dead enable_object_cache line:

// Cache Parquet file metadata (schema, row group info) to reduce I/O on repeated access
if _, err := db.Exec("SET parquet_metadata_cache=true"); err != nil {
    logger.Warn().Err(err).Msg("Failed to enable parquet metadata cache (continuing without it)")
}

2. Keep prefetch_all_parquet_files inside configureS3Access() ✅

This one is correctly placed — prefetching reduces HTTP round-trips for remote storage but has negligible impact on local disk. No change needed here.

Summary

• parquet_metadata_cache → move to general init, replace enable_object_cache
• prefetch_all_parquet_files → keep where it is
• Remove the old enable_object_cache line (line 140)

[khalid244]

Done

[xe-nvdk]

Looks good now — placement is correct. One small nit:

Both lines use SET GLOBAL but the rest of the init function uses SET. For consistency, change to:

if _, err := db.Exec("SET parquet_metadata_cache=true"); err != nil {

and

if _, err := db.Exec("SET prefetch_all_parquet_files=true"); err != nil {

Once that's updated, this is good to merge.

[khalid244]

SET - Sets the value for the current connection only

• Resets when the connection closes
• Other connections don't see the change

SET GLOBAL - Sets the value for all connections in the database instance

• Persists across all connections in the connection pool
• New connections inherit the global setting

For Arc, which uses a connection pool, SET GLOBAL is important because:

• Queries can run on any connection from the pool
• Without SET GLOBAL, only the connection that ran SET would have the setting
• Other pooled connections would use the default value

Example:
// Only affects this one connection
db.Exec("SET parquet_metadata_cache=true")

// Affects all connections in the pool
db.Exec("SET GLOBAL parquet_metadata_cache=true")

We should change all SET to SET GLOBAL in Arc codebase

[xe-nvdk]

Thanks for the contribution! Merged. 🎉