Fix query failure when daily compacted files don't exist

[khalid244]

Summary

• Add retry logic for queries failing with "No files found" on day-level paths
• Zero overhead on happy path (retry only on failure)

[xe-nvdk]

Thanks for investigating this! You've found a real edge case.

After digging deeper, I believe the issue is in how the pruner validates day-level paths. Currently:

1. Pruner generates db/meas/2026/01/15/*.parquet for daily files
2. It checks if directory 2026/01/15/ exists (it does - has hourly subdirs like 00/, 01/, etc.)
3. But no *.parquet files exist directly in that directory (daily compaction hasn't run)
4. Path is included → DuckDB fails with "No files found"

The fix should be in the pruner itself rather than retry logic. A couple of options:

Option A: Change filterExistingRemotePaths() in internal/pruning/partition_pruner.go to check for actual file presence at day level, not just directory existence

Option B: Only generate day-level paths if we know daily compaction has run (check for _daily.parquet files)

Would you be interested in implementing one of these fixes instead? The retry approach with SQL string manipulation is fragile and could break in edge cases (e.g., if only day-level paths exist, or if the regex doesn't match certain path formats).

I can help review a pruner-level fix if you'd like to take this on!

[khalid244]

Thanks for the suggestions! I agree the fix should be in the pruner.

Regarding the two options:

Option 1 (check file presence) is thorough but could impact performance when users have many uncompacted files,
since we'd need to verify each path.

Option 2 (only generate day-level paths after daily compaction) addresses this specific issue cleanly without the
performance overhead.

However, I'm thinking about other edge cases where paths might not have files—for example, if a user manually
deletes files from S3, or if network issues cause file loss. In those scenarios, Option 1's validation would catch
the problem.

What do you think about combining both approaches?

• Implement Option 2 as the primary fix (check for _daily.parquet before including day-level paths)
• Add Option 1 as a fallback validation layer that surfaces a warning/error to the user when files are missing

This way we get the performance benefit of Option 2 for the common case, while Option 1 handles unexpected
scenarios like deleted files gracefully.

One more idea: when we detect missing files, we could create empty placeholder files to fill the gap. This would
prevent repeated validation overhead on subsequent queries and make the system more robust. What do you think?

[xe-nvdk]

Thanks for the investigation, but after deeper analysis we're going to close this PR. Here's why:

The Scenario Doesn't Happen in Practice

After tracing through the code:

1. For local storage: filterExistingLocalPaths() uses filepath.Glob() which correctly returns empty for day-level paths with no files (even if the directory exists with hourly subdirs). These paths get filtered out before reaching DuckDB.

2. For S3/Azure: The edge case you're describing would require:

   • Storage backend NOT configured (p.storage == nil) - a config bug
   • OR a very specific query that only generates day-level paths (not typical)

In normal usage, queries include both hourly AND daily paths. The hourly paths have data, so DuckDB can establish the schema and the query succeeds.

If Queries Return No Data, That's Correct Behavior

If daily compacted files don't exist, the query returns 0 rows from that path, not a failure. The data is in the hourly paths.

The Proposed Fixes Are Over-Engineering

The suggestions (pruner-level validation, placeholder files, fallback layers) add complexity for an edge case that:

• Doesn't occur in normal usage
• Would be a configuration bug if it did occur
• Has undefined reproduction steps

If you can provide a specific reproduction case with steps and logs showing the actual error, we'd reconsider. But based on code analysis, this appears to be a non-issue.

Thanks for contributing - please don't let this discourage you from future PRs!

[khalid244]

This is actually a real issue I hit. I migrated about a year's worth of data (~700 million records), and after running compaction, all my queries started failing with "No files found" on day-level paths. I'll test more to provide reproduction steps for it.

[xe-nvdk]

Ok, let's do this. Let's convert this to an issue, so, we can work and go deep on this. If you can provide how you ingest the data, how the dataset is stored, folders and sub folders and data structure, would be good to reproduce.

[khalid244]

After further debugging, I've identified that this issue is related to #131.
The compactor sends requests to S3 using HTTP instead of HTTPS, even when ARC_STORAGE_S3_USE_SSL=true is set.
This causes daily compaction jobs to fail.

[xe-nvdk]

Oh, OK, is the issue opened, that Im going to tackle today. I will keep you posted.