fix(pruning): verify files exist at day-level paths

[khalid244]

Summary

• Fixes S3 partition pruning bug where queries fail with "No files found" when daily compaction hasn't run
• Day-level paths were included when directory existed but contained only hourly subdirectories
• Adds file existence check for day-level paths (5 segments) while keeping fast directory check for hourly paths

Fixes #144

[xe-nvdk]

Hi @khalid244,

Thank you for this PR! I've tested your changes against the issue and found that your fix addresses part of the problem but misses a critical edge case.

What Your PR Fixes

Your PR correctly improves filterExistingRemotePaths() to verify that .parquet files exist at the day-level (5 segments: db/measurement/YYYY/MM/DD), not just that the directory exists. This is the right approach!

Before your fix:

• Day-level path: s3://bucket/db/cpu/2026/01/21/*.parquet
• Directory 2026/01/21/ exists (has hourly subdirs: 00/, 01/, etc.)
• Filter passes (directory exists)
• DuckDB tries to read 2026/01/21/*.parquet (no files) FAILS

After your fix:

• Day-level path: s3://bucket/db/cpu/2026/01/21/*.parquet
• Directory 2026/01/21/ exists (has hourly subdirs)
• Your code checks for .parquet files directly at 2026/01/21/ level
• No files found → filter excludes this path
• Only hourly paths included

This is great progress!

What Your PR Doesn't Fix

However, there's an edge case your PR doesn't handle: queries that span time ranges with NO data at all (e.g., querying future dates or dates before any data was ingested).

The Problem

Testing with MinIO, I found this scenario:

• Data exists: s3://arc-test/production/cpu/2026/01/21/17/*.parquet (today, hour 17)
• Query: SELECT * FROM cpu WHERE time >= '2026-01-20' AND time < '2026-01-22' (yesterday to tomorrow)

What happens:

1. Partition pruner generates paths for 2026-01-20, 2026-01-21, and 2026-01-22
2. Your filterExistingRemotePaths() correctly filters out:
   • 2026/01/20/ (no data - before ingestion started)
   • 2026/01/21/*.parquet (day-level file doesn't exist)
   • 2026/01/22/ (future date, no data)
3. Result: Empty array [] is returned
4. In OptimizeTablePath() at line 578-582:

   if len(partitionPaths) == 0 {
       p.logger.Info().Msg("No data exists for time range, using fallback")
       return originalPath, false  // ← Returns s3://bucket/db/cpu/**/*.parquet
   }

5. Query falls back to scanning all files: s3://bucket/db/cpu/**/*.parquet
6. If the time range truly has no data, DuckDB still fails: "No files found that match the pattern"

Logs from Testing

INF Generated partition paths ... daily_paths=2 hourly_paths=48 total_paths=50
INF Filtered existing remote paths ... existing_count=0 original_count=50
INF No data exists for time range, using fallback
ERR Query failed error="No files found that match the pattern s3://arc-test/production/cpu/**/*.parquet"

Even though filtering works correctly, the fallback path still causes the query to fail.

Recommended Fix

You need to handle the case when len(existingPaths) == 0 before returning in filterExistingRemotePaths(). Here are two approaches:

Option 1: Return Sentinel Value (Recommended)

In filterExistingRemotePaths(), detect when all paths are filtered out and return a sentinel value to signal "no data":

func (p *PartitionPruner) filterExistingRemotePaths(paths []string) []string {
    // ... existing code ...

    existingPaths := []string{}
    for _, path := range paths {
        // ... your existing filtering logic ...
    }

    p.logger.Info().
        Int("existing_count", len(existingPaths)).
        Int("original_count", len(paths)).
        Msg("Filtered existing remote paths")

    // NEW: If all paths filtered out, return a special marker
    // This tells OptimizeTablePath to return empty result instead of fallback
    if len(existingPaths) == 0 {
        p.logger.Info().Msg("No partitions have data for the specified time range")
        return []string{"__EMPTY__"}  // Sentinel value
    }

    return existingPaths
}

Then in OptimizeTablePath(), check for the sentinel:

// Filter out non-existent paths
partitionPaths = p.filterExistingPaths(partitionPaths)

// NEW: Check if sentinel value returned (no data exists)
if len(partitionPaths) == 1 && partitionPaths[0] == "__EMPTY__" {
    p.logger.Info().Msg("No data exists for time range, will return empty result")
    p.partitionCache.set(cacheKey, []string{}, true)  // Cache empty result
    return []string{}, true  // Signal to query handler to return empty result
}

if len(partitionPaths) == 0 {
    p.logger.Info().Msg("No partitions found, using fallback")
    p.partitionCache.set(cacheKey, originalPath, false)
    return originalPath, false
}

Then in internal/api/query.go at the buildReadParquetExpr() functions (lines ~1477 and ~1511), add:

if pathList, ok := optimizedPath.([]string); ok {
    // NEW: Handle empty path list (no data in time range)
    if len(pathList) == 0 {
        h.logger.Info().Msg("No partitions for time range, returning empty result")
        // Return a query that produces an empty result with correct schema
        // This avoids DuckDB trying to read non-existent files
        return keyword + " (SELECT NULL AS time, NULL AS value LIMIT 0)"
    }

    // ... existing code for non-empty path lists ...
}

Option 2: Simpler Approach

Alternatively, just modify OptimizeTablePath() to not fall back when len(partitionPaths) == 0:

if len(partitionPaths) == 0 {
    p.logger.Info().Msg("No data exists for time range, will return empty result")
    p.partitionCache.set(cacheKey, []string{}, true)
    return []string{}, true  // Don't fall back - signal empty result
}

And handle the empty array in buildReadParquetExpr() as shown above.

Test Case to Add

Please add a test case in internal/pruning/partition_pruner_test.go:

func TestFilterExistingRemotePaths_NoDataInTimeRange(t *testing.T) {
    // Scenario: Query time range that has no data (future dates, or before ingestion)
    // Expected: Query should return empty result gracefully, not fail with "No files found"

    // Setup: Storage with only one hourly partition
    // Query: Time range that excludes that partition
    // Expected: filterExistingRemotePaths returns empty array
    // Expected: Query returns 0 rows instead of error
}

Summary

Your PR fixes the day-level file existence check, which is great! But it needs to also handle the empty result case where all paths are filtered out (no data in time range).

The fix requires changes in three places:

1. filterExistingRemotePaths() - Your PR already improves this
2. OptimizeTablePath() - Need to handle len(partitionPaths) == 0 better
3. buildReadParquetExpr() in internal/api/query.go - Need to handle empty path list

Would you like to update your PR to include these changes? I'm happy to help if you have questions!

Testing

To reproduce the issue:

1. Start MinIO: docker run -p 9000:9000 -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin quay.io/minio/minio server /data
2. Configure Arc with S3 storage (MinIO endpoint)
3. Ingest data for today only
4. Query: SELECT * FROM cpu WHERE time >= '2026-01-20' AND time < '2026-01-22' (includes tomorrow)
5. Before full fix: Query fails with "No files found"
6. After full fix: Query returns 0 rows (empty result)

---

Thank you again for your contribution! The file-level verification logic you added is exactly the right approach. We just need to extend it to handle the edge case where no files exist at all in the queried time range.