Skip to content

Tiered storage: X-Arc-Database header queries fail for cold-only data #166

Description

@xe-nvdk

Bug Description

When data has been migrated to cold storage via tiered storage, queries using the X-Arc-Database header fail because the query path still resolves to local (hot) storage instead of checking tiered storage metadata.

Symptoms

  1. X-Arc-Database header queries fail for cold-only data:

    POST /api/v1/query
    X-Arc-Database: benchmark_logs
    
    SELECT * FROM logs LIMIT 100;
    

    Error: IO Error: No files found that match the pattern "/Users/.../data/arc/benchmark_logs/logs/**/*.parquet"

  2. Qualified name queries work:

    SELECT * FROM benchmark_logs.logs LIMIT 10;

    ✅ Works correctly, returns data from cold tier

  3. Database listing doesn't show cold-only databases:

    • GET /api/v1/databases only returns databases with hot tier data
    • Databases migrated entirely to cold storage are invisible

Root Cause Analysis

Issue 1: Fast path bypasses tiering metadata

The optimized fast path for header-based queries (convertSingleTableQuery()) calls getStoragePath() which returns the hot tier path directly, before tiering metadata can be consulted.

Flow comparison:

X-Arc-Database header path (BROKEN):
  convertSQLToStoragePathsWithHeaderDB()
  → convertSingleTableQuery() [fast path]
  → h.getStoragePath() [returns HOT TIER ONLY]
  → buildReadParquetExpr() [tiering check happens too late]
  → ❌ Fails for cold-only data

Qualified name path (WORKS):
  convertSQLToStoragePaths() [regex-based]
  → Extracts database/measurement before path resolution
  → buildReadParquetExpr() [proper tiering check]
  → buildMultiTierReadParquet()
  → ✅ Works for cold tier data

Issue 2: Database listing only scans hot tier

DatabasesHandler.listDatabasesWithMeasurementCounts() only queries the hot tier storage backend. It should also query tiering metadata to include cold-only databases.

Key Code Locations

Issue File Function
Fast path tiering bypass internal/api/query.go convertSingleTableQuery()
Database listing missing cold tier internal/api/databases.go listDatabasesWithMeasurementCounts()
Missing tiering integration internal/api/databases.go DatabasesHandler struct

Expected Behavior

  1. Queries with X-Arc-Database header should check tiering metadata and route to the appropriate tier
  2. GET /api/v1/databases should return all databases, including those with data only in cold storage
  3. Both query syntaxes should work identically for tiered data

Logs

2026-01-26T11:38:23-03:00 INF Migration cycle completed component=tiering-manager migrated=66
2026-01-26T11:39:21-03:00 ERR Query failed error="IO Error: No files found that match the pattern..."

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions