Skip to content

Partition pruning fails for queries with newlines in SQL #146

Description

@khalid244

Summary

The partition pruner's WHERE clause regex pattern fails to match across newlines, causing partition pruning to be skipped entirely for multi-line SQL queries. This results in full table scans (/**/*.parquet) instead of targeted partition paths.

Impact

  • Queries that should scan a few hourly partitions end up scanning the entire dataset
  • Increased S3/storage costs due to unnecessary data transfer

Root Cause

The whereClausePattern regex in internal/pruning/partition_pruner.go uses .+? which does not match newlines in Go regex:

whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+(.+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)

When a query has newlines between the WHERE clause and GROUP BY/ORDER BY:

SELECT region, COUNT(*)
FROM metrics
WHERE time >= '2026-01-21T07:00:00Z' AND time < '2026-01-21T08:00:00Z'
GROUP BY region

The regex fails to capture the WHERE clause content, ExtractTimeRange() returns nil, and partition pruning is skipped.

Debug Logs

No time range found in query, skipping partition pruning
Executing query converted_sql="... FROM read_parquet('s3://bucket/db/table/**/*.parquet'..."

Suggested Fix

Change .+? to [\s\S]+? which explicitly matches any character including newlines:

whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+([\s\S]+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)

Activity

added theissue type on Jan 21, 2026

xe-nvdk commented on Jan 21, 2026

@xe-nvdk
Member

Thank you for the detailed bug report and the fix, @khalid244! You're absolutely correct.

Confirmed Bug

The whereClausePattern regex uses .+? which does NOT match newlines in Go's regex engine. This causes partition pruning to fail for any multi-line SQL query.

Current code (partition_pruner.go:116):

whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+(.+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)

The .+? pattern stops at newlines, so when your query has:

SELECT region, COUNT(*)
FROM metrics
WHERE time >= '2026-01-21T07:00:00Z' AND time < '2026-01-21T08:00:00Z'
GROUP BY region

The regex fails to capture the WHERE clause content, ExtractTimeRange() returns nil, and partition pruning is disabled entirely.

Your Fix is Correct

Changing .+? to [\s\S]+? is the right solution:

whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+([\s\S]+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)

The [\s\S] pattern explicitly matches any character including newlines, which fixes the issue.

Would you like to submit a PR?

This is a simple one-line fix that would be perfect for a PR! If you'd like to submit it:

  1. Change line 116 in internal/pruning/partition_pruner.go:

    whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+([\s\S]+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)
  2. Add a test case in internal/pruning/partition_pruner_test.go to prevent regression:

    func TestExtractTimeRange_MultilineQuery(t *testing.T) {
        sql := `SELECT region, COUNT(*)
    FROM metrics
    WHERE time >= '2026-01-21T07:00:00Z' AND time < '2026-01-21T08:00:00Z'
    GROUP BY region`
    
        pruner := NewPartitionPruner(config.PartitionPrunerConfig{Enabled: true}, nil, zerolog.Nop())
        timeRange := pruner.ExtractTimeRange(sql)
    
        require.NotNil(t, timeRange, "Should extract time range from multi-line query")
        assert.Equal(t, "2026-01-21T07:00:00Z", timeRange.Start.Format(time.RFC3339))
        assert.Equal(t, "2026-01-21T08:00:00Z", timeRange.End.Format(time.RFC3339))
    }
  3. Run tests to verify no regressions:

    go test ./internal/pruning/...

Alternatively, I can apply this fix directly to the codebase if you prefer. Let me know!

Impact

This fix will:

  • Enable partition pruning for multi-line SQL queries (Grafana, BI tools, formatted queries)
  • Reduce S3 costs by avoiding full table scans
  • Improve query performance significantly
  • No breaking changes or regressions

Great catch on this bug! Multi-line queries are very common in practice (Grafana dashboards, SQL editors, formatted scripts), so this fix will have significant impact.

khalid244 commented on Jan 21, 2026

@khalid244
Author

Done #148

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions