Repository navigation
Partition pruning fails for queries with newlines in SQL #146
Description
Activity
Thank you for the detailed bug report and the fix, @khalid244! You're absolutely correct.
Confirmed Bug
The whereClausePattern regex uses .+? which does NOT match newlines in Go's regex engine. This causes partition pruning to fail for any multi-line SQL query.
Current code (partition_pruner.go:116):
whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+(.+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)The .+? pattern stops at newlines, so when your query has:
SELECT region, COUNT(*)
FROM metrics
WHERE time >= '2026-01-21T07:00:00Z' AND time < '2026-01-21T08:00:00Z'
GROUP BY regionThe regex fails to capture the WHERE clause content, ExtractTimeRange() returns nil, and partition pruning is disabled entirely.
Your Fix is Correct
Changing .+? to [\s\S]+? is the right solution:
whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+([\s\S]+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)The [\s\S] pattern explicitly matches any character including newlines, which fixes the issue.
Would you like to submit a PR?
This is a simple one-line fix that would be perfect for a PR! If you'd like to submit it:
-
Change line 116 in
internal/pruning/partition_pruner.go:whereClausePattern = regexp.MustCompile(`(?i)\bWHERE\b\s+([\s\S]+?)(?:\bGROUP BY\b|\bORDER BY\b|\bLIMIT\b|$)`)
-
Add a test case in
internal/pruning/partition_pruner_test.goto prevent regression:func TestExtractTimeRange_MultilineQuery(t *testing.T) { sql := `SELECT region, COUNT(*) FROM metrics WHERE time >= '2026-01-21T07:00:00Z' AND time < '2026-01-21T08:00:00Z' GROUP BY region` pruner := NewPartitionPruner(config.PartitionPrunerConfig{Enabled: true}, nil, zerolog.Nop()) timeRange := pruner.ExtractTimeRange(sql) require.NotNil(t, timeRange, "Should extract time range from multi-line query") assert.Equal(t, "2026-01-21T07:00:00Z", timeRange.Start.Format(time.RFC3339)) assert.Equal(t, "2026-01-21T08:00:00Z", timeRange.End.Format(time.RFC3339)) }
-
Run tests to verify no regressions:
go test ./internal/pruning/...
Alternatively, I can apply this fix directly to the codebase if you prefer. Let me know!
Impact
This fix will:
- Enable partition pruning for multi-line SQL queries (Grafana, BI tools, formatted queries)
- Reduce S3 costs by avoiding full table scans
- Improve query performance significantly
- No breaking changes or regressions
Great catch on this bug! Multi-line queries are very common in practice (Grafana dashboards, SQL editors, formatted scripts), so this fix will have significant impact.
Done #148
Summary
The partition pruner's WHERE clause regex pattern fails to match across newlines, causing partition pruning to be skipped entirely for multi-line SQL queries. This results in full table scans (
/**/*.parquet) instead of targeted partition paths.Impact
Root Cause
The
whereClausePatternregex ininternal/pruning/partition_pruner.gouses.+?which does not match newlines in Go regex:When a query has newlines between the WHERE clause and GROUP BY/ORDER BY:
The regex fails to capture the WHERE clause content,
ExtractTimeRange()returns nil, and partition pruning is skipped.Debug Logs
Suggested Fix
Change
.+?to[\s\S]+?which explicitly matches any character including newlines: