Panic: index out of range in sliceColumnsByIndices during high concurrency writes

[khalid244]

Panic: index out of range in sliceColumnsByIndices during high concurrency writes

Environment

• Arc version: 26.01.2
• Kubernetes, single arc-write pod

Reproduce

1. Send high volume writes to /api/v1/write/line-protocol
2. 48 concurrent workers, 5000-row batches
3. Crash occurs intermittently during flush

Error

panic: runtime error: index out of range [20000] with length 20000

goroutine 82 [running]:
github.com/basekick-labs/arc/internal/ingest.sliceColumnsByIndices(...)
	/build/internal/ingest/arrow_writer.go:2259
github.com/basekick-labs/arc/internal/ingest.(*ArrowBuffer).flushPartitionedData(...)
	/build/internal/ingest/arrow_writer.go:1775

Root Cause

Race condition in flushBufferLocked() (arrow_writer.go:1817-1870):

// Line 1851: Lock released before merge
shard.mu.Unlock()

// Line 1854: Merge batches creates column arrays
merged, err := b.mergeBatches(recordsToFlush)

// Line 1862: Flush uses these arrays
b.flushBufferLockedDataTime(...)

In flushPartitionedData():

• groupByHour(times) creates indices 0..N-1 based on times length
• sliceColumnsByIndices(merged, bucket.indices) accesses other columns using these indices
• If another column has different length due to race → panic

Suggested Fix

Add bounds check in sliceColumnsByIndices():

for i, idx := range indices {
    if idx >= len(col) {
        return nil, fmt.Errorf("index %d out of bounds for column %s (len=%d)", idx, colName, len(col))
    }
    newCol[i] = col[idx]
}

Or validate all columns have equal length after mergeBatches().

[khalid244]

Additional Root Cause: Sparse Columns (Non-Race Condition)

Encountered this same panic during a database migration with a single writer. The issue can occur without concurrency when ingesting data with inconsistent column presence across rows.

Reproduction

Migration script that conditionally includes tags/fields based on NULL values:

for col in TAG_COLUMNS:
    val = row_dict.get(col)
    if val is not None:
        sanitized = sanitize_string(val)
        if sanitized:  # Only added if non-empty
            point = point.tag(col, sanitized)

This produces line protocol where each row may have different columns present.

Why This Causes the Panic

When mergeBatches() processes batches with sparse columns:

Row	time	sensor_id	location	value
1	✓	✓	NULL	✓
2	✓	✓	✓	NULL
3	✓	NULL	✓	✓

Result after merge:

• time column: length 3
• sensor_id column: length 2
• location column: length 2
• value column: length 2

Then in flushPartitionedData():

1. groupByHour(times) generates indices [0, 1, 2] based on time column
2. sliceColumnsByIndices() tries to access index 2 on columns with length 2 → panic

Logs Before Crash

{"level":"warn","message":"Data timestamp is >7 days old - possible backfill or clock skew"}
{"level":"info","num_hours":9,"total_records":50000,"message":"Splitting batch across multiple hour partitions"}
{"level":"info","records":5000,"message":"Wrote hour partition"}
panic: runtime error: index out of range [45000] with length 45000

The ~10% difference (45000 vs 50000) corresponds to rows missing that column.

Suggested Fix

Handle sparse columns by padding with zero-values during merge in mergeBatches():

// After PHASE 1 (calculating sizes), determine max row count
maxRows := 0
for _, info := range sizes {
    if info.totalSize > maxRows {
        maxRows = info.totalSize
    }
}

// In PHASE 2, allocate all columns to maxRows length
for name, info := range sizes {
    switch info.colType {
    case "int64":
        merged[name] = make([]int64, maxRows)    // Zero-filled by default
    case "float64":
        merged[name] = make([]float64, maxRows)  // Zero-filled by default
    case "string":
        merged[name] = make([]string, maxRows)   // Empty string by default
    case "bool":
        merged[name] = make([]bool, maxRows)     // false by default
    }
}

This ensures all columns have consistent length regardless of sparse input data. Go's zero-values (0, 0.0, "", false) provide sensible defaults for missing data.

[xe-nvdk]

Thanks for reporting this! After extensive investigation, I've identified and fixed the root cause.

Investigation Findings

Your suggested fix (recovering from panic) would have masked the real issue - the data would have been silently dropped. Good thing we dug deeper!

The Real Root Cause: Sparse Column Handling

The panic isn't caused by a race condition in lock handling (that part is actually correct). The issue is in how mergeBatches() handles schema evolution during high-concurrency writes.

When batches have different column sets:

Batch 1: {time: [1,2,3], cpu: [10,20,30]}           // 3 rows
Batch 2: {time: [4,5], temperature: [70,80]}        // 2 rows (new column!)
Batch 3: {time: [6,7,8], cpu: [40,50,60]}           // 3 rows

After merge:
  time:        [1,2,3,4,5,6,7,8]  // 8 elements (correct)
  cpu:         [10,20,30,40,50,60] // 6 elements (SPARSE!)
  temperature: [70,80]            // 2 elements (SPARSE!)

When groupByHour() generates indices from the time column ([0,1,2,3,4,5,6,7]) and then sliceColumnsByIndices() tries to access those indices on the shorter columns... PANIC: index out of range.

The Fix (commit 0e3e05b)

1. mergeBatches(): Now normalizes all columns to the same length (totalRows from time column), filling sparse positions with zero values

2. sliceColumnsByIndices(): Added defensive bounds checking as an additional safety layer

3. Added tests to reproduce the scenario and verify the fix

Branch: fix/arrow-writer-sparse-columns-panic
PR coming shortly for review.

This should completely resolve the panic during high-concurrency writes with schema evolution. No silent data loss, no recover() needed - just proper sparse column handling.