Skip to content

Compaction temp-dir cleanup can os.RemoveAll a concurrently running job's working directory #749

Description

@xe-nvdk

Found while working #744. This outranks everything listed there.

internal/compaction/manager.go:557:

partitionPrefix := sanitizeDBForName(candidate.Database) + "_" +
    strings.ReplaceAll(candidate.PartitionPath, "/", "_") + "_"
...
if entry.IsDir() && strings.HasPrefix(entry.Name(), partitionPrefix) {
    os.RemoveAll(dirPath)
}

_ is both the field separator and the / replacement, and it is legal in database and measurement names. PartitionPath already begins with the database, so the database is folded in twice. Two ordinary names therefore produce one prefix:

database measurement partition path prefix
a a_a_cpu a/a_a_cpu/2026/09/12/14 a_a_a_a_cpu_2026_09_12_14_
a_a cpu a_a/cpu/2026/09/12/14 a_a_a_a_cpu_2026_09_12_14_

No edge-sync pseudo-database is needed; both names pass isValidDatabaseName and isValidMeasurementName.

Why it matters

Partitions run MaxConcurrent at a time within a tier (manager.go:929, manager.go:1008). So when one job finishes, this sweep deletes the other job's live temp directory, which holds its downloaded source files and its in-progress compacted output. The victim job then fails or produces a truncated result, and the sweep is not guarded by anything: HasPrefix on a folded name, then os.RemoveAll.

The _b{batch} and nanosecond components that make JobID unique do not help, because the prefix truncates before them.

Related, currently defused

A daily prefix such as prod_prod_cpu_2026_09_12_ is a prefix of every hourly temp directory for that day, so a daily job would sweep all 24 hourly working directories. That one is only defused because tiers run sequentially under m.cycleRunning (manager.go:825, manager.go:901). It is one scheduling change away from live.

Suggested fix

Stop deriving the sweep key from a folded string. The job already has an unambiguous identity, JobID, and the temp directory is created from it. Sweep by exact directory name, or record the created path and remove that, rather than prefix-matching a lossy encoding.

If prefix matching has to stay, the separator must be a byte that cannot appear in the encoded components, and the database must not be folded in twice.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions