…async (#314, #337, #305) (#565) Three independent bug fixes closing out the 26.09.1 bug sprint. #314 -- ListManifests discarded every storage error and returned an empty list, with no log line. "No manifests exist" and "we could not find out" are opposite instructions: an empty set means nothing is being compacted and the candidate may proceed. filterCandidateFiles already had the right guard -- it skips the partition when the lookup fails, explicitly "to avoid re-compaction" -- but swallowing the error made that guard unreachable, so a transient S3 throttle or credential blip let Arc re-compact files another job already had in flight. Errors now propagate. A missing manifest directory stays a non-error: LocalBackend.List skips absent directories and the object stores return an empty listing for an empty prefix, so anything reaching this layer is a real failure. #337 -- a column that was entirely null within a batch was dropped, since no type can be inferred from it. Readers mostly absorbed this because they union schemas by name, but a column null in EVERY batch never appeared in any file, so querying it failed to bind rather than returning NULLs. Verified against the running binary: pre-fix the query returns `Binder Error: Referenced column "depth" not found`; after, three NULL rows. Such columns are now written as an all-null placeholder with a validity mask, and a later batch with real values still infers its own type. time is exempt and an all-null time is rejected outright -- a VARCHAR time column makes a partition un-compactable. #305 -- wal.sync_mode accepted fdatasync and defaults to it, but both fsync and fdatasync called the same full Sync(). Linux now uses a real syscall.Fdatasync, retried on EINTR so an interrupted call cannot report durability it did not achieve. Go does not expose fdatasync on darwin or windows, so those fall back to Sync() via build-tagged files and now log that fact once at startup instead of reporting a mode they do not perform. Verified by running the Linux test binary in a container, not just cross-compiling it. Expect no throughput change from #305: WAL syncs run off a 100ms ticker, not per-write, so this is a correctness and honesty fix. Also closes #293 and #294 as obsolete -- both describe ResilientBackend, deleted in #561. Co-authored-by: Ignacio Van Droogenbroeck <ignacio@vandroogenbroeck.net>

medium(ingest): convertColumnsToTyped silently drops all-nil columns #337

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions