Discovered during architecture review of #76 (feat: primary-key upsert/delete + ALTER TABLE schema evolution). Follow-up: not blocking the merge of #76 / #77 but should be addressed before this DELETE path stabilizes.
Context
LanceUpsertSink.buildDeletePredicate translates DELETE events into an
OR-of-AND SQL string fed to Dataset.delete(String). Pragmatic for the
initial PR, not the right long-term shape.
Problems
- Type coverage gap.
RowDataFieldAccessor supports VarBinaryType
but formatSqlValue does not; DATE / TIME / TIMESTAMP /
TIMESTAMP_LTZ / DECIMAL are unsupported on both sides.
- Injection surface. Column names are inlined without escaping. Any
name that collides with a Lance SQL keyword or contains parser-special
characters is a latent bug.
- Formatting fragility. Non-finite floats (
NaN, Infinity) had to
be rejected explicitly. Any future support for BigDecimal would need
bespoke formatting; the current dispatch is fragile.
- Predicate length blow-up.
N deleted keys → N OR-joined
parenthesized triples. Large batches produce arbitrarily long SQL
strings, pressuring the Lance parser and observability tooling.
Proposed direction (preference order)
mergeInsert with WhenMatched.Delete if the Lance Java SDK
exposes it — hand Lance a RecordBatch of the primary-key projection,
let it emit the tombstones. No string round-trip.col IN (list-encoded batch) for single-column PK. O(N) growth
with a smaller constant, and a well-understood parser shape.- Otherwise: extract encoding into a tested
PrimaryKeyEncoder class,
share type dispatch with RowDataFieldAccessor so read/delete sides
cannot drift.
Acceptance criteria
Refs
Context
LanceUpsertSink.buildDeletePredicatetranslates DELETE events into anOR-of-AND SQL string fed to
Dataset.delete(String). Pragmatic for theinitial PR, not the right long-term shape.
Problems
RowDataFieldAccessorsupportsVarBinaryTypebut
formatSqlValuedoes not;DATE/TIME/TIMESTAMP/TIMESTAMP_LTZ/DECIMALare unsupported on both sides.name that collides with a Lance SQL keyword or contains parser-special
characters is a latent bug.
NaN,Infinity) had tobe rejected explicitly. Any future support for
BigDecimalwould needbespoke formatting; the current dispatch is fragile.
Ndeleted keys →NOR-joinedparenthesized triples. Large batches produce arbitrarily long SQL
strings, pressuring the Lance parser and observability tooling.
Proposed direction (preference order)
mergeInsertwithWhenMatched.Deleteif the Lance Java SDKexposes it — hand Lance a
RecordBatchof the primary-key projection,let it emit the tombstones. No string round-trip.
col IN (list-encoded batch)for single-column PK.O(N)growthwith a smaller constant, and a well-understood parser shape.
PrimaryKeyEncoderclass,share type dispatch with
RowDataFieldAccessorso read/delete sidescannot drift.
Acceptance criteria
RowDataFieldAccessor#readFieldare supported endto end.
NULLand non-finite float PK values continue to be rejected withclear errors (regression from fix(sink): resolve architecture-review must-fix items for upsert sink (#76 follow-up) #77).
Refs