Row lineage #4080

Feature Request / Improvement

We want to be able to read the row-lineage values on a table. This is a blocker for writing v3 tables.

table.scan(selected_fields=("id", "_row_id", "_last_updated_sequence_number"))

• Data files with a null first_row_id inherit it from the manifest's first_row_id plus a running record count: Add row-lineage support with Spark tests #3972
• Define the reserved columns _row_id (2147483540) and _last_updated_sequence_number (2147483539)
• Let scans project them
• Use the values stored in the file if there are any. Fill nulls with first_row_id + pos and the file's data sequence number.
• Compute pos before applying deletes
• Show first_row_id in inspect.files()
• REST scan planning drops first_row_id
• Integration test: each file's first_row_id matches Spark's MIN(_row_id): Add row-lineage support with Spark tests #3972
• Integration test: Spark MERGE / UPDATE / rewrite_data_files, then compare with Spark's SELECT _row_id, _last_updated_sequence_number