Remove FILE self-references

[alkis]

Remove self-references from the FILE specification:

• offset and size apply only to data referenced by uri
• a byte range within the current file cannot be referenced
• drop the compression and encryption semantics this PR previously specified, so Encryption.md returns to its state on master
• leave inline values and external references unaffected

A large value that should live elsewhere in the same file is better served by out-of-band pages, which can hold it outside the page data without the FILE type defining its own compression, encryption, and addressing rules. That will be proposed separately.

This supersedes the earlier direction of this PR, which specified storage inheritance for self-references. The review threads on that approach are left in place for history — see in particular the AAD identity discussion, which is what motivated moving this into the page layer instead.

Discussion: https://lists.apache.org/thread/zrzc7t9fccg92rx3h4fw3ndw3bdo5xr7

[alkis]

Change of direction: this PR now removes FILE self-references instead of specifying their storage semantics (d022d27).

The AAD discussion is what convinced me. Making a self-referenced range a first-class encryption module meant the FILE type had to define its own compression block, its own encryption module, its own AAD identity, and its own size accounting — a parallel storage mechanism living next to pages but not being one. Every review comment here was a symptom of that: whether the range is bound to a page, what its identity is for AAD, whether the size is pre- or post-encryption, whether the decompressed size needs storing. Those questions only exist because the bytes sit outside any page while still needing page-like machinery.

Out-of-band pages answer all of them once, in the page layer, for every type rather than just FILE. A large value gets stored in a page that lives outside the normal page sequence, and it inherits compression, encryption, AAD, and size accounting from the existing page rules with nothing new to specify. I will send that as a separate proposal.

What this PR does now:

• offset and size apply only to data referenced by uri
• a byte range within the current file cannot be referenced
• Encryption.md returns to its state on master
• inline values and external references are unchanged

@rok @wgtmac @danielcweeks @brkyvz — the open threads all point at text that no longer exists, so nothing there needs an answer from you. I have left them unresolved rather than resolving them myself, since the reasoning in them is the reason for this change and worth keeping visible. The parts of your feedback that survive independently of self-references are already in: the aligned resolution table (@brkyvz) and the normative size wording (@danielcweeks).

[etseidl]

I like removing the self-reference, thanks @alkis.

[etseidl]

What if uri is the current file?

[wgtmac]

That sounds like hacking self-reference

[emkornfield]

I think from an Engine's perspective it doesn't really change anything, it is all handled uniformly? From the writers perspective I guess they could do this but it is up to them to figure out the semantics, I don't think we need to specify anything in parquet.

[etseidl]

My point is that

A byte range within the current file cannot be referenced

implies self-reference is disallowed, but the uri hack is seemingly still possible. It's these types of grey areas in the spec that always get us in trouble. I'd be more comfortable either explicitly disallowing self reference by saying uri cannot reference the current file, or discuss the possibility of self reference and the ramifications (no encryption, no compression), and perhaps retain the uri unset/offset+size set form of self reference.

[alkis]

Good point, and I agree the previous wording created exactly the grey area you describe. Fixed in f4b79fe.

I did not take either of your two options though, because I think a ban on uri naming the current file is not checkable: relative URIs, copies, renames, and object-store aliases all defeat it, and a file that was legal when written becomes illegal when someone copies it. Neither the writer nor the reader can verify the rule, so it would be a grey area of a different kind.

Instead the external path is now uniform, which is what @emkornfield was getting at:

offset and size apply only to data referenced by uri; there is no form that addresses a byte range in the current file directly.

A uri is always resolved as an external reference, even when it names the file that contains it. Parquet applies no compression or encryption of its own to the referenced bytes, and a reference remains the writer's responsibility if the file is copied or renamed.

So the uri form is specified rather than left ambiguous, and it is specified as the same thing in all cases: no inheritance, no Parquet-applied transforms, no special handling. A writer pointing a uri at its own file gets external-reference semantics and owns the consequences. That is not a self-reference back door (@wgtmac) because nothing about it is privileged.

I also chose not to retain the uri-unset form. Keeping it would mean re-specifying the compression and encryption ramifications, which is what this PR is removing, and the mid-sized-value use case it existed for is better served by non-contiguous pages in the page layer. More on that in a follow-up proposal.

[wgtmac]

I'm fine with removing self-reference. Since this is a spec change after previous vote, perhaps we need yet another vote for this?

[wgtmac]

Do we allow both inline and uri (as well as other fields) are set?

[rok]

Would inline be like a preview slice of the bytes at URI?

[alkis]

Yes, allowed — specified in f4b79fe. This was a real gap: the text said locator fields alongside inline are "provenance only" but the resolution table did not cover the combination, so it was underspecified.

@rok no, not a preview slice. When both are set they must denote the same bytes; the locator records where those bytes came from. Anything else would give a value two different meanings depending on which representation you read.

The table row is now:

inline	uri	offset	size	Resolves to
set	any	any	any	the inline bytes

with the inline field description saying the two must denote the same bytes and that a locator must not be a partial or otherwise different representation, and a matching validation bullet. inline always wins for reading, so a reader never has to reconcile the two.

[rok]

@rok no, not a preview slice. When both are set they must denote the same bytes; the locator records where those bytes came from. Anything else would give a value two different meanings depending on which representation you read.

Ok that makes sense, just thinking out loud that having an inlined slice of data that is in full available at the uri could be interesting for filtering. But given this is possible on application level it's probably better to leave spec out of this.

[alkis]

Following up on this — the previous wording said inline "supplies the bytes for readers" without saying whether reading the locator instead is a legitimate path. It has to be: a reader that projects uri/offset/size and not inline cannot tell whether inline was set, so it would be out of spec through no fault of its own. Made explicit in 5facb7f:

Both representations must denote the same bytes, so a reader may resolve the value from either and obtain the same result; reading inline requires no external access and is the cheaper path.

And the validation bullet now says what happens if a producer breaks the rule:

If inline and a locator are both set, they must denote the same bytes, and a reader may resolve the value from either. If they disagree the value is invalid; a reader is not required to detect this and may return the bytes of either representation.

So the file is at fault when they disagree, and readers are not obliged to verify.

One asymmetry worth noting: checksum with ETAG is scoped to the whole file referenced by uri, not to the resolved bytes, so an ETAG alongside inline describes the external object rather than the inline payload. That is the existing exception in the checksum section and I have left it as is.

[alkis]

Agreed on both counts. An inline prefix of a larger object at uri is a genuinely useful pattern for filtering, but it needs its own field to say "this is a prefix, not the value" — overloading inline would make every reader guess which it is. An application can do it today with a separate column, and if there is demand later it can be a new optional field with explicit semantics rather than a reinterpretation of this one.

[alkis]

Updating my earlier answer to you here, since it no longer holds. Per @RussellSpitzer above, the same-bytes requirement is gone in 17aa46b because nothing can verify it.

So the answer to "would inline be a preview slice" is no longer "invalid" but "unspecified, and a reader may resolve from either, so do not rely on it" — a reader that happens to read inline would get your slice while one that reads the uri gets the full object. Same practical advice, weaker footing. Your original point stands: if a prefix-for-filtering pattern is wanted, it should be its own field with explicit semantics rather than a reinterpretation of inline.

[wgtmac]

That sounds like hacking self-reference

[rok]

I like the simplification!
Minor comment about encryption.
Re: compression I suppose without ranges it's pretty straight forward.

[rok]

But the physical column data can be encrypted as per modular encryption spec? Shall we state this explicitly if so?

[alkis]

Yes, and worth stating explicitly — the neighbouring sentence about external files being out of scope invites the wrong inference. Added in f4b79fe:

Encryption of external files referenced by uri is outside the scope of the Parquet format. The fields of a FILE-annotated group are ordinary columns and are encoded, compressed, and encrypted like any other column, inline included.

So the two are separate concerns: the columns holding the reference get full modular encryption like any other column, while the bytes sitting in some external object store are not something Parquet can speak to. inline is called out because it is the case where the payload itself lives in a Parquet column and therefore is covered.

[rok]

Perhaps a later spec change could add key_id etc for the buffer behind the URI. Out of scope.

[alkis]

Yes, out of scope here. Worth noting the reason it is a bigger change than it looks: a key for the bytes behind a uri means Parquet describing the encryption of data it does not own and cannot re-encrypt on rewrite, so key rotation and compaction both need answers. Happy to look at it as a separate proposal if you want to write it up.

[rok]

I'm not a fan of process, but self-references and byte ranges were formally voted on and approved as part of #585. This PR now removes that functionality and we should probably call a formal vote for it.

edit: I see @wgtmac already proposed this.

[rok]

Parquet.thrift should also be updated to reflect removal of self-reference, currently it states:

 * Annotates a group that represents a reference to a file, or to a range of
 * bytes that may be stored inline, elsewhere in this file, or in an external
 * file.

[alkis]

Thanks for catching parquet.thrift — I had only looked at the two Markdown files. Updated in ed8bb0b:

 * Annotates a group that represents a reference to a file, or to a range of
 * bytes that may be stored inline or in an external file.

I checked the rest of the file and that comment was the only place self-references were mentioned outside LogicalTypes.md and Encryption.md. No Thrift structures change in this PR.

[RussellSpitzer]

Slight new change here? I I think this is an interesting semantic but also i'm not sure we need to be putting it in the spec?

[alkis]

Yes, new — it came in during review (f4b79fe, 5facb7f) after @wgtmac asked whether inline and uri may both be set, which the merged text left underspecified. Agreed the semantic was more than the spec needs, and dropped in 17aa46b. The paragraph is now:

The referenced bytes stored inline in the value. If inline is set, it supplies the bytes and any locator fields (uri, offset, size) that are set record where those bytes came from. A reader may resolve the value from inline or from the locator, whichever suits it; inline requires no external access and is the cheaper path.

So it states what a reader may do and nothing about what the two representations must contain. I kept that much because a reader that projects uri/offset/size without inline cannot tell whether inline was set, so it needs to know reading the locator is a legitimate path.

[rok]

inline requires no external access and is the cheaper path.

Is the cheaper path comment required here? I don't disagree with it, it just doesn't feel necessary.

[alkis]

Agreed, it was advisory rather than normative. Removed in 1125912, so the paragraph ends:

A reader may resolve the value from inline or from the locator, whichever suits it.

The rule is the same and the cost difference is obvious from one path touching the network.

[RussellSpitzer]

I'm not sure this is something that can really be guaranteed in any system. I think the second part of this, "A user is allowed to read from either the URI or inline" is fine as an detail to folks using the format.

[alkis]

Agreed, and this is the same argument I made against banning uri from naming the current file earlier in this PR — a rule nothing can check does not belong in the spec. Nothing verifies that the bytes behind a uri match inline without fetching and comparing them. Taking your split in 17aa46b: the must/invalid language is gone, the permission stays.

If inline and a locator are both set, a reader may resolve the value from either. Producers are expected to write the same bytes in both, but a reader is not required to check this and may return the bytes of either. Producers may treat inline and the locator fields as mutually exclusive.

Worth being explicit about the consequence: two conforming readers of the same file can now return different bytes for one value and neither is wrong. That is the honest outcome of not enforcing it, and the previous wording only pretended otherwise since it also said readers need not check. Happy to cut the "Producers are expected to" sentence too if you would rather the bullet carry no writer guidance at all.