Various fields in Manifest & ManifestList not written in accordance to the Spec

[JFinis]

Apache Iceberg version

1.3.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

I created some Icebergs with the latest Iceberg/Spark and checked whether the schemas of the generated Manifest files and Manifest Lists are in accordance to the Spec.

This issue also is based on the discussions in this PR and this Slack Thread, where we discussed what the semantics should be, if a field is labeled optional by the spec:

So far, the outcome of the discussion is that if a field is labeled optional for an Iceberg format version, then writers writing that format version should include a column for that field in the Avro file but tag that column optional (i.e., nullable, i.e.,[Null, T] in Avro). They should not just leave out the column.

This Issue contains all deviations from the Spec I could find. All deviations found were in Icebergs created with Spark 3.4.1 using Iceberg 1.3.1 (latest release).

In a format_version=2 Manifest List:

• column key_metadata is not written at all, even though the Spec tags it as optional in v2 (and v1).
• The fields added_files_count, existing_files_count, deleted_files_count are not named correctly. They have an additional data_ infix. This was already reported separately in Issue 8684, but I include it here as well for the sake of completeness.

In a format_version=1 Manifest File:

• column file_ordinal is not written at all, even though the Spec tags it as optional in v1 (it is considered deprecated though)
• column sort_columns is not written at all, even though the Spec tags it as optional in v1 (it is considered deprecated though)
• [same as v2] column distinct_counts is not written at all, even though the Spec tags it as optional in v1 (and v2).

In a format_version=1 Manifest List:

• The position of the column 507 partitions in the manifest_entry struct is different than in the Spec:
  • In the spec, it is placed behind 514 deleted_rows_count
  • In the file, it is placed behind 506 deleted_files_count
• The field added_snapshot_id has type ["null","long"], but the spec says it is required, so the type should just be "long".
• [same as v2] column key_metadata is not written at all, even though the Spec tags it as optional in v2 (and v1).
• [same as v2] The fields added_files_count, existing_files_count, deleted_files_count are not named correctly. They have an additional data_ infix. This was already reported separately in Issue 8684, but I include it here as well for the sake of completeness.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'