Docs: Document all metadata tables. #8709
Conversation
|
@nk1506: I think it is hard to review. Can you please keep the scope of this PR to add the missing metadata tables? Rearranging can be done in a followup PR. |
docs/spark-queries.md
Outdated
| ``` | ||
|
|
||
| | status | snapshot_id | sequence_number | file_sequence_number | data_file | readable_metrics | | ||
| |--------| -- |-----------------|----------------------| -- | -- | |
There was a problem hiding this comment.
this style is not same as existing tables.
There was a problem hiding this comment.
same comment for all new tables added.
| @@ -268,6 +268,18 @@ order by made_current_at | |||
| | 2019-02-09 16:32:47.336 | append | 57897183625154 | true | application_1520379288616_155055 | | |||
| | 2019-02-08 03:47:55.948 | overwrite | 51792995261850 | true | application_1520379288616_152431 | | |||
|
|
|||
| ### Entries | |||
|
|
|||
| To show all the table's current manifest entries for both data and delete files. | |||
There was a problem hiding this comment.
I think this statement is confusing. "all the table + current" ?
docs/spark-queries.md
Outdated
| @@ -335,6 +347,18 @@ Note: | |||
|
|
|||
| 2. The partitions metadata table shows partitions with data files or delete files in the current snapshot. However, delete files are not applied, and so in some cases partitions may be shown even though all their data rows are marked deleted by delete files. | |||
|
|
|||
| ### Positional Delete Files | |||
|
|
|||
| To show all positional delete files from a table: | |||
There was a problem hiding this comment.
only from current snapshot? If it is not referenced by current snapshot, it won't show right? I think we need to clarify
docs/spark-queries.md
Outdated
| @@ -357,6 +381,31 @@ SELECT * FROM prod.db.table.all_data_files; | |||
| | 0|s3://.../dt=20210103/00000-0-26222098-032f-472b-8ea5-651a55b21210-00001.parquet| PARQUET|{20210103}| 14| 2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}| null| [4]| null| 0| | |||
| | 0|s3://.../dt=20210104/00000-0-a3bb1927-88eb-4f1c-bc6e-19076b0d952e-00001.parquet| PARQUET|{20210104}| 14| 2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210104}|{1 -> 3, 2 -> 20210104}| null| [4]| null| 0| | |||
|
|
|||
| #### All Delete Files | |||
|
|
|||
| To show all the table's delete files and each file's metadata: | |||
There was a problem hiding this comment.
same comment for "all the table's"
do we need each file's metadata?
There was a problem hiding this comment.
followed the same convention of previous tables like files
There was a problem hiding this comment.
ok. The existing convention is still confusing for me to read. But we can optimize in a follow up. Ok for me to keep it similar now.
docs/spark-queries.md
Outdated
|
|
||
| #### All Entries | ||
|
|
||
| To show all the table's manifest entries from any reachable snapshot for both data and delete files: |
There was a problem hiding this comment.
from any reachable snapshot -> from all the snapshots
There was a problem hiding this comment.
also " all the table's" is confusing here too.
|
@szehon-ho , Please review and share the feedback. |
added missing metadata tables
all_delete_filesall_entriesentriesposition_deletesFixes #757