Iceberg REST catalog does not delete Puffin stats files when purging the table

[findepi]

Repro steps using QueryRunner as created by io.trino.plugin.iceberg.catalog.rest.TestIcebergTrinoRestCatalogConnectorTest#createQueryRunner

trino:tpch> CREATE TABLE t AS SELECT 1 a;
CREATE TABLE: 1 row

Query 20230316_102953_00021_w7bqc, FINISHED, 2 nodes
http://localhost:8080/ui/query.html?20230316_102953_00021_w7bqc
Splits: 20 total, 20 done (100.00%)
CPU Time: 0.0s total,     0 rows/s,     0B/s, 34% active
Per Node: 0.0 parallelism,     0 rows/s,     0B/s
Parallelism: 0.1
Peak Memory: 2.38KB
0.17 [0 rows, 0B] [0 rows/s, 0B/s]

trino:tpch> ANALYZE t;
ANALYZE
 rows
------
 NULL
(1 row)

Query 20230316_102958_00022_w7bqc, FINISHED, 3 nodes
http://localhost:8080/ui/query.html?20230316_102958_00022_w7bqc
Splits: 11 total, 11 done (100.00%)
CPU Time: 0.0s total,    52 rows/s, 16.1KB/s, 70% active
Per Node: 0.0 parallelism,     1 rows/s,   466B/s
Parallelism: 0.1
Peak Memory: 716B
0.22 [1 rows, 313B] [4 rows/s, 1.37KB/s]

trino:tpch> SHOW CREATE TABLE t;
                                                                        Create Table
------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE TABLE iceberg.tpch.t (
    a integer
 )
 WITH (
    format = 'ORC',
    format_version = 2,
    location = '/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294'
 )
(1 row)

$ find /var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data/.20230316_102953_00021_w7bqc-b912a350-3c8d-4847-b8b8-9085d25c856d.orc.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data/20230316_102953_00021_w7bqc-b912a350-3c8d-4847-b8b8-9085d25c856d.orc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.00001-f30b6abd-3491-4a4a-a5df-e046bf2ecff5.metadata.json.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/b6e1e849-607a-4d3a-8d23-d6b79ca19651-m0.avro
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.b6e1e849-607a-4d3a-8d23-d6b79ca19651-m0.avro.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.00000-d34296ab-a0c2-4b2f-bdf6-7888a03b3415.metadata.json.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/snap-1422061323858791482-1-b6e1e849-607a-4d3a-8d23-d6b79ca19651.avro
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/00001-f30b6abd-3491-4a4a-a5df-e046bf2ecff5.metadata.json
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.snap-1422061323858791482-1-b6e1e849-607a-4d3a-8d23-d6b79ca19651.avro.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/00000-d34296ab-a0c2-4b2f-bdf6-7888a03b3415.metadata.json

trino:tpch> DROP TABLE t;
DROP TABLE

After the DROP, one file still remains (+ its crc)

$ find /var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/data
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/.20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats.crc
/var/folders/dr/jhnfvsrd2zl925dyfsctj0pr0000gn/T/03948b7b-ffb2-406c-96e3-b9d599f27384/iceberg_data/tpch/t-f7252f40e016439488b5e949240c3294/metadata/20230316_102958_00022_w7bqc-0e2ca33b-3bf3-4cb3-bcab-60bd1e63362a.stats

[findepi]

cc @danielcweeks @nastra @alexjo2144 @grantatspothero

[haquem1]

Can I give this issue a shot?

[MehulBatra]

I would like to assign it to me and start working on it

[vakarisbk]

Hi, is there any update on this issue?

[findepi]

for REST catalog this probably will be fixed by apache/iceberg#9305