fix(iceberg): add_files correctly check duplicates

[Erigara]

Which issue does this PR close?

• Closes bug(iceberg): add_files doesn't actually check duplicated files #1394.

What changes are included in this PR?

• compare duplicates by loading manifest files and taking file_path from it
• use direct calls instead of scan

Are these changes tested?

• work for me local experiments
• fixed existing tests
• added a new test to showcase behavior

[Erigara]

I changed my initial implementation so that DELETED manifest entries are skipped.
Which is in line with pyiceberg behavior:

• https://github.com/apache/iceberg-python/blob/3070e7a2b8d681cd02f753b3a46e6ff1b27b76cf/pyiceberg/table/__init__.py#L835
• https://github.com/apache/iceberg-python/blob/3070e7a2b8d681cd02f753b3a46e6ff1b27b76cf/pyiceberg/table/inspect.py#L660
• https://github.com/apache/iceberg-python/blob/3070e7a2b8d681cd02f753b3a46e6ff1b27b76cf/pyiceberg/table/inspect.py#L655
• https://github.com/apache/iceberg-python/blob/3070e7a2b8d681cd02f753b3a46e6ff1b27b76cf/pyiceberg/table/inspect.py#L535
• https://github.com/apache/iceberg-python/blob/3070e7a2b8d681cd02f753b3a46e6ff1b27b76cf/pyiceberg/manifest.py#L717
  • here you can see that DELETED entries are skipped

[Erigara]

I'm not sure what was initial purpose of the test.
But it looks like to check ability of adding existing parquet files to the table (physically existing not in table).
In which case adding with_check_duplicate(false) here is fine.