Delta Migration: Handle delta lake's automatic log clean and VACUUM

[JonasJ-ap]

Issue Fixed

This PR fixes #6781 .

The automatic log clean whenever a new checkpoint is created and the manual data file clean by VACUUM sometimes lead to inconsistency between the first constructable delta lake version and the earliest delta lake version in the table. The inconsistency will cause file not found errors during the migration.

This PR fix the issue by forcing the snapshot delta lake action to start from the first constructable delta lake version instead of the earliest version.

Test

Added Integration tests to mock the file clean scenario.

[jackye1995]

thanks for the fix, overall looks good to me, just nit for test

[jackye1995]

This seems needed as a fix for Delta conversion to work properly, because most users run VACUUM against the table. Mark it as required for 1.2 release

[amogh-jahagirdar]

Just want to confirm we don't need to explicitly handle IOException here for when the file isn't found, we'll get a NotFoundException?

[JonasJ-ap]

Thank you for pointing this out. The HadoopFileIO re-throw FileNotFoundExceptiion as NotFoundException and general IOException as RuntimeIOException.

iceberg/core/src/main/java/org/apache/iceberg/hadoop/HadoopInputFile.java

Lines 159 to 170 in b5a31a1

	private FileStatus lazyStat() {
	if (stat == null) {
	try {
	this.stat = fs.getFileStatus(path);
	} catch (FileNotFoundException e) {
	throw new NotFoundException(e, "File does not exist: %s", path);
	} catch (IOException e) {
	throw new RuntimeIOException(e, "Failed to get status for file: %s", path);
	}
	}
	return stat;
	}

In this case, I think we only need to handle the NotFoundException since VACUUM may delete the data file. I've added a code block in buildDataFileFromAction to explicitly check file existence and throw NotFoundException if necessary.

Please let me know if you have any other concern here.

[JonasJ-ap]

Thanks @jackye1995, @amogh-jahagirdar for reviewing this. I added the file existence check and refactored the integration tests based on your suggestions. @amogh-jahagirdar May I ask if you could review this PR again? Thank you in advance for your help!

[jackye1995]

looks good to me!

[jackye1995]

Merging, thanks for the fix @JonasJ-ap , and thanks for the review @yyanyy and @amogh-jahagirdar !