API, Spark: Update remove orphan files procedure to use bulk deletion if applicable#5373

Closed

amogh-jahagirdar wants to merge 2 commits intoapache:masterfrom amogh-jahagirdar:remove-orphan-files-deletion-batch-size

Commits on Aug 1, 2022

AWS: S3FIleIO will by default use thread pool when performing batch deletion
amogh-jahagirdar
committed

Commits on Aug 2, 2022

Spark: Update DeleteOrphanFilesSparkAction to use batch deletion by default if it's supported and a custom delete func is not passed in.
amogh-jahagirdar
committed