Add scan concurrency config for datafusion source

[AdamGS]

This PR adds a new configuration to the FileSource based API, allowing users to control the intra-partition scan concurrency.

[a10y]

doesn't DF let you set this for an entire sessioncontext? do we want to override this on a per-source basis?

[AdamGS]

I think that's the copy of VortexOptions that is part of VortexFormat, which propagates it downstream from there through either VortexFormatFactory::create or VortexFormat::file_source

[a10y]

I was wondering if it makes sense to use the target_partitions config off the environment, but I realize that's different.

Maybe we can make it clear in the doc comment that this is the intra-partition concurrency

[AdamGS]

intra-partition was the term I was looking for!

[codecov]

Codecov Report

❌ Patch coverage is 60.71429% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.39%. Comparing base (d757be1) to head (f1a5852).
⚠️ Report is 390 commits behind head on develop.

Files with missing lines	Patch %	Lines
vortex-datafusion/src/persistent/source.rs	30.00%	7 Missing ⚠️
vortex-datafusion/src/persistent/format.rs	62.50%	3 Missing ⚠️
vortex-datafusion/src/persistent/opener.rs	90.00%	1 Missing ⚠️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

This PR has been marked as stale because it has been open for 30 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

[a10y]

I really forgot about this. Think we still want this @AdamGS ?

[AdamGS]

yeah I think so, I'll try and unconflict it later today 😢

[a10y]

pre-approving