ARROW-9749: [C++][GLib][Python][R][Ruby][Dataset] Introduce FragmentScanOptions, consolidate ScanContext/ScanOptions

[lidavidm]

• ScanContext/ScanOptions have been merged, since they were essentially always passed together.
• For scan options that are specific to a scan (e.g. CSV conversion options), a new FragmentScanOptions has been added to ScanOptions. Currently only CSV has this and it only wraps csv::ConvertOptions; follow up issues can tackle the rest.
• GLib, Python, R, and Ruby bindings have been updated.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-9749

[lidavidm]

As it stands implementing fragment scan options for a union dataset is a bit tricky, because the fragment doesn't know which dataset it originally belonged to (and it would be weird for every scan task implementation to have to know about a hypothetical UnionFragmentScanOptions). We could add a FragmentScanOptions field to Fragment, allowing UnionDataset::GetFragmentsImpl to do the work of matching up options to fragments there. (That'd also let an advanced caller manually specify per-fragment options if they really wanted to.)

However given we might just delete UnionDataset I'm going to leave this out of this PR for now.

[lidavidm]

@mrkn would you be willing to look at the GLib/Datasets bindings changes here again? This reduces the API surface that has to be bound by a bit, fortunately.

[mrkn]

@lidavidm I will look at it tomorrow.

[bkietz]

Thanks for this cleanup, just a couple comments:

[bkietz]

Could you add a follow up JIRAs for extracting ParquetFragmentScanOptions and IpcFragmentScanOptions?

[lidavidm]

Could you add a follow up JIRAs for extracting ParquetFragmentScanOptions and IpcFragmentScanOptions?

Done, ARROW-11972.

[mrkn]

+1 for GLib part.

[mrkn]

I created a new issue to support CsvFragmentScanOption in GLib binding.

[bkietz]

LGTM, merging