[DEV-14094] Update Spark downloads to support older syntax and fix TODOs

[sethstoudenmier]

Description:

Updates were made to account for older Spark syntax and also address some TODOs with cutover to EMR downloads.

Technical Details:

1. Support older Spark versions with the download code
2. Revert changes to the spark download factories that used the Delta table file location instead of the name stored in the metastore.

Requirements for PR Merge:

1. Unit & integration tests updated
2. API documentation updated (examples listed below)
   1. API Contracts
   2. API UI
   3. Comments
3. Data validation completed (examples listed below)
   1. Does this work well with the current frontend? Or is the frontend aware of a needed change?
   2. Is performance impacted in the changes (e.g., API, pipeline, downloads, etc.)?
   3. Is the expected data returned with the expected format?
4. N/A Appropriate Operations ticket(s) created
5. Jira Ticket(s)
   1. DEV-14094

Explain N/A in above checklist: