Design the Apache Spark version 3.5 support #1302

Part of #1301.

This issue records the design decisions for the Apache Spark version 3.5 support.
It has no code of its own; the PR for #1304 closes it.

Decision 1: Public API

The existing Spark cursors send PySpark code blocks as strings with StartCalculationExecution.
Spark Connect has no equivalent: the client builds DataFrame and SQL plans locally and sends them to the session endpoint, and SparkContext, RDDs, broadcast variables, and direct JVM access are unavailable.

Candidates:

• A. Session helper. An object or context manager that starts an Athena session on a 3.5 workgroup, waits until the endpoint is available, returns a pyspark.sql.SparkSession connected over Spark Connect, and terminates the Athena session on exit. Users write ordinary Spark Connect code.
• B. DB API cursor for Spark SQL. A PEP 249 cursor whose execute() runs Spark SQL through Spark Connect and whose fetch*() return rows. It fits the rest of PyAthena, but it accepts SQL only, not PySpark code blocks.
• C. Both, with B built on A.

Questions that apply to every option:

• How the Athena session lifecycle maps onto the new object, following the existing contract (a session the object starts is terminated on close; a supplied session is not).
• Whether sync, thread-based async, and asyncio variants are all needed, given that the Spark Connect client is synchronous.
• How users pass EngineConfiguration (Classifications) and the default configuration, since the current default (CoordinatorDpuSize, MaxConcurrentDpus, DefaultExecutorDpuSize) is rejected by 3.5.
• What happens when the auth token returned by GetSessionEndpoint expires during a long session.

Decision 2: Client dependency

• pyspark 3.5.x lists Python 3.8-3.11 in its classifiers, and its connect extra requires numpy<2.
• PyAthena requires Python 3.11+ and its pandas extra requires pandas>=3.0.0.
• Whether a pyspark 4.x client can connect to the Athena 3.5 endpoint is unknown.

To decide: the optional extra (name and version range) and the Python versions on which the 3.5 support is tested.

Decision 3: Lifetime of the Calculation API cursors

PySpark engine version 3 stops running sessions and calculations on 2027-04-01.

To decide: whether and when SparkCursor, AsyncSparkCursor, and AioSparkCursor emit a DeprecationWarning, and in which release they are removed (#1305, #1306).

Verification

Run against the workgroup from #1303 and record the results in comments:

• StartSession on a 3.5 workgroup: session states, how long until GetSessionEndpoint succeeds, and the accepted EngineConfiguration
• The error returned when the Calculation APIs are called on a 3.5 session
• Auth token lifetime and whether a client can renew it without reconnecting
• Which pyspark[connect] versions connect on the Python versions PyAthena supports
• DPU cost of a minimal session, to size the CI tests