Skip to content

Add an Apache Spark version 3.5 workgroup to the CI CloudFormation stack #1303

Open

Description

@laughingman7743

Part of #1301.

Problem

The CI stack defines only a PySpark engine version 3 Spark workgroup.
An existing workgroup cannot be upgraded to Apache Spark version 3.5, and replacing pyathena-spark would break the current Spark tests, so both workgroups must exist side by side until the Calculation API cursors are removed.

Proposed change

In cloudformation/github_actions_oidc.yaml:

  • Add a new AWS::Athena::WorkGroup with SelectedEngineVersion: "Apache Spark version 3.5", an execution role, no ResultConfiguration, and a small spark-defaults classification (for example spark.dynamicAllocation.maxExecutors) to bound CI cost.
  • Add or reuse an execution role whose policy covers the session APIs and GetSessionEndpoint, without the Calculation and notebook actions.
  • Optionally set MonitoringConfiguration for session logs.
  • Add the workgroup name as a parameter, pass it to the test workflow as an environment variable (for example AWS_ATHENA_SPARK35_WORKGROUP), and document it with the other .env variables.

Leave the pyathena-spark workgroup unchanged.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions