[FEATURE] OpenSearch and EMR-Serverless Integration

[penghuo]

Phase - 1: Spark Connector and Flint API Support [Done]

• OpenSearch Release: 2.9.0

Phase - 2: Support EMR-Serverless as compute engine.

Goals

• OpenSearch Release: 2.11.0
  • Support user could create GlueS3 datasource.
  • Support User could configure EMR-S as compute engine.
  • Add new async query API.

Tasks

• Add GlueS3 datasource
• EMR-S Interface
  • Add EMR-S configuration
  • Add EMR-S and Spark configuration when submit job to EMR-S
• Add Async query API
  • Add Create Job API
    • Add AuthZ to user has permission to access datasource
  • Add Fetch job result API
  • Add Transport Action to Security Plugin
• Query Engine
  • Support parsing datasource from SparkSQL and PPL query
  • Support difference DDL / DQL / Streaming Query.
• Query Result Index
  • Query Result Index Specification
  • Query Result Index Template
  • Query Result Index Life cycle management

Phase - 3: Add Session based query execution engine

• Design - Session based EMR-S Integration #2271
• [FEATURE] API - Session Management API #2332
• Add sessionId parameters for create async query API #2312
• add InteractiveSession and SessionManager #2290
  • Add allocateSession
• add InteractiveSession and SessionManager #2290
  • Add Session State Machine
  • Add Session State store
• Add statement
  • Add Statement #2294
  • [FEATURE] Cancel Running Statement in Spark Application. opensearch-spark#79
  • [FEATURE] Fail long waiting statement opensearch-spark#80
  • [FEATURE] Cancel Running Statement. #2333
• Integration with REPL Spark job #2327
  • Integrate with query execution
  • Integrate with fetch query result
  • Integrate with cancel query
  • Add session management index for each datasource
• Test
  • Integ test with SparkApp

Phase - 4: Add batch session and streaming session

• Deployment
  • [FEATURE] Deployment - Session Upgrade #2334
• Fault tolerant
  • [FEATURE] Failure Recovery - Support session state monitor and session recovery #2330
  • Session creation limittion #2344
  • Statement back pressure #2345
• Operation
  • [FEATURE] Operation - Purge request index #2331
• Add Batch Session
• Add Streaming Session