[FEATURE] Deployment - Session Upgrade

[penghuo]

Requirements:

• EMR-S application associate with the session could be upgraded.
• Upgrade should not impact running/waiting interactive query.
• Upgrade should not impact streaming query.

Opt-1 Client manage EMR-S Job during BG upgrade

1. new job will start, sessionId is same is oldJob
   spark job update state as RUNNING.
   spark job update new JobId and new AppId
   keep update the heartbeat.
   found statements by query
   select * form request_index
   where
   sessionId="sessionId"
   and statemtnState = "waiting"
   and appId = "newAppId"
   and jobId = "newJobId"
   order by submitTime

2. old job keep running, until finish all the tasks.
   found statements by query
   select * form request_index
   where
   sessionId="sessionId"
   and statemtnState = "waiting"
   and appId = "oldAppId"
   and jobId = "oldJobId"
   order by submitTime

Opt-2 Plugin manage EMR-S job during BG upgrade

1. Blue-Green deployment, create newApp
2. Update setting with newAppId, BG=true.
3. DP-Deployment monitor is scheduled to run every 30mis, and detect BG=True, newAppId.

for session in Running-Sessions
if it is streaming

1. old job will stop, spark job update state as DEAD.
   - SQL in old job is CREATE SKIPPING INDEX on mys3.default.http_logs
2. new job will start, spark job update state as RUNNING. keep update the heartbeat.
   - SQL in new job is RECOVER INDEX JOB flintJobName

if it is interactive session

1. start newJob, sessionId is same is oldJob
   new jobs check if session.BG=true and myJobId == newJobId, it will continue running.
2. set session.BG=true, session.jobId = newJobId.
3. when old job finish existing work, it check the condition if session.BG=true, session.myJobId != myJobId, oldJob exit.

Use cases

• case-1, old job failed when process the query
  opt-1. SessionStateMonitor should not retry failed job
  opt-2, oldJob detect the BG=true. does not update sessionState.

• case-2, old job failed when CP force close the oldApp
  opt-1, SessionStateMonitor should not retry failed job
  opt-2, oldJob detect the BG=true. does not update sessionState.

• case-3, too much tasks on old job
  opt-1, How long client should wait for oldJob finish?
  opt-2, At max, 10mins. Session will timeout.