[FEATURE][UMBRELLA] Data Agent Engine — isolation, identity, and security hardening

[wangzhigang1999]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

Umbrella for the security-hardening / isolation follow-ups deferred while landing the Data Agent engine (#7379, last PR #7431). Sub-tasks are tracked inline, not split into separate issues.

Root cause behind most of these: DataAgentProcessBuilder extends ProcBuilder, so the engine runs as a local child process on the Kyuubi server host (no k8s/YARN launch path) and shares the server's Kerberos credential cache, network identity, and filesystem. That sharing is why Kerberos is currently banned and why credentials must go in the JDBC URL. Giving the engine its own isolated runtime + identity unblocks the rest.

Sub-tasks

• ①a Cross-server Data Agent session routing. Make Kyuubi servers stateless proxies for Data Agent REST sessions. Register a discovery mapping from the external session UUID to its Data Agent engine, then let any server attach to the existing engine-side session handle instead of depending on the creating server's in-memory KyuubiSessionImpl. The conversation, approval, and tool state remains in the engine. This fixes requests being routed to another server in a multi-server deployment; it does not preserve a local child engine when its launching server fails.
• ①b Isolated runtime with its own identity (keystone). Add a remote launch path so DATA_AGENT runs in its own container with a distinct identity instead of as a local child of the server — k8s first, YARN later; keep local-process as a dev fallback. This decouples engine lifetime from a server instance and unblocks ② and ③.
• ② Kerberos support (depends on ①b). Engine logs in with its own principal/keytab and impersonates the session user via hadoop.proxyuser ACLs, lifting the current ban. Removes the two TODOs in EngineRef#deriveDataAgentJdbcUrl and ChatCompletionProvider.
• ③ Credential propagation (design depends on ①b). Dedicated config / UI inputs + a propagateCredential opt-in so secrets aren't embedded in kyuubi.engine.data.agent.jdbc.url. Align with the JDBC engine's propagateCredential and Trino's connection.user fallback. Raised in [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI #7431.
• ④ Cheaper summarization model for context compaction (independent). Let CompactionMiddleware use a model distinct from the main agent model, falling back when unset (see its TODO).
• ⑤ Proper SQL read-only check for run_select_query — tracked in [FEATURE] Add a proper SQL read-only check for run_select_query #7412; linked for visibility, orthogonal to the above.

Dependency order: ①a fixes cross-server request routing first. ①b then decouples engine lifetime and identity from the launching server; ② and ③ build on ①b. ④ and #7412 are independent.

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.