Add transformWithState Python async guidance to structured streaming skill

[jiteshsoni]

Summary

Without this reference, agents re-run the same transformWithState (TWS) experiments: exists() then get(), sequential await getValue, “always use iterator()”, collapsing a large MapState into ValueState(Array), and A/B jobs on autoscale 1–N workers (wall-clock is noise). Those paths fail or look like wins until they are measured on a fixed-size cluster.

This PR extends skills/databricks-spark-structured-streaming/ with a new reference rather than a top-level skill:

• New: skills/databricks-spark-structured-streaming/references/transform-with-state.md — API facts + copy-paste Python:
  • Quick Start is sync StatefulProcessor (Spark 4.0+ / DBR 16.2+). AsyncStatefulProcessor is a gated Databricks Beta (DBR 19+, classic/assigned compute only — not OSS, not serverless)
  • RocksDB-only; processor method restrictions; ValueState vs ListState vs MapState
  • Row API; ValueState.update / MapState.updateValue take a positional tuple (not a dict; same-arity dict can silently write keys as values)
  • Timers register absolute ms (now + duration); TTL via ttlDurationMs int (no TTLConfig); per ListState value / per MapState entry; no TTL with EventTime (eventTimeColumnName required)
  • Async MapState: asyncio.gather(getValue) is concurrent/batched IPC; documented async for k, v in self.rows.iterator(); 64 / 400 / 20k / 500k are starting points to measure, not Spark defaults
  • Load-once / write-once; hoist per-key ValueState; explicit memory ceilings; re-key fat 100k-row keys
  • Fixed-size clusters for TWS benchmarks
  • initialState= is GroupedData (batch_df.groupBy(same keys)); handleInitialState(key, initialState, timerValues)
  • statestore reader via .load(path); TwsTester on sync processors only (cannot drive async)
• Navigator row in SKILL.md Core Patterns
• One-line pointer from stateful-operations.md (no duplicated TWS body)

Generated manifest.json + plugins/databricks/* copies via python3 scripts/skills.py generate (required by CI).

Test plan

• python3 scripts/skills.py validate passes locally
• Review transform-with-state.md length (~2k words, peer of other references in the folder — not a wholesale paste of a 1300-line guide)
• Confirm SKILL.md Core Patterns row and stateful-operations.md pointer
• Spot-check Python fences compile mentally: sync Quick Start, gather(getValue), documented async for iterator, timer replace (now_ms + duration_ms), update((n,))