[INFRA-880] feat(kustomize): add a Pi agent-run worker

[Saurabhkmr98]

What

Adds a dedicated Celery worker for Pi's native agent runs to the pi-service component, and fixes an otel patch that would have silently broken it.

• New kustomize/components/pi-service/agent-worker.yaml — plane-pi-agent-worker-wl, cloned from worker.yaml with CELERY_QUEUE=plane_pi_agent_queue, CELERY_CONCURRENCY=4, CELERY_MAX_TASKS_PER_CHILD=100. Registered in the component's resources.
• worker.yaml narrowed to CELERY_QUEUE=plane_pi_queue, since the two now run side by side.
• New components/otel-observability/patches/append-otel-pi-agent-worker.json, registered against the new deployment.
• Fix: patches/append-otel-pi-worker.json used "op": "add", "path": "/spec/template/spec/containers/0/env" with a whole array value. JSON Patch add replaces an existing path, so with otel enabled it would have discarded the CELERY_QUEUE this change puts in the base and sent the general worker back to draining both queues. Now appends a single entry via /env/-:

-    "path": "/spec/template/spec/containers/0/env",
-    "value": [ { "name": "OTEL_SERVICE_NAME", "value": "pi-worker" } ]
+    "path": "/spec/template/spec/containers/0/env/-",
+    "value": { "name": "OTEL_SERVICE_NAME", "value": "pi-worker" }

• kustomize/README.md: pi-service now lists 4 deployments, plus a queue table.

Why

Pi had one queue, plane_pi_queue, consumed by one worker. Native agent runs shared it with vectorize_workspace and friends, which run for minutes to hours. With Celery's worker_prefetch_multiplier=1, two of those occupy the whole worker while an agent run — the interactive path, with a person watching the run's activities appear — waits behind them.

makeplane/plane-ee#9440 moves agent runs onto plane_pi_agent_queue. This is the manifest side: the worker that drains it.

Scope / behavior

This does change behavior for anyone running the pi-service component — unlike the Helm PRs, where the agent worker is behind a flag. Kustomize components have no conditional logic, so enabling pi-service now gets you the isolated two-worker topology: one extra Deployment, and the existing plane-pi-worker-wl restricted to plane_pi_queue. That is the intended end state for a commercial Kubernetes deployment, and it matches what enabling pi_agent_worker does in the charts.

The agent worker is an addition, never a replacement. plane-pi-worker-wl still drains plane_pi_queue — vector sync, docs sync, plan sync, search indexing, memory. extract_chat_memory, which an agent run dispatches on completion, deliberately stays there so a saturated agent worker cannot also stall memory extraction. Removing CELERY_QUEUE from a worker makes it consume both queues again.

Intentionally not affected: plane-pi-api-wl, plane-pi-beat-wl, the migrator, and every non-Pi component. The ten other otel patches that share the replace-the-array shape are left alone — none of their base deployments define env, so they are currently harmless.

Testing

Built with kubectl kustomize (kustomize v5.0.4) against overlays/default, with pi-service enabled and the required PI_* vars supplied.

pi-service + otel-observability — the case the patch fix covers:

plane-pi-agent-worker-wl -> [CELERY_QUEUE=plane_pi_agent_queue, CELERY_CONCURRENCY=4,
                             CELERY_MAX_TASKS_PER_CHILD=100, OTEL_SERVICE_NAME=pi-agent-worker]
plane-pi-worker-wl       -> [CELERY_QUEUE=plane_pi_queue, OTEL_SERVICE_NAME=pi-worker]

CELERY_QUEUE survives and OTEL_SERVICE_NAME is appended — before the fix, the general worker's CELERY_QUEUE would have been gone.

pi-service only, otel off:

plane-pi-agent-worker-wl -> [CELERY_QUEUE=plane_pi_agent_queue, CELERY_CONCURRENCY=4,
                             CELERY_MAX_TASKS_PER_CHILD=100]
plane-pi-worker-wl       -> [CELERY_QUEUE=plane_pi_queue]

Note: the committed overlays can't be built as-is — they require an operator-supplied vars.yaml / secrets-vars.yaml that are gitignored. Testing used a scratch copy seeded from the .example files. This is pre-existing, not a regression.

Upgrade notes

Requires an image containing makeplane/plane-ee#9440. Applying this against an older image would narrow plane-pi-worker-wl to a queue that still carries everything (fine) while the agent worker idles on a queue nothing publishes to — agent runs would stall. Roll the image first, or apply both together.

Related

• Work item: INFRA-880
• makeplane/plane-ee#9440 — adds the queue, the route, and the worker entrypoint default
• makeplane/helm-charts-private#711 — plane-cloud and plane-pi charts
• makeplane/helm-charts#313 — plane-enterprise chart

🤖 Generated with Claude Code

[makeplane]

Linked to Plane Work Item(s)

• (INFRA-880) Add a dedicated Pi agent-run worker to the kustomize pi-service

This comment was auto-generated by Plane