Skip to content

Pre-Discovery produces zero output for all orgs — suspected celery-worker startup probe failures #430

@Noahcasarotto

Description

@Noahcasarotto

Confirmed symptom: Pre-Discovery (the scheduled background agent) produces zero output for all 8 active orgs — sessions are created, marked completed in ~16s, with empty messages and empty llm_context_history. Custom on_schedule actions never fire. Dead window spans Apr 27 → May 14 2026 (confirmed via direct prod DB query on May 14).

Impact: This is the activation blocker. A new user connects an integration, the background agent runs and produces nothing, and they churn.

Leading hypothesis (unconfirmed): celery-worker pods in the aurora namespace (GKE aurora-prod, project aurora-saas-prod, us-west1) are failing startup probes; suspected dispatch path celery beat → celery worker. Surfaced by an Aurora investigation trace but not yet verified. Chatbot pod issues also flagged in the same namespace.

Aurora RCA incident: fa836bba-1d92-4486-a483-ef8c0444d8a0 (investigation in progress).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions