feat(feature-flags): port local evaluation endpoint to Rust

[patricio-posthog]

Problem

The /local_evaluation endpoint for feature flags is currently served entirely by Django. This adds latency and load to the Django service for what is a high-frequency, performance-sensitive path, SDKs use this endpoint regularly to keep their local flag state up to date. We've already proven the pattern of moving hot endpoints to Rust with /decide, and local evaluation should follow the same approach.

Solution

Migrate local evaluation to a new Rust endpoint at /flags/definitions, with Django continuing to build and maintain the cache for now (in the future we would like to move that to Rust too) Then update SDKs to point to the new endpoint and keep /local_evaluation as a temporal backward-compatible proxy in Rust.

Tasks

0. (Investigate this first) Rust ↔ Django communication mechanism to trigger tasks
   • Design and implement a generic mechanism for Rust to trigger work in Django and (optionally) Django to signal Rust (e.g. Redis-based messaging plus Celery/Celery Beat on the Django side).
   • To be used initially for cache-miss fallback: on cache miss, Rust writes a “rebuild needed” signal that Django consumes to rebuild the cache.
   • Make the pattern reusable so future features can safely trigger background work across the Rust/Django boundary without ad‑hoc one-off solutions.
1. Cache-miss fallback handling (keep building the cache in Django)
   • On cache miss in Rust, return 503 and signal Django to rebuild the cache (e.g. Redis-based message that Celery/Celery Beat polls)
   • Instrument and log cache-miss events so we can monitor frequency and validate the fallback behavior
2. Port ETag handling to Rust
   • Mirror the current Django ETag behavior in /flags/definitions
   • Support If-None-Match / 304 Not Modified semantics
   • Ensure ETag generation is consistent with Django so responses are interchangeable during migration
3. Rate limiting (confirm and align behavior)
   • Verify the Rust endpoint applies the same rate limits as the Django /local_evaluation endpoint (including any special quotas for local eval)
   • Add metrics/logging to detect misconfigurations or regressions
4. Review auth.rs for correctness
   • Re-audit authentication/authorization logic for local eval / flags definitions
   • Confirm support for feature flag secret API tokens and current auth flows used by customers
   • Add/extend tests covering token validation, org/project association, and edge cases (revoked tokens, wrong project, etc.)
5. Track billing for the new endpoint
   • Confirm whether Rust requests are counted in billing/quota checks; if not, wire into the existing billing pipeline
   • Ensure local eval requests against Rust contribute correctly to usage metrics, limits, and invoices
   • Add observability (dashboards, events) to verify volumes line up with Django during rollout
6. Testing (end-to-end and safety nets)
   • Unit and integration tests for Rust logic: cache-miss paths, ETag behavior, auth, rate limiting, and billing
   • End-to-end tests comparing Django vs Rust responses for the same inputs (including cohort / non-cohort variants)
   • Canary/gradual rollout plan: small percentage of traffic first, with clear rollback and alerts
7. Enable traffic to /flags/definitions (Rust)
   • Wire up the Rust endpoint in production and begin routing a controlled portion of traffic
   • Monitor latency, error rates, cache hit/miss behavior, and impact on Postgres/Redis
   • Gradually increase traffic share as confidence grows
8. Start updating SDKs to point to the new endpoint
   • Update SDK configuration/URLs from /local_evaluation to /flags/definitions (keeping response format and query params compatible, including send_cohorts)
   • Do one SDK first as a template, then propagate to other server SDKs
   • Coordinate releases and document any configuration changes needed for customers
9. Add /local_evaluation proxy to Rust (like /decide)
   • Implement a Rust-side proxy at /local_evaluation that forwards to /flags/definitions for backward compatibility
   • Ensure no infinite loops with future proxy layers
   • Once SDKs are stable on /flags/definitions, use this as a compatibility shim and plan for eventual deprecation