Repository navigation
feat(pricing): live cross-checked prices, refreshed on every install, at first boot and on a set cadence (#16229, #16231) - #16236
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
One real regression, which is the answer to the question you asked me to check hardest. The rest verifies, including the one line I nearly flagged wrongly. Your index question: yes, the admin emergency override stops taking effect
# admin_pricing.py:54 # redis_store.py, PricingRedisStore.set
ok = await store.set(pricing) key = _model_key(pricing.provider, pricing.model_id)
await redis.setex(key, _TTL_SECONDS, ...)The cost tracker used to read those per-provider keys directly, so an override for # base, llm_cost_tracker.py:550
for provider in ("anthropic", "openai", "google", "deepseek"):
cached = await store.get(provider, model_lower)It now reads only the by-model index: cached = await store.get_by_model(model_lower)and the only writer of that index is the refresh —
An emergency control that returns success and does nothing is the worst shape a regression can take, because it fails exactly when someone is relying on it. It is in this PR's scope rather than #16230's: this PR is what rewired the cost tracker. Two ways out, either fine:
If the owner ruling ("no hardcoded prices and no price caches") is meant to retire overrides too, that is a legitimate answer — but then the endpoint should be removed or refuse, not return success. Holding for this. Minor:
|
✅ SSOT Configuration Compliance: Passing🎉 No new hardcoded values of either class — Known backlog in |
…the refresh (#16229) Review on #16236: the cost tracker now reads only the by-model index, which only the refresh writes, so an admin override returned success and changed nothing, and a DELETE was equally inert. Overrides get their own key (model_pricing:override:<model>), with no TTL, never written by a refresh. PricingRedisStore.resolve() reads the override first, then the refreshed price, and the tracker calls it. set()/delete() become set_override()/delete_override(); admin_pricing was their only caller. PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the OpenAPI schema and the generated types are identical. Also from review: per_1m rejects infinity as it does NaN, and the one guard_egress=False call site says what that mode is.
Review addressed in
|
|
Re-review at The override now takes effect and survives the refresh.
The tests cover the failure itself, not just the new code. The minor findings are done. Ratchets unchanged in direction: One note, not a change request. The override is keyed by model id alone, so the |
|
Correction to my note above: the generated types did change, by two lines only. The bot's |
|
Delta re-review since my approval (at
Both env vars are registered. Note, non-blocking: |
|
CI red at 1. CodeQL: 2 new high alerts, "Uncontrolled data used in path expression"
2.
To fix, either:
Then re-push. Once that is green, this PR merges on its own after the train. |
|
Second-session review of
This PR is a merge-train candidate once the four-layer pre-flight passes at |
|
Delta re-review The delta touches only 5 files, all from the base merge, and none of the pricing code this PR adds.
Not re-derived: whether #16229's acceptance criteria are met. That was settled at the original approval, and the delta doesn't touch that code. |
5483844 to
4eefb1c
Compare
…the refresh (#16229) Review on #16236: the cost tracker now reads only the by-model index, which only the refresh writes, so an admin override returned success and changed nothing, and a DELETE was equally inert. Overrides get their own key (model_pricing:override:<model>), with no TTL, never written by a refresh. PricingRedisStore.resolve() reads the override first, then the refreshed price, and the tracker calls it. set()/delete() become set_override()/delete_override(); admin_pricing was their only caller. PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the OpenAPI schema and the generated types are identical. Also from review: per_1m rejects infinity as it does NaN, and the one guard_egress=False call site says what that mode is.
…he guard CodeQL reads leaves no equality branch (#16236)
…he guard (#16490) py/path-injection raised two alerts on the transcript delete: validate_relative_path contains the path, but CodeQL only credits a guard in the sink's own scope, spelled realpath + startswith(root + os.sep), the form #16229 and #16236 settled on. The validator call stays; the sink now uses the realpath it checks.
|
Delta |
…hecked, with honest freshness (#16229) Every provider "source" returned a hardcoded _BASELINE and stamped it updated_at=now, so the nightly refresh re-copied a static table and made it look fresh; the Redis-based staleness check could never fire, and the code priced claude-haiku-4-5 at $0.80/$4.00 per 1M while LiteLLM lists $1.00/$5.00. - live_sources.py: LiteLLM price map (primary) and OpenRouter models API (cross-check), fetched through the shared client with guard_egress=False (public-only, no redirects). Per-token prices become per-1M. An unstated price is None, never 0: OpenRouter "-1" is a variable-price sentinel, "0" a real free price; an absent cache price is unknown. - crosscheck.py: compares the catalogues and reports compared / agreed / disagreed / only-primary / only-secondary / not-comparable separately, so a model only one catalogue lists never reads as an agreement. - pricing_refresh: fetches both, flags disagreements beyond an env-backed tolerance, writes nothing when the primary fails. The old drift check against the hardcoded table is replaced by the cross-check. - redis_store: last_refresh_at is now the last SUCCESSFUL refresh (a failed attempt keeps it, so staleness fires); a by-model index; retired sources pruned from the status record; the cross-check report is stored, and the pricing health probe states how many models were compared and disagreed. - llm_cost_tracker: the by-model index replaces the fixed four-provider lookup (which never tried LiteLLM's "gemini"); the file stays at its ceiling rather than growing. - The catalogue URL defaults live once, in the env registry, keyed by the variable that overrides them; live_sources.py reads them back. - conftest: the pricing submodules are listed once and loaded and bound from that one tuple, registering the two new modules while the file shrinks to 1578 lines; its ceiling is lowered to match in both ratchet copies. Removing the hardcoded tables and baselines is #16230; install/update refresh is #16231.
…the refresh (#16229) Review on #16236: the cost tracker now reads only the by-model index, which only the refresh writes, so an admin override returned success and changed nothing, and a DELETE was equally inert. Overrides get their own key (model_pricing:override:<model>), with no TTL, never written by a refresh. PricingRedisStore.resolve() reads the override first, then the refreshed price, and the tracker calls it. set()/delete() become set_override()/delete_override(); admin_pricing was their only caller. PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the OpenAPI schema and the generated types are identical. Also from review: per_1m rejects infinity as it does NaN, and the one guard_egress=False call site says what that mode is.
Auto-regenerated (py3.14) to match the backend and/or SLM backend schema so the required verify-generated-types gate(s) pass. Triggered by auto-fix-generated-types.yml.
… a set cadence, and on demand (#16231) Adds a post-sync step for autobot-backend in the builtin updater (api/_pricing_post_sync.py, new module so code_sync.py stays at its size ceiling), a first-boot refresh when the Redis store is empty (refresh_if_empty), an env-backed beat cadence (AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS) replacing the fixed crontab, and an admin "refresh now" endpoint. A failed or timed-out refresh never fails the sync -- prices stay unknown until the next attempt, with the reason recorded.
Auto-regenerated (py3.14) to match the backend and/or SLM backend schema so the required verify-generated-types gate(s) pass. Triggered by auto-fix-generated-types.yml.
Collection error: _pricing_post_sync.py called get_logger(__name__) at import time. code_sync.py imports this module, and tests/api/test_collect_outdated_node_ids.py stubs config as a MagicMock, so logging_manager.py's level comparison against that mock raised during test collection. Switch to stdlib logging.getLogger(__name__), the same exception password_epoch.py already documents and uses. CodeQL py/path-injection (high, alerts #1132/#1133): _load_env_file's env_path flowed from request.component through get_live_dir / get_release_component_dir with no containment check CodeQL can see (the real code_sync.py handlers allowlist component names, but that isn't visible to the dataflow analysis). Add deployed_root() (reads SLM_DEPLOYED_ROOT at call time, never import time) and within_deployed_root() -- the realpath-equality-or-startswith(root+sep) sanitiser CodeQL's own help documents -- to deployed_dir_resolver.py, and route _load_env_file's env_path through it before exists()/read_text(). A path that resolves outside the root now raises ValueError instead of silently returning {}; the pricing caller already runs inside a never-fail try (recorded as a "failed to start" step), and the alembic / pg_dump callers in code_sync.py were already unwrapped, so they fail loudly, which is correct. Five tests in test_code_sync_deploy_bugs.py reach _load_env_file with a tmp_path deployed dir through real (non-mocked) code paths and needed SLM_DEPLOYED_ROOT pointed at that tmp root: the three direct _pg_dump_before_migration callers, plus two _run_alembic_migrations callers that only mock _pg_dump_before_migration and fall through to _run_alembic_migrations' own _load_env_file call once the dump "succeeds". The other five named test files (test_code_sync_symlink_ restore.py, test_component_resolve_job.py, test_drift_resolve.py, test_pricing_post_sync_16231.py, test_venv_reconcile_wiring_15063.py) were checked and need no change: each either mocks get_release_component_dir/_run_alembic_migrations entirely, or (the two real-subprocess AC4 tests in test_pricing_post_sync_16231.py) already passes a deployed_dir under the default /opt/autobot root. Added dedicated tests for the sanitiser itself (deployed_dir_resolver_test.py: TestWithinDeployedRoot, TestDeployedRoot) and for _load_env_file's own validation behaviour (test_pricing_post_sync_16231.py) -- a .. escape, an absolute path outside the root, and a same-prefix sibling directory are all refused; a path under the root, the root itself, and a validated-but-missing file are all accepted.
…he guard CodeQL reads leaves no equality branch (#16236)
…file containment guard runs (#16229)
… file-size ceiling holds (#16229)
4eefb1c to
c4919e1
Compare
|
Delta
The branch is 0 behind main. CI had 26 checks pending when I posted. The env-docs sync check will confirm the autogen block against the registry. |
|
Findings-first review at The "one real regression" flagged earlier (comment at 2026-09-10T18:28:30Z) was that the admin emergency-override endpoint ( Independently re-verified against the current diff (not just the prior review trail):
Everything else already vetted across the comment trail checks out at this head: no-hardcode/ approve for merge at c4919e1 🤖 Generated with Claude Code |
…sync.py (#16713) Inline a containment check (os.path.realpath + startswith(root + os.sep), a single condition with no equality branch) in the same function as each flagged filesystem sink, following the #16229/#16236 precedent that CodeQL only recognises a guard as a sanitiser within the scope it runs in: - _deploy_constraints_dir / _deploy_repo_root_requirements: source_root must resolve under the code_source root (_get_code_source_root()). - _pg_dump_before_migration: dump_path must resolve under _DB_BACKUP_DIR. - _run_alembic_migrations: cfg_path must resolve under SLM_DEPLOYED_ROOT. - _ensure_autobot_shared_symlink: link_path must resolve under the deploy base. Moves the constraints/reqs functions and the alembic subprocess runner to a new api/code_sync_paths.py, since code_sync.py sits at its #14236 file-size ceiling and cannot grow; adds _get_code_source_root(), mirroring the existing _get_deploy_base() pattern. Registers the three timeouts this split surfaced (constraints rsync, root-reqs cp, alembic upgrade) as env-var-backed constants rather than literals. Relocates the constraints/reqs/alembic containment tests (plus new refuses-traversal-before-filesystem-access tests for each guard) into a new tests/api/test_code_sync_path_containment_16713.py, keeping test_code_sync_deploy_bugs.py under its own recorded ceiling rather than growing it. Lowers both file-size ratchets to match the resulting sizes.
Thinking Path
#16228 (the owner's rule: no hardcoded prices or price caches in the codebase) starts here.
Every provider "source" today is a
BaselinePricingSource: it returns a hardcoded_BASELINEand stamps every entryupdated_at=now. That breaks freshness in three ways:set_refresh_statusre-stampslast_refresh_ateven on a failed attempt, so the staleness check (and the health probe) can never fire.claude-haiku-4-5is priced at $0.80/$4.00 per 1M, while LiteLLM lists $1.00/$5.00.The owner chose the sources: LiteLLM's price map is primary, and OpenRouter's models API is an independent cross-check. Disagreements are flagged, never resolved silently. An unknown price is never counted as $0.
What Changed
Folded in: #16231, refresh on every install and update (owner's CI-load decision)
#16231 had been built on this branch's own code, so the two ship as one PR and one CI cycle. Commit
109d67818, merged inba4d4081c:autobot-slm-backend/api/_pricing_post_sync.py. After anautobot-backendcode-sync it runspython -m services.pricing_refreshin the backend's own venv, with its deployed.env, the way alembic runs. It records the counts, or the failure reason, as one step line, and never fails the sync.code_sync.pygoes 6094 → 6082, because_load_env_filemoved into the new module and is imported back under the same name; the ceiling is lowered in both ratchet files.pricing.refresh_if_emptyCelery task, queued by aworker_readyhandler with.delay(), never run inline. It's a worker signal rather than a web-lifespan hook, becauselifespan.py's pre-existing long functions block any edit (tech-debt(guards): function-length-check is whole-file, so a legacy long function blocks every unrelated change to its file #16191).lifespan.pyis untouched.AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS(default 24) drives the beat schedule, and the Redis TTL is the interval plus one hour.POST /api/admin/pricing/refresh(admin/superadmin) returns the per-source summary.refresh_all()is public.AUTOBOT_PRICING_REFRESH_INTERVAL_HOURSandAUTOBOT_PRICING_POST_SYNC_TIMEOUT_S(default 120), each registered and tabled. The table stays unique and sorted at 225 rows.refresh_if_emptyrefreshes only a never-refreshed storeworker_readyqueues the task and is wiredmain()returns 1 and still writes its summary when nothing was writtenFrom #16229
llm_shared/pricing/live_sources.py(new):LiteLLMPricingSourceandOpenRouterPricingSource. Both fetch throughget_http_client().tracked_request(..., guard_egress=False), which is public-only and refuses redirects (CLAUDE rule 8).per_1m()returnsNonefor absent, boolean, malformed, NaN or negative values. OpenRouter's"-1"(measured on 5 router entries) is a variable-price sentinel, and"0"(22:freevariants) is a real free price.None, not 0.llm_shared/pricing/crosscheck.py(new): matches models across the two catalogues by vendor family plus normalised name. It reportscompared,agreed,disagreed[],only_primary,only_secondaryandnot_comparable(reseller routes such asbedrock/…and variants such as:free) separately. A model only one catalogue lists is never an agreement.services/pricing_refresh.py:agree,disagree,singleorunchecked, and flags disagreements beyond the env-backedAUTOBOT_PRICING_CROSSCHECK_TOLERANCE_PERCENT.llm_shared/pricing/redis_store.py:last_refresh_atnow means the last successful refresh. A failed attempt keeps it and recordslast_attempt_at.get_by_model/set_model_indexindex prices by bare name, so reseller routes and variants never shadow the direct price.retain_refresh_statusdrops retired sources, so the four old provider keys can't hold the probe at degraded forever.llm_shared/pricing/sources.py:ModelPricinggainssourceandcrosscheck, and the cache prices become optional.api/pricing_health.py: the detail line now says how many models were compared and how many disagreed.services/llm_cost_tracker.py: the by-model index replaces the fixed four-provider lookup, which never tried LiteLLM'sgemini. The file stays at its 1216-line ceiling.AUTOBOT_PRICING_{LITELLM_URL,OPENROUTER_URL,FETCH_TIMEOUT_SECONDS,CROSSCHECK_TOLERANCE_PERCENT}are registered inenv_registry_backend_services.py, and the generated docs table is updated. The URL defaults live once, in the registry, keyed by their variable, and the code reads them back.conftest.py: the pricing submodules are listed once and loaded and bound from one tuple. That registers the two new modules and shrinks the file 1580 → 1578, and the ceiling is lowered to match in both ratchet copies.Verification
live_sources_test.pycovers:per_1m: none of its unknown inputs become 0guard_egress=Falseis passed, with noallow_redirectscrosscheck_test.pycovers: agreement within tolerance, a flagged disagreement, a model only one catalogue lists (never counted as agreement), non-comparable routes and variants, and the same name under two vendors.services/pricing_refresh_test.py:model_pricing_tables_agree_15912_test.py) never reads a fixture as a price table.Model Used
Claude Opus 5 (
claude-opus-5)Single-issue rationale: the umbrella's children are deliberately separate PRs. Removing the tables (#16230) and the install/update trigger (#16231) are blocked by this one.
Closes #16229
Refs #16231 (fully delivered in code here; stays open only for its AC5 host evidence, an update through the GUI after merge)
Refs #16228