Skip to content

feat(pricing): live cross-checked prices, refreshed on every install, at first boot and on a set cadence (#16229, #16231) - #16236

Merged
mrveiss merged 10 commits into
mainfrom
issue-16229-live-pricing
Sep 14, 2026
Merged

mrveiss merged 10 commits into
mainfrom
issue-16229-live-pricing

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 10, 2026 •

Copy link
Copy Markdown
Owner

Thinking Path

#16228 (the owner's rule: no hardcoded prices or price caches in the codebase) starts here.

Every provider "source" today is a BaselinePricingSource: it returns a hardcoded _BASELINE and stamps every entry updated_at=now. That breaks freshness in three ways:

  • The nightly refresh re-copies a static table and marks it fresh.
  • set_refresh_status re-stamps last_refresh_at even on a failed attempt, so the staleness check (and the health probe) can never fire.
  • The prices themselves are already wrong: claude-haiku-4-5 is priced at $0.80/$4.00 per 1M, while LiteLLM lists $1.00/$5.00.

The owner chose the sources: LiteLLM's price map is primary, and OpenRouter's models API is an independent cross-check. Disagreements are flagged, never resolved silently. An unknown price is never counted as $0.

What Changed

Folded in: #16231, refresh on every install and update (owner's CI-load decision)

#16231 had been built on this branch's own code, so the two ship as one PR and one CI cycle. Commit 109d67818, merged in ba4d4081c:

  • Updater post-sync step (AC1, AC4): new autobot-slm-backend/api/_pricing_post_sync.py. After an autobot-backend code-sync it runs python -m services.pricing_refresh in the backend's own venv, with its deployed .env, the way alembic runs. It records the counts, or the failure reason, as one step line, and never fails the sync. code_sync.py goes 6094 → 6082, because _load_env_file moved into the new module and is imported back under the same name; the ceiling is lowered in both ratchet files.
  • First boot (AC2): a pricing.refresh_if_empty Celery task, queued by a worker_ready handler with .delay(), never run inline. It's a worker signal rather than a web-lifespan hook, because lifespan.py's pre-existing long functions block any edit (tech-debt(guards): function-length-check is whole-file, so a legacy long function blocks every unrelated change to its file #16191). lifespan.py is untouched.
  • Cadence (AC3): AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS (default 24) drives the beat schedule, and the Redis TTL is the interval plus one hour.
  • Refresh now: POST /api/admin/pricing/refresh (admin/superadmin) returns the per-source summary. refresh_all() is public.
  • Env vars: AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS and AUTOBOT_PRICING_POST_SYNC_TIMEOUT_S (default 120), each registered and tabled. The table stays unique and sorted at 225 rows.
  • Tests:
    • backend sync invokes the step, frontend sync doesn't
    • a failing or timing-out refresh records its reason and the sync still succeeds
    • refresh_if_empty refreshes only a never-refreshed store
    • worker_ready queues the task and is wired
    • the cadence is env-backed, and the TTL outlasts it
    • main() returns 1 and still writes its summary when nothing was written

From #16229

  • llm_shared/pricing/live_sources.py (new): LiteLLMPricingSource and OpenRouterPricingSource. Both fetch through get_http_client().tracked_request(..., guard_egress=False), which is public-only and refuses redirects (CLAUDE rule 8).
    • Per-token prices are converted to per-1M.
    • per_1m() returns None for absent, boolean, malformed, NaN or negative values. OpenRouter's "-1" (measured on 5 router entries) is a variable-price sentinel, and "0" (22 :free variants) is a real free price.
    • An absent cache price is None, not 0.
  • llm_shared/pricing/crosscheck.py (new): matches models across the two catalogues by vendor family plus normalised name. It reports compared, agreed, disagreed[], only_primary, only_secondary and not_comparable (reseller routes such as bedrock/… and variants such as :free) separately. A model only one catalogue lists is never an agreement.
  • services/pricing_refresh.py:
    • Fetches both catalogues, cross-checks them, labels every price agree, disagree, single or unchecked, and flags disagreements beyond the env-backed AUTOBOT_PRICING_CROSSCHECK_TOLERANCE_PERCENT.
    • An empty or failed primary writes nothing.
    • The old drift check against the hardcoded table is gone; the cross-check replaces it.
  • llm_shared/pricing/redis_store.py:
    • last_refresh_at now means the last successful refresh. A failed attempt keeps it and records last_attempt_at.
    • New get_by_model / set_model_index index prices by bare name, so reseller routes and variants never shadow the direct price.
    • retain_refresh_status drops retired sources, so the four old provider keys can't hold the probe at degraded forever.
    • The cross-check report is stored.
  • llm_shared/pricing/sources.py: ModelPricing gains source and crosscheck, and the cache prices become optional.
  • api/pricing_health.py: the detail line now says how many models were compared and how many disagreed.
  • services/llm_cost_tracker.py: the by-model index replaces the fixed four-provider lookup, which never tried LiteLLM's gemini. The file stays at its 1216-line ceiling.
  • Env vars: AUTOBOT_PRICING_{LITELLM_URL,OPENROUTER_URL,FETCH_TIMEOUT_SECONDS,CROSSCHECK_TOLERANCE_PERCENT} are registered in env_registry_backend_services.py, and the generated docs table is updated. The URL defaults live once, in the registry, keyed by their variable, and the code reads them back.
  • conftest.py: the pricing submodules are listed once and loaded and bound from one tuple. That registers the two new modules and shrinks the file 1580 → 1578, and the ceiling is lowered to match in both ratchet copies.

Verification

  • Not run locally; this repo verifies code through CI.
  • Tests:
    • live_sources_test.py covers:
      • per_1m: none of its unknown inputs become 0
      • the LiteLLM and OpenRouter parsers, run against samples of the real response shapes
      • wrong-shape documents
      • the guarded call: guard_egress=False is passed, with no allow_redirects
      • three failed-fetch paths (HTTP 500, a 200 with an empty body, a transport error), each yielding nothing
    • crosscheck_test.py covers: agreement within tolerance, a flagged disagreement, a model only one catalogue lists (never counted as agreement), non-comparable routes and variants, and the same name under two vendors.
    • services/pricing_refresh_test.py:
      • the refresh writes cross-checked prices and reports its denominator
      • a failed or raising primary writes nothing and is recorded as failed
      • a failed attempt keeps the last successful time
      • the model index skips routes and variants
      • the tracker finds a live price by name
  • Sample shapes: built through helpers, so the repo-wide pricing-table sweep (model_pricing_tables_agree_15912_test.py) never reads a fixture as a price table.
  • Pre-commit: passes, including the size ratchet, the hardcoded-value detector, flake8 and the env registry and docs check.
  • Endpoints, measured live on 10 Sep:
    • LiteLLM: 3,886 entries, 3,232 priced, 2,671 of them reseller routes.
    • OpenRouter: 437 models; prompt prices are 410 positive, 22 zero and 5 negative.

Model Used

Claude Opus 5 (claude-opus-5)

Single-issue rationale: the umbrella's children are deliberately separate PRs. Removing the tables (#16230) and the install/update trigger (#16231) are blocked by this one.

Closes #16229
Refs #16231 (fully delivered in code here; stays open only for its AC5 host evidence, an update through the GUI after merge)
Refs #16228

@coderabbitai

coderabbitai Bot commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: be56a70c-20e9-4492-8aa6-38fe87c76cfc


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mrveiss

mrveiss commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

One real regression, which is the answer to the question you asked me to check hardest. The rest verifies, including the one line I nearly flagged wrongly.

Your index question: yes, the admin emergency override stops taking effect

api/admin_pricing.py documents itself as "manual pricing override… PUT /api/admin/pricing/{provider}/{model} — emergency override" (GH#6480). It writes through the per-provider key only:

# admin_pricing.py:54                     # redis_store.py, PricingRedisStore.set
ok = await store.set(pricing)             key = _model_key(pricing.provider, pricing.model_id)
                                          await redis.setex(key, _TTL_SECONDS, ...)

The cost tracker used to read those per-provider keys directly, so an override for anthropic/openai/google/deepseek took effect:

# base, llm_cost_tracker.py:550
for provider in ("anthropic", "openai", "google", "deepseek"):
    cached = await store.get(provider, model_lower)

It now reads only the by-model index:

cached = await store.get_by_model(model_lower)

and the only writer of that index is the refresh — set_model_index is called at pricing_refresh.py:114, nowhere else. So:

  • an emergency override is written, logged as success=True, and has no effect on cost tracking. The operator gets a green response and nothing changes.
  • DELETE to remove an override is equally inert, for the same reason.
  • even if an override did reach the index, _setex_many is a plain SETEX, so the next daily refresh would overwrite it.

An emergency control that returns success and does nothing is the worst shape a regression can take, because it fails exactly when someone is relying on it.

It is in this PR's scope rather than #16230's: this PR is what rewired the cost tracker. Two ways out, either fine:

  • the tracker checks an override first and falls through to get_by_model — overrides keep their precedence by construction; or
  • set() also writes the by-model key for a bare id, and set_model_index stops clobbering an entry that came from an override.

If the owner ruling ("no hardcoded prices and no price caches") is meant to retire overrides too, that is a legitimate answer — but then the endpoint should be removed or refuse, not return success. Holding for this.

Minor: per_1m rejects NaN but admits infinity

if per_token != per_token or per_token < 0:   # NaN is malformed; negative is a sentinel
    return None
return per_token * _TOKENS_PER_MILLION

float("inf") passes both tests and becomes an infinite price, so every cost computed from it is infinite. Unlikely from a catalogue, but reachable: OpenRouter's prices are strings that float() parses, float("Infinity") succeeds, and Python's json accepts the Infinity literal. It is the same boundary the NaN check exists for — if not math.isfinite(per_token) or per_token < 0 covers both.

Verified, including your five checks

  • Empty 200 is a failure. _fetch sets ok = bool(pricings), and a raised fetch resets pricings = {}, so both record failure and write nothing.
  • Freshness is honest. A failed attempt moves only last_attempt_at; last_refresh_at stays at the last success, and _previous_success trusts an old-shape record's timestamp only when that record says success.
  • No or 0. per_1m returns None for None, bool, unparseable, NaN and negative input.
  • Single-catalogue models are never agreement. _merge labels secondary-only entries crosscheck="single".
  • Ratchets moved the right way. conftest.py 1580 → 1578, matching its −2. env_registry_backend_services.py is 482 lines against the default 600 limit, so it needed no entry.

Egress is correct — and the parameter name nearly fooled me. guard_egress=False reads as "guard off". request() documents it as "permit only public addresses": loopback, link-local (including cloud metadata), multicast and reserved are refused, redirects are forced off, and _assert_egress_allowed(url, allow_private=False) runs. That is exactly the right mode for fetching public catalogues. I would add one comment at the call site saying so. The next reviewer will read False the way I first did, and this call site will be the first place anyone sees guard_egress=False — there is no other non-test use on base.

@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

mrveiss added a commit that referenced this pull request Sep 11, 2026
…the refresh (#16229)

Review on #16236: the cost tracker now reads only the by-model index, which
only the refresh writes, so an admin override returned success and changed
nothing, and a DELETE was equally inert.

Overrides get their own key (model_pricing:override:<model>), with no TTL,
never written by a refresh. PricingRedisStore.resolve() reads the override
first, then the refreshed price, and the tracker calls it. set()/delete()
become set_override()/delete_override(); admin_pricing was their only caller.
PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the
OpenAPI schema and the generated types are identical.

Also from review: per_1m rejects infinity as it does NaN, and the one
guard_egress=False call site says what that mode is.
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Review addressed in 336f147f7

The override regression is fixed. Overrides now live in their own key namespace. That makes them take precedence by construction and puts them out of the refresh's reach:

  • model_pricing:override:<model> is written with SET, not SETEX, so it has no TTL. No refresh writes to that prefix. set_many and set_model_index only touch model_pricing:<provider>:* and model_pricing:by_model:*.
  • PricingRedisStore.resolve(model) returns the override if there is one, and otherwise the get_by_model price. The cost tracker now calls resolve. llm_cost_tracker.py stays at 1216 lines, so it stays at its ceiling.
  • set() / delete() → set_override() / delete_override(). admin_pricing.py was their only caller: grep -rn 'store\.(set|delete)(' finds no other non-test hit. PUT also stamps source="override" on what it stores.
  • PricingOverrideRequest moves unchanged into the new api/schemas_pricing.py. That is the move the no-local-schemas hook asks for, and because the class is unchanged, the OpenAPI schema and autobot-frontend/src/types/generated/api.ts stay identical. Its 0.0 cache defaults are a manufactured "free" price. That belongs to pricing(remove): delete every hardcoded price and price table, rewire consumers, unknown is never $0 #16230, because fixing it means regenerating the types; I recorded it there.

Tests (services/pricing_refresh_test.py) now run a real PricingRedisStore over an in-memory Redis that records each key's TTL. The earlier tests used a MagicMock store, which could not have caught this:

  • test_an_override_is_stored_apart_from_refreshed_prices_and_never_expires: exactly one key, model_pricing:override:gpt-4.1, and no TTL.
  • test_an_override_outranks_the_refreshed_price_and_survives_the_next_refresh: index, override, index again (the next refresh), then resolve returns the override. After delete_override, the live price is back, and a second delete returns False.
  • test_an_admin_override_changes_what_the_cost_tracker_charges: end to end. The PUT endpoint changes what _redis_pricing_lookup returns, and the DELETE endpoint restores it. This is the exact "reports success and changes nothing" shape.
  • test_cost_tracker_finds_a_live_price_by_model_name now runs through the real store instead of a mocked get_by_model.

Minor findings:

  • per_1m now uses math.isfinite and rejects infinity the way it rejects NaN. float("inf") and "Infinity" are added to the unknown-input test.
  • The guard_egress=False call site now carries a comment: public addresses only, redirects refused; None is the unguarded mode.

A base merge follows. docs/developer/CLAUDE_RULES.md conflicted in the generated env-var table.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Re-review at e9e6292f3: the override regression is fixed, and tested the way it failed. Approve.

The override now takes effect and survives the refresh.

  • It has its own namespace, model_pricing:override:{model_id}, written with a plain SET and no TTL (redis_store.py:118). It stands until an operator removes it.
  • resolve() reads the override first and falls back to get_by_model, and the tracker calls store.resolve(model_lower) at llm_cost_tracker.py:562.
  • The only writer of that key is admin_pricing.py:48 (set_override); DELETE goes through delete_override at :75.
  • The refresh's writes go through _setex_many, which writes per-provider and by-model keys only, so a daily refresh can no longer clobber an override.

The tests cover the failure itself, not just the new code. test_an_admin_override_changes_what_the_cost_tracker_charges runs from the admin endpoint to the tracker on an in-memory Redis. That is exactly the path that returned success and did nothing before. …_outranks_the_refreshed_price_and_survives_the_next_refresh pins the clobbering case, and …_stored_apart_from_refreshed_prices_and_never_expires pins the namespace.

The minor findings are done. per_1m uses if not math.isfinite(per_token) or per_token < 0, so both NaN and infinity are rejected. The guard_egress=False call site now explains that it permits public addresses only and refuses redirects — that comment will save the next reviewer the misreading I nearly made.

Ratchets unchanged in direction: conftest.py 1580 → 1578.

One note, not a change request. The override is keyed by model id alone, so the {provider} segment in PUT/DELETE /api/admin/pricing/{provider}/{model} no longer affects which entry is set or removed. That matches the by-model design and is probably right, but the URL still reads as if provider matters. A line in the endpoint docstring would keep an operator from assuming two providers can carry different overrides for the same model.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Correction to my note above: the generated types did change, by two lines only. The bot's 84630e557 regenerated api.ts with the new endpoint descriptions from the PUT/DELETE docstrings. PricingOverrideRequest's schema is unchanged, which is what the move needed. No other difference: git show --stat 84630e557 shows 2 insertions and 2 deletions, both @description lines.

@mrveiss mrveiss changed the title feat(pricing): fetch live prices from LiteLLM and OpenRouter, cross-checked, with honest freshness (#16229) feat(pricing): live cross-checked prices, refreshed on every install, at first boot and on a set cadence (#16229, #16231) Sep 11, 2026
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Delta re-review since my approval (at 84630e557, carried over at a0633032df): approved at ba4d4081cc. It merges on a settled green at that SHA. Closes #16229 with Refs #16231 is right, since #16231's AC5 needs host evidence.

ba4d4081c merges the #16231 branch (109d67818) with an empty combined diff. The delta is 109d67818:

#16231 AC Evidence
AC1: a code-sync of autobot-backend triggers a refresh code_sync.py calls run_pricing_refresh_post_sync(...), and _pricing_post_sync.py runs the one-shot python -m services.pricing_refresh CLI as a subprocess, with the deployed env and an env-backed timeout (AUTOBOT_PRICING_POST_SYNC_TIMEOUT_S, 120). Tests: test_backend_sync_invokes_the_pricing_post_sync_step, with the contrast test_frontend_sync_never_invokes_the_pricing_post_sync_step.
AC2: an empty store at startup triggers a refresh worker_ready queues pricing.refresh_if_empty and never runs it inline. It refreshes only a never-refreshed store and never raises. Tests: test_refresh_if_empty_refreshes_a_never_refreshed_store, …_does_nothing_when_status_is_not_empty, …_never_raises_on_a_broken_store, test_the_first_boot_refresh_handler_is_actually_wired_to_worker_ready.
AC3: the cadence is env-backed AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS (24) drives celery_app's timedelta schedule, and the TTL is that interval plus one hour. Tests: test_beat_cadence_is_env_backed_not_a_fixed_crontab, test_ttl_always_outlasts_the_refresh_interval_it_is_meant_to_survive.
AC4: offline, the install completes with a visible reason A failing or timed-out refresh records its reason, and the sync still succeeds (test_a_failing_… / test_a_timed_out_pricing_refresh_records_the_reason_and_the_sync_still_succeeds). The CLI exits 1 but still writes its summary when nothing was written.
On demand POST /api/admin/pricing/refresh carries Depends(require_role("admin", "superadmin")), the same gate as the override PUT/DELETE and the status GET. test_admin_refresh_now_returns_the_refresh_summary.

Both env vars are registered. code_sync.py goes 6094 → 6082, identical in both ratchet files and matching the file (_load_env_file moved). lifespan.py is untouched.

Note, non-blocking: REFRESH_INTERVAL_HOURS reads through a plain env_int. An operator who sets it to 0 to "turn refreshes off" gets timedelta(hours=0): a beat entry due on every tick, hammering both external catalogues, with a one-hour TTL. A negative value is worse. env_int_clamped(…, min_v=1) from autobot_shared.env_utils closes it, in this PR or a filed follow-up.

Comment thread autobot-slm-backend/api/_pricing_post_sync.py Fixed
Comment thread autobot-slm-backend/api/_pricing_post_sync.py Fixed
@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

CI red at d88b57d35a, caused by this PR, so it is out of the merge train. There are two independent reds.

1. CodeQL: 2 new high alerts, "Uncontrolled data used in path expression"

  • Where: autobot-slm-backend/api/_pricing_post_sync.py:48 and :50. That is _load_env_file, at env_path.exists() and env_path.read_text(...).
  • The file is new in this PR.
  • CodeQL traces env_path back to two user-provided values; the alert's source links name them.

2. authz-and-selection: collection error in autobot-slm-backend/tests/api/test_collect_outdated_node_ids.py

  • That test file is on base and is not touched here.
  • Its from api.code_sync import ... now loads api._pricing_post_sync.
  • The new module's import-time get_logger(__name__) runs into that test's mocking:
autobot_shared/logging_manager.py:312: in _get_file_handler
E   TypeError: '>' not supported between instances of 'MagicMock' and 'int'

To fix, either:

  • defer the logger or config reach out of import time in _pricing_post_sync.py, or
  • make the test's mock cover the new import chain.

Then re-push. Once that is green, this PR merges on its own after the train.

@mrveiss

mrveiss commented Sep 11, 2026

Copy link
Copy Markdown
Owner Author

Second-session review of 6353779a1 (in head 12d81cdee): passes. The branch is 0 behind a12b4bf79d, it includes #16300's base, and there are no trailers.

  • CodeQL py/path-injection (feat(browser): screenshot-based visual browser for chat browser tab (#1130) #1132, feat(analytics): GitHub-backed code source registry for codebase analytics #1133). services/deployed_dir_resolver.py adds two functions.

    • deployed_root() (:39-48) resolves SLM_DEPLOYED_ROOT with os.path.realpath at call time.
    • within_deployed_root() (:51-64) resolves the candidate the same way, then accepts it only when it equals the root or starts with root + os.sep, so a sibling directory whose name merely begins with the root can't pass. Anything else raises a ValueError naming the path.

    _load_env_file routes through it (_pricing_post_sync.py:58,61).

  • Tests (deployed_dir_resolver_test.py):

    • accepted: a path under the root (:201), the root itself (:206);
    • refused: a .. escape (:210), an absolute path outside (:220), a sibling directory that shares the root as a string prefix (:230);
    • the root is read live, not at import time (:246).

    test_code_sync_deploy_bugs.py sets SLM_DEPLOYED_ROOT (8 references).

  • Logging. It uses plain stdlib logging.getLogger, with the reason in the file (:33-38): it follows the password_epoch.py:50-58 precedent for a module imported under config-mocking tests, which is the documented exception.

  • Not introduced here: the "/opt/autobot" default in deployed_root() was already in _resolve_deployed_dir before this commit, and it only moved.

This PR is a merge-train candidate once the four-layer pre-flight passes at 12d81cdee.

@mrveiss

mrveiss commented Sep 12, 2026

Copy link
Copy Markdown
Owner Author

Delta re-review 814777f68..5483844c7: approved. It merges at 5483844c7 on a settled green, after #16487.

The delta touches only 5 files, all from the base merge, and none of the pricing code this PR adds.

Check Result
Size-ratchet dicts (both files), key by key 496 entries on each side. None raised, none dropped, none added. This PR's own lowered entries survived: code_sync.py 6082 and conftest.py 1578, both equal to wc -l at head
.secrets.baseline Same 284 result keys. Only line-number drift from an unrelated upstream commit
api.ts, .pre-commit-config.yaml Only base churn from #16178, #16343 and #16191. This PR's pricing paths are byte-identical at both SHAs
Closing refs [16229] only, matching the body. #16231 and #16228 stay open as stated

Not re-derived: whether #16229's acceptance criteria are met. That was settled at the original approval, and the delta doesn't touch that code.

@mrveiss
mrveiss force-pushed the issue-16229-live-pricing branch from 5483844 to 4eefb1c Compare September 12, 2026 22:33
mrveiss added a commit that referenced this pull request Sep 12, 2026
…the refresh (#16229)

Review on #16236: the cost tracker now reads only the by-model index, which
only the refresh writes, so an admin override returned success and changed
nothing, and a DELETE was equally inert.

Overrides get their own key (model_pricing:override:<model>), with no TTL,
never written by a refresh. PricingRedisStore.resolve() reads the override
first, then the refreshed price, and the tracker calls it. set()/delete()
become set_override()/delete_override(); admin_pricing was their only caller.
PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the
OpenAPI schema and the generated types are identical.

Also from review: per_1m rejects infinity as it does NaN, and the one
guard_egress=False call site says what that mode is.
mrveiss added a commit that referenced this pull request Sep 12, 2026
…he guard CodeQL reads leaves no equality branch (#16236)
mrveiss added a commit that referenced this pull request Sep 12, 2026
…he guard (#16490)

py/path-injection raised two alerts on the transcript delete: validate_relative_path contains the path, but CodeQL only credits a guard in the sink's own scope, spelled realpath + startswith(root + os.sep), the form #16229 and #16236 settled on. The validator call stays; the sink now uses the realpath it checks.
@mrveiss

mrveiss commented Sep 12, 2026

Copy link
Copy Markdown
Owner Author

Delta 5483844c7..4eefb1c8b (the rebase onto main): approve. Every code, doc and ratchet file is unchanged from the approved net diff; the one file that differs is .secrets.baseline. It holds exactly main's entries by (filename, hashed_secret, type), with 0 added and 0 dropped. That matches the approved version, which added or dropped none either. It carries 7 line moves, all in files this PR changes (test_code_sync_deploy_bugs.py and the regenerated api.ts), consistent with the PR's own edits. The api.ts:62316 entry is identical to main's. The branch is 0 behind main.

mrveiss and others added 10 commits September 13, 2026 09:44
…hecked, with honest freshness (#16229)

Every provider "source" returned a hardcoded _BASELINE and stamped it
updated_at=now, so the nightly refresh re-copied a static table and made it
look fresh; the Redis-based staleness check could never fire, and the code
priced claude-haiku-4-5 at $0.80/$4.00 per 1M while LiteLLM lists $1.00/$5.00.

- live_sources.py: LiteLLM price map (primary) and OpenRouter models API
  (cross-check), fetched through the shared client with guard_egress=False
  (public-only, no redirects). Per-token prices become per-1M. An unstated
  price is None, never 0: OpenRouter "-1" is a variable-price sentinel,
  "0" a real free price; an absent cache price is unknown.
- crosscheck.py: compares the catalogues and reports compared / agreed /
  disagreed / only-primary / only-secondary / not-comparable separately, so
  a model only one catalogue lists never reads as an agreement.
- pricing_refresh: fetches both, flags disagreements beyond an env-backed
  tolerance, writes nothing when the primary fails. The old drift check
  against the hardcoded table is replaced by the cross-check.
- redis_store: last_refresh_at is now the last SUCCESSFUL refresh (a failed
  attempt keeps it, so staleness fires); a by-model index; retired sources
  pruned from the status record; the cross-check report is stored, and the
  pricing health probe states how many models were compared and disagreed.
- llm_cost_tracker: the by-model index replaces the fixed four-provider
  lookup (which never tried LiteLLM's "gemini"); the file stays at its
  ceiling rather than growing.
- The catalogue URL defaults live once, in the env registry, keyed by the
  variable that overrides them; live_sources.py reads them back.
- conftest: the pricing submodules are listed once and loaded and bound from
  that one tuple, registering the two new modules while the file shrinks to
  1578 lines; its ceiling is lowered to match in both ratchet copies.

Removing the hardcoded tables and baselines is #16230; install/update
refresh is #16231.
…the refresh (#16229)

Review on #16236: the cost tracker now reads only the by-model index, which
only the refresh writes, so an admin override returned success and changed
nothing, and a DELETE was equally inert.

Overrides get their own key (model_pricing:override:<model>), with no TTL,
never written by a refresh. PricingRedisStore.resolve() reads the override
first, then the refreshed price, and the tracker calls it. set()/delete()
become set_override()/delete_override(); admin_pricing was their only caller.
PricingOverrideRequest moves unchanged to api/schemas_pricing.py, so the
OpenAPI schema and the generated types are identical.

Also from review: per_1m rejects infinity as it does NaN, and the one
guard_egress=False call site says what that mode is.
Auto-regenerated (py3.14) to match the backend and/or SLM backend
schema so the required verify-generated-types gate(s) pass.
Triggered by auto-fix-generated-types.yml.
… a set cadence, and on demand (#16231)

Adds a post-sync step for autobot-backend in the builtin updater
(api/_pricing_post_sync.py, new module so code_sync.py stays at its size
ceiling), a first-boot refresh when the Redis store is empty
(refresh_if_empty), an env-backed beat cadence
(AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS) replacing the fixed crontab, and an
admin "refresh now" endpoint. A failed or timed-out refresh never fails the
sync -- prices stay unknown until the next attempt, with the reason recorded.
Auto-regenerated (py3.14) to match the backend and/or SLM backend
schema so the required verify-generated-types gate(s) pass.
Triggered by auto-fix-generated-types.yml.
Collection error: _pricing_post_sync.py called get_logger(__name__) at
import time. code_sync.py imports this module, and
tests/api/test_collect_outdated_node_ids.py stubs config as a MagicMock,
so logging_manager.py's level comparison against that mock raised during
test collection. Switch to stdlib logging.getLogger(__name__), the same
exception password_epoch.py already documents and uses.

CodeQL py/path-injection (high, alerts #1132/#1133): _load_env_file's
env_path flowed from request.component through get_live_dir /
get_release_component_dir with no containment check CodeQL can see (the
real code_sync.py handlers allowlist component names, but that isn't
visible to the dataflow analysis). Add deployed_root() (reads
SLM_DEPLOYED_ROOT at call time, never import time) and
within_deployed_root() -- the realpath-equality-or-startswith(root+sep)
sanitiser CodeQL's own help documents -- to deployed_dir_resolver.py, and
route _load_env_file's env_path through it before exists()/read_text().
A path that resolves outside the root now raises ValueError instead of
silently returning {}; the pricing caller already runs inside a
never-fail try (recorded as a "failed to start" step), and the alembic /
pg_dump callers in code_sync.py were already unwrapped, so they fail
loudly, which is correct.

Five tests in test_code_sync_deploy_bugs.py reach _load_env_file with a
tmp_path deployed dir through real (non-mocked) code paths and needed
SLM_DEPLOYED_ROOT pointed at that tmp root: the three direct
_pg_dump_before_migration callers, plus two _run_alembic_migrations
callers that only mock _pg_dump_before_migration and fall through to
_run_alembic_migrations' own _load_env_file call once the dump
"succeeds". The other five named test files (test_code_sync_symlink_
restore.py, test_component_resolve_job.py, test_drift_resolve.py,
test_pricing_post_sync_16231.py, test_venv_reconcile_wiring_15063.py)
were checked and need no change: each either mocks
get_release_component_dir/_run_alembic_migrations entirely, or (the two
real-subprocess AC4 tests in test_pricing_post_sync_16231.py) already
passes a deployed_dir under the default /opt/autobot root.

Added dedicated tests for the sanitiser itself
(deployed_dir_resolver_test.py: TestWithinDeployedRoot,
TestDeployedRoot) and for _load_env_file's own validation behaviour
(test_pricing_post_sync_16231.py) -- a .. escape, an absolute path
outside the root, and a same-prefix sibling directory are all refused; a
path under the root, the root itself, and a validated-but-missing file
are all accepted.
…he guard CodeQL reads leaves no equality branch (#16236)
@mrveiss
mrveiss force-pushed the issue-16229-live-pricing branch from 4eefb1c to c4919e1 Compare September 13, 2026 06:55
@mrveiss

mrveiss commented Sep 13, 2026

Copy link
Copy Markdown
Owner Author

Delta 4eefb1c8b..c4919e113 (the rebase onto post-2a main): approve. All three conflict resolutions check out, and the PR's own scope is unchanged.

  • Scope: the net diff against main is 29 files, +1956/−216, the same as the approved version. Only the three conflict files differ from the approved net diff: CLAUDE_RULES.md and the two ratchet files.
  • docs/developer/CLAUDE_RULES.md footer. Line 676 reads "227 variables registered as of last generation." I counted the | AUTOBOT_… table rows strictly between <!-- BEGIN_AUTOGEN_ENV_DOCS --> (line 445) and <!-- END_AUTOGEN_ENV_DOCS --> (line 677): exactly 227. The whole file has 228, and the extra row is the cache-TTL table's AUTOBOT_CHAT_SESSION_CACHE_TTL at line 111, outside the block, so 227 is right. The PR's footer delta is preserved: 219→225 approved becomes 221→227 now, +6 either way, on top of main's two new AUTOBOT_API_KEY_* rows.
  • Ratchets. Every ceiling equals the file's real line count at this head, in both scripts/python_file_size_known_large.py and repo_tests/python_file_size_ratchet_baseline.py:
    • autobot-backend/conftest.py: 1568. This was the conflict; main had also changed the file.
    • autobot-slm-backend/api/code_sync.py: 6082.
    • autobot-slm-backend/tests/api/test_code_sync_deploy_bugs.py: 2166.
    • autobot-backend/services/llm_cost_tracker.py: 1216.
  • autobot_shared/env_registry_slm.py. Main's AUTOBOT_API_KEY_LEGACY_GRACE_DAYS (:271) and AUTOBOT_API_KEY_SCOPES_ENFORCED_FROM (:285) are kept alongside this PR's AUTOBOT_PRICING_POST_SYNC_TIMEOUT_S (:299). No name= appears twice, and the PR's own change to the file is identical to the approved one.

The branch is 0 behind main. CI had 26 checks pending when I posted. The env-docs sync check will confirm the autogen block against the registry.

@mrveiss

mrveiss commented Sep 13, 2026

Copy link
Copy Markdown
Owner Author

Findings-first review at c4919e113601a2442d160c43588f2e0231bd8cea. Verdict: approve for merge.

The "one real regression" flagged earlier (comment at 2026-09-10T18:28:30Z) was that the admin emergency-override endpoint (PUT /api/admin/pricing/{provider}/{model}) wrote through the per-provider Redis key while the rewired cost tracker read only the by-model index — the override was accepted with success=True and silently had zero effect on billing, and even if it had landed, the next daily refresh's SETEX would have clobbered it. Fixed at 336f147f7: overrides now live in their own model_pricing:override:{model} namespace written with a bare SET (no TTL), PricingRedisStore.resolve() checks the override first and falls back to get_by_model, and llm_cost_tracker._redis_pricing_lookup calls resolve() (services/llm_cost_tracker.py:1143). Confirmed present and unchanged through every subsequent rebase, and covered by test_an_admin_override_changes_what_the_cost_tracker_charges (exercises the exact "reports success, changes nothing" path) plus two more pinning the namespace and refresh-survival.

Independently re-verified against the current diff (not just the prior review trail):

Severity file:line Finding Fix / status
— redis_store.py:930-940 Override regression (see above) Fixed at 336f147f7, confirmed present at head
— live_sources.py:588 per_1m rejected NaN but admitted float("inf")/"Infinity" Fixed: not math.isfinite(per_token) or per_token < 0
— _pricing_post_sync.py:2037-2039 CodeQL py/path-injection (#1132-#1135): compound ==root or startswith guard not recognized as a sanitiser Fixed: single inlined if not real.startswith(root + os.sep): raise, in the same scope as the exists()/read_text() sinks. CodeQL Analyze (python) is green at this head
LOW redis_store.py:842 REFRESH_INTERVAL_HOURS = env_int("AUTOBOT_PRICING_REFRESH_INTERVAL_HOURS", 24) has no floor — 0 or negative produces a beat schedule that fires every tick and a TTL ≤ 3600s Use env_int_clamped(..., min_v=1). Flagged non-blocking in an earlier round; still open at this head, filing recommended rather than blocking
LOW admin_pricing.py:92-96 docstring PUT/DELETE /api/admin/pricing/{provider}/{model} still reads as if {provider} selects the override entry; it's actually keyed by model id alone One-line docstring clarification. Also flagged non-blocking previously; still open

Everything else already vetted across the comment trail checks out at this head: no-hardcode/or 0 masking, empty-200-is-failure semantics, honest freshness timestamps, guard_egress=False now commented at its call site, ratchets/env-registry/generated-types diffs are all no-op churn from base merges, and CI (including CodeQL, secret scanning, SAST) is green.

approve for merge at c4919e1

🤖 Generated with Claude Code

mrveiss added a commit that referenced this pull request Sep 14, 2026
…sync.py (#16713)

Inline a containment check (os.path.realpath + startswith(root + os.sep),
a single condition with no equality branch) in the same function as each
flagged filesystem sink, following the #16229/#16236 precedent that CodeQL
only recognises a guard as a sanitiser within the scope it runs in:

- _deploy_constraints_dir / _deploy_repo_root_requirements: source_root
  must resolve under the code_source root (_get_code_source_root()).
- _pg_dump_before_migration: dump_path must resolve under _DB_BACKUP_DIR.
- _run_alembic_migrations: cfg_path must resolve under SLM_DEPLOYED_ROOT.
- _ensure_autobot_shared_symlink: link_path must resolve under the
  deploy base.

Moves the constraints/reqs functions and the alembic subprocess runner to
a new api/code_sync_paths.py, since code_sync.py sits at its #14236
file-size ceiling and cannot grow; adds _get_code_source_root(), mirroring
the existing _get_deploy_base() pattern. Registers the three timeouts this
split surfaced (constraints rsync, root-reqs cp, alembic upgrade) as
env-var-backed constants rather than literals.

Relocates the constraints/reqs/alembic containment tests (plus new
refuses-traversal-before-filesystem-access tests for each guard) into a
new tests/api/test_code_sync_path_containment_16713.py, keeping
test_code_sync_deploy_bugs.py under its own recorded ceiling rather than
growing it. Lowers both file-size ratchets to match the resulting sizes.
@mrveiss
mrveiss merged commit 358727d into main Sep 14, 2026
81 checks passed
@mrveiss
mrveiss deleted the issue-16229-live-pricing branch September 14, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pricing(sources): fetch live prices from LiteLLM and OpenRouter, cross-checked, with honest freshness

2 participants