Skip to content

build: add a docs constant drift check - #1383

Open
davidberenstein1957 wants to merge 1 commit into
docs/methodology-rewritefrom
docs/drift-check
Open

build: add a docs constant drift check#1383
davidberenstein1957 wants to merge 1 commit into
docs/methodology-rewritefrom
docs/drift-check

Conversation

@davidberenstein1957

Copy link
Copy Markdown
Collaborator

Stacks on #1377.

Adds a check that fails when a numeric constant changes in the code while the docs keep the old value.

Why

The methodology rewrite in #1377 found that methodology.md had been documenting a CPU fallback ladder the code does not implement, plus several constants that were stale, wrong, or undocumented. Those docs were correct when written. They drifted silently, and nothing caught it.

This closes the mechanical half of that gap. It cannot catch prose going stale, but it can catch a number going stale, and a number going stale is what turned 85 W into a value users were reading off a page while the code did something else.

What it does

scripts/check_docs_drift.py guards nine constants:

Constant Source
POWER_CONSTANT codecarbon/external/hardware.py:23
CONSUMPTION_PERCENTAGE_CONSTANT codecarbon/external/hardware.py:26
DEFAULT_POWER_PER_CORE codecarbon/core/cpu.py:29
RAM_SLOT_POWER_X86 + the x86 floor codecarbon/external/ram.py:14
world_average data/private_infra/carbon_intensity_per_source.json
the three equivalence divisors codecarbon/viz/data.py:54-96

Every value is imported from the codebase, or loaded from the actual data file, then asserted to appear verbatim in the page documenting it. Nothing re-parses Python source with regexes — a checker that reads the code textually can drift from the code it is meant to guard.

Scope, deliberately narrow

Not in scope, and the module docstring says so to stop the next contributor extending it wrongly:

  • Prose accuracy — not mechanically checkable; attempting it produces noise.
  • Link checkingscripts/check_docs_links.py already does it.
  • The order of any fallback ladder — verifying narrative structure false-fails, and a check that false-fails gets disabled, which is worse than no check. If ladder ordering needs guarding, the honest tool is a unit test over resource_tracker.py.

Verification

Passes on this branch:

$ uv run python scripts/check_docs_drift.py
docs drift check: 9 constants match the docs

And it has been seen to fail. Changing the documented value to 999 in a scratch copy:

Documentation drift: a constant changed in the code but the docs still show the old value.

  POWER_CONSTANT
    code value : 85  (docs must contain 'POWER_CONSTANT = 85')
    docs       : page says: ... `POWER_CONSTANT = 999` ...
    fix        : edit docs/explanation/methodology.md to match the code -- or fix the code if the docs are the correct value

Exit 1, and the message names the constant, both values, and the file to edit, so a failure is fixable without reading the script. tests/test_docs_drift.py covers this with temp fixtures rather than touching the real docs — 3 tests, passing.

Note for review

pyproject.toml had a docs-check-drift task pointing at scripts/check-docs-drift.py, a file that never existed. This repoints it at the real script (underscored, matching check_docs_links.py) and wires it into the docs CI job.

PR #1369, which targets master, deletes that dead task. Whichever lands second will conflict on that line; the resolution is to keep the corrected task from this PR.

🤖 Generated with Claude Code

The methodology rewrite found that the docs had been describing a CPU
fallback ladder the code does not implement, along with several stale or
undocumented constants. The docs were correct when written and drifted
silently afterwards.

This makes the numeric half of that drift impossible to ship again. Nine
constants are imported from the codebase (or loaded from the data file)
and asserted to appear verbatim in the page documenting them. Nothing
re-parses Python source with regexes, so the check cannot drift from the
code it guards.

Scope is deliberately narrow: no prose checking, no link checking, and
no assertion about the order of any fallback ladder. A check that tries
to verify narrative structure false-fails, and one that false-fails gets
disabled. The rationale is recorded in the module docstring so the next
contributor does not extend it the wrong way.

`docs-check-drift` pointed at scripts/check-docs-drift.py, which never
existed; it now points at the real file and runs in the docs CI job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (docs/methodology-rewrite@e040858). Learn more about missing BASE report.

Additional details and impacted files
@@                     Coverage Diff                     @@
##             docs/methodology-rewrite    #1383   +/-   ##
===========================================================
  Coverage                            ?   91.77%           
===========================================================
  Files                               ?       49           
  Lines                               ?     5057           
  Branches                            ?        0           
===========================================================
  Hits                                ?     4641           
  Misses                              ?      416           
  Partials                            ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@davidberenstein1957
davidberenstein1957 marked this pull request as ready for review August 13, 2026 05:05
@davidberenstein1957
davidberenstein1957 requested a review from a team as a code owner August 13, 2026 05:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant