docstring-drift (Python · stdlib AST · static analysis)

Documentation that quietly stopped being true

Results · Method · Problems hit · Limitations · Future work · Use it

---

517 documented parameters across 161 Python packages do not exist on the function they describe — 1.93% of every function that documents its parameters.

Rename a parameter and nothing fails. No test breaks, no linter complains, no type checker objects — the docstring simply keeps describing a function that no longer exists.

This compares what a docstring claims the parameters are against what the signature actually says. Pure AST: no imports, no execution, no model, no network.

This number replaces an earlier one, and it is lower on purpose. The first version reported 101 phantom parameters across eight libraries. Scanning 161 packages instead of eight raised the count to 956 — and then reading the findings showed that 53% of them were not parameters at all. The six most-reported "documented parameters" in the whole corpus were docstring section headers:

 84  Returns        49  Examples      21  Shape
 59  Example        25  See            8  Inputs

Three bugs caused it. The section-header pattern only matched a header alone on its line, so Returns: the loss value fell through and was recorded as a parameter. Seven common headers — Definitions, Requirements, Relations, Shape, Inputs, Outputs and singular Warning — were missing from the list entirely. And a URL in a description parsed as a parameter, because https is a word followed by a colon.

Fixed, the same 161 packages give 517. Every one of the three bugs is now a test using the string that caused it.

---

Results

Package	functions	documenting params	phantom
transformers	35,432	2,740	105
torch	42,608	2,183	105
sympy	35,561	1,361	52
networkx	7,207	1,076	49
pandas	28,295	1,631	40
scipy	24,004	1,872	19
fontTools	5,524	155	12
rich	912	265	10
websockets · scikit-learn · sentence-transformers	13,448	1,965	27
numpy	11,215	732	8
151 others	122,626	12,876	90
161 packages	326,832	26,856	517

45 of 161 packages carry at least one. The rest are clean, which is the other half of the result: this is not a rate that applies everywhere.

Phantom rate: 1.93% of every function that documents its parameters (517 / 26,856).

📊 Full results, per-package breakdown, and a verified example →

---

A real one, from pandas

def round_trip_pickle(obj, tmp_path):          # parameter is tmp_path
    """
    Parameters
    ----------
    path : str, path object or file-like object, default None    # documents `path`
        The path where the pickled object is written and then read.
    """

The parameter was renamed. The docstring was not. Nothing in the toolchain noticed.

---

How it works

flowchart LR
    A["any Python package"] --> B["ast.parse<br/>no imports, no execution"]
    B --> C["documented_params()<br/>Google / NumPy / Sphinx"]
    B --> D["signature_params()"]
    C --> E{"compare"}
    D --> E
    E --> F["PHANTOM<br/>documented, absent"]
    E --> G["undocumented<br/>weaker signal"]
    F --> H{"*args or **kwargs?"}
    H -->|"yes"| I["skip"]
    H -->|"no"| J["report"]

    style F fill:#dc2626,color:#fff
    style J fill:#dc2626,color:#fff
    style I fill:#94a3b8,color:#fff

Loading

🔍 How the parsing and comparison actually work →

---

Use it

pip install -e .                # scanner itself: zero dependencies
docstring-drift <path>          # scan a directory, or a single .py file
docstring-drift --help          # usage
docstring-drift <path> --json   # machine-readable output, for scripting
python src/drift.py <path>      # same thing without installing

pip install -e ".[dev]"         # + pytest, ruff (or: uv sync --all-groups)
pytest -q                       # 57 tests, no network

It works as a CI check: no dependencies beyond the standard library for the scanner itself, it imports nothing from the code it scans, and the exit code means something:

Exit	Meaning
0	clean: every file parsed, no phantom parameters
1	phantom parameters found
2	the path given does not exist
3	one or more files could not be parsed (syntax error, null bytes, unreadable); each is listed with its line and error on stderr, or under "unparseable" in --json. Takes precedence over 1, so a scan that skipped files never passes a gate.

The ci.yml in this repo runs python src/drift.py src as its own gate.

Scanning third-party source can hit non-ASCII paths (author names, unicode identifiers). On Windows, chcp 65001 or setting PYTHONUTF8=1 avoids console encoding errors when printing them.

---

Input

Point it at any directory of Python. Here, an installed numpy.

Output

genfromtxt documents skiprows and missing. Neither is a parameter. The docstring even states that skiprows was removed in numpy 1.10 — and still lists it.

Also worth reading

⚠️ Limitations	What it deliberately does not detect, and why precision was chosen over recall
🚀 Future work	Type checking, stale descriptions, **kwargs recall, pre-commit hook
📐 Method	Docstring styles supported, normalisation, comparison rules

---

Layout

src/drift.py     parsing, comparison, scanning, CLI
tests/           57 tests, including one per false-positive class and one per CLI bug
docs/            detailed documentation
scripts/         scan_ecosystem.py - reproduces the ecosystem headline (see docs/RESULTS.md)
results/         measured output

Stack

Python 3.11+ · ast (standard library) · pandas · pytest · ruff · GitHub Actions — zero runtime dependencies for the scanner itself

Keywords

docstring linter · documentation drift · stale documentation · Python AST · static analysis · code quality · documentation testing · pydocstyle alternative · darglint alternative · Google style docstrings · NumPy docstrings · Sphinx docstrings · technical debt · pre-commit hook · CI linting · developer tooling

Licence

MIT — see LICENSE.