nullbar

Pre-registration and honest statistics for trading research.

Most backtesting tools help you find an edge. This library assumes you will fool yourself while trying — because in two years of production algorithmic trading, that is what happened to us, over and over, despite code review, 2,000+ tests, and genuine care. nullbar is the machinery that caught it, extracted into a standalone package.

The name is the two things it makes you commit to before you are allowed to believe a number: the null control and the pre-registered bar.

The one-day story this library made possible

On 2026-08-11 we designed a new ML model (a cross-sectional transformer over 379 crypto assets), pre-registered its evaluation — target, architecture, splits, and a three-condition pass bar, committed before training — trained it, and evaluated it once on a test window the training code could not read even by bug.

The model achieved 5× the rank correlation of every predecessor. It also failed all three bar conditions: its extra correlation lived in the middle of the ranking, and the only part of a ranking that can pay trading costs is the tail — which a four-line moving average already owned. Design to honest verdict: one day. Without this machinery, that same question had previously consumed months and produced numbers we later had to retract.

What's inside, and the production bug behind each piece

Module	What it does	The bug it exists because of
registration	Freeze design + pass bar before results; one test look, enforced; verdicts graded fail-closed against the file on disk	"One more epoch, one more threshold" after seeing test data
ledger	Append-only trial count with per-trial metrics (no delete API)	A t=2.68 celebrated, when a best-of-64 noise search clears 3.35 five percent of the time
stats	Clustered t, PSR/DSR in strictly per-period units; dsr returns None, never 0, when the trial count or the spread is unknown	Overlapping windows inflating results 1.9×; a gate that logged PSR=0.000 for months because annualized and per-period units were mixed
evaluate	Block-clustered evaluation, hold baseline, and null controls graded against it	Pipelines that "find" effects their own machinery created
fills	Touch/through fill brackets for resting bids and asks	Assumed fills overstated executed gross 1.3–1.5× — the entries that never fill are the best ones
leaklint	Static lookahead lint (CLI) + prefix-replay check (sound one way: it cannot see a leak baked into a constant fitted outside the callable)	A multi-timeframe resampling leak that fed +23h of future into features, survived two years and every review, and explained a deployed model's entire measured edge
anchor	Git anchoring: which commits carry the registration, the trial ledger and the stamp, whether the bar precedes the result, and whether anyone outside this machine ever saw them	Timestamps a researcher writes with their own clock, in a record whose whole claim is that the bar came first
report	The whole record — registration, hash, ledger, null, result, fills, deflation, test look, graded bar — as one self-contained HTML page	Results that lived as print() calls, so what was promised and what was measured were never in the same artifact

The prefix-replay check deserves a sentence: recompute any feature on a data prefix and compare with the full-sample computation at the same rows. Any feature whose past changes when the future is appended is leaking, whatever its source looks like. This one test, run on day one, would have saved us a year.

Install

pip3 install nullbar

# or from a clone (editable — edits are live):
git clone https://github.com/brunopereira81/nullbar && cd nullbar
pip3 install -e .

# run the examples:
python3 examples/01_full_workflow.py
python3 examples/02_catch_a_leak.py

# lint a tree for lookahead patterns:
nullbar lint strategy/            # or: python3 -m nullbar strategy/

# render a finished registration as one self-contained report:
nullbar report experiments/mr24.json --ledger experiments/trials.jsonl

# anchor the record in git — before the run, and again after:
nullbar anchor experiments/mr24.json --commit
nullbar verify experiments/mr24.json

Requires Python 3.10+; numpy and pandas (2.x and 3.x are separate CI legs).

Quickstart

import nullbar

# 1. Register before you run
reg = nullbar.Registration(
    name="mean-reversion-24h",
    hypothesis="bottom-decile dist_ma168 mean-reverts over 24h",
    design={"hold_bars": 24, "entry_pct": 0.10, "cost_pct": 0.230},
    bar={"null_flat": "null control indistinguishable from holding",
         "t3": "clustered t >= 3.0 on 24h blocks",
         "beats_hold": "net beats unconditional exposure"},
)
reg.freeze("experiments/mr24.json")          # hashed; edits now visible

# 2. Count every variant you evaluate — with its Sharpe
ledger = nullbar.TrialLedger("experiments/trials.jsonl")
ledger.record("mr24", {"entry_pct": 0.10}, metrics={"sr": cell_sharpe})

# 3. Null control FIRST — against the hold baseline, not against zero
nv = nullbar.null_verdict(entry_mask, fwd_returns)        # nv["ok"] must hold
result = nullbar.block_cluster_eval(entry_mask, fwd_returns)

# 4. Price fills honestly
bracket = nullbar.fill_bracket(entry_mask, limit_px, low_px, fwd_returns)

# 5. Deflate by what you actually searched
d = nullbar.dsr(observed_sr, n=result["clusters"], n_trials=ledger.count(),
                sr_variance=ledger.sr_variance())

# 6. Spend the single test look, on the record
measured = nullbar.evidence(result, null=nv, fills=bracket, net=net)
reg.spend_test_look("experiments/mr24.json", results=measured)
print(reg.verdict({"null_flat": nv["ok"],
                   "t3": result["t"] >= 3.0,          # numpy bool: fine
                   "beats_hold": net > hold_net}))

# 7. the artifact you hand someone else
nullbar report experiments/mr24.json --ledger experiments/trials.jsonl

One self-contained HTML page (prints to PDF) carrying the frozen registration and its hash, the trial count against the registered budget, the null control, the clustered result, the fill bracket, the deflation against its 95th-percentile noise threshold, the spent test look, and the bar with every condition's observed value. Nothing is recomputed from market data, and anything the record does not contain is listed as missing. A gap the verdict DEPENDS on blocks the pass — a registration claiming a multi-cell search with no trial count renders INCOMPLETE, never as a pass, and exits non-zero. Gaps that do not bear on the verdict (an unanchored record, say) are reported without blocking it, because demanding git would put PASS out of reach for anyone not using it.

What this library will not do

It will not find you an edge. Ours, measured with these exact tools across architecture, features, horizons, and training breadth, was zero net of costs — and we can prove it, which is the point. If your strategy survives this harness, you have something. If it doesn't, you found out for the price of compute instead of capital.

It is also tamper-evident, not tamper-proof: it grades the frozen file and binds the test-look stamp to that file's hash, but anyone with write access can delete both. It is built for a researcher keeping themselves honest.

nullbar anchor closes the half of that a repository can. Anchor the registration before you run and the stamp after, and git holds the ordering: the bar's commit is an ancestor of the result's, and changing that means rewriting history that a push has already shown someone else. The trial ledger is covered too — a bar nobody can move is worth little beside a cell count anybody can shrink — and because the ledger is append-only it is checked as a prefix: recording another trial keeps the anchor intact, rewriting or deleting one breaks it. It still does not prove wall-clock time — commit dates are self-reported — nor that you had not already seen the test window. For those, anchor the hash with an RFC-3161 timestamp or a transparency log, and have somebody else hold the data.

Docs & examples

• The honest workflow — the six steps, each annotated with the production failure it prevents, plus the deflation cheat sheet.
• examples/01_full_workflow.py — the whole sequence end-to-end on synthetic data; runs in seconds, CI-tested, and writes the report at the end.
• examples/02_catch_a_leak.py — four features, two leaks, one 50ms check; includes the leak that inspired the library.
• A finished report — rendered. Our own best strategy, seven years of real hourly bars, 32 two-sided cells of search, the bar committed to git before the run and the result after: FAIL on three of four conditions. That is what the artifact looks like when the answer is no — including the anchor's own caveat, that a local-only repository was witnessed by nobody.
• The leak that survived two years — the full story.
• CHANGELOG — what moved, and why.

Status

v0.7.0 — extracted 2026-08-12 from a live production system (Coinbase spot, TimescaleDB, 2,100+ tests), then put through three independent audit passes. Every finding is fixed, each one mutation-checked against a test that fails when the bug is restored; the full list is in the CHANGELOG. The ones worth knowing, because they are the failure modes this library is about: verdict() could grade a failing strategy as PASS, fill_bracket could overstate 9× on misaligned axes, clustered_t inflated t when a cluster held no finite observation, and the frozen bar could say something different from the code grading it. API will move; the philosophy won't. MIT licensed — the statistics stay open, permanently.