A chunk can lose filings to a WAF disconnect and still log `chunk_done`. The ledger knows
better - it records the month PARTIAL, not SUCCESS - but the log line is what I was
counting, and it overstates progress badly. It read 92 of 139 months while scraper_runs
held 20 SUCCESS, 82 PARTIAL and 44 FAILED. 2016 and 2020 contained not one complete month.

That also explains something I had noted and not chased: KAP disclosures per year appeared
to fall from 2,317 (2015) to 300 (2021). Turkish insider filings did not drop eightfold
through a retail boom. Later months simply took more WAF damage and lost more filings.

The loss is recoverable. backfill_kap_insider.py replays non-SUCCESS months, fresh first
and WAF stragglers last. But its todo list is computed once at startup, so a month that
goes PARTIAL *during* a run is not swept by that run - the script must be re-run until the
SUCCESS count stops rising. The completion criterion is count(SUCCESS), never the log.

Also documents the quarantine, which turns out to be two different things wearing one name:

  - ~75% of the ~1,200 quarantined PDFs carry NO extractable text. They are scanned images
    with no text layer. No parser will ever read them. This is a hard ceiling of the same
    kind as the 250,000 TRY disclosure threshold - a property of the source. OCR could
    attack it, but an OCR'd number needs its own accuracy audit before it can be trusted,
    and none is attempted here.
  - the rest carry text and fail for an unrelated reason: they are narrative filings with
    no table, so the arithmetic row-validation gate has nothing to bind to. Recoverable.

The per-year quarantine rates (13% in 2015 rising to 32% in 2020) are deliberately NOT
published as a finding: the denominator is itself incomplete, and quoting a rate off an
incomplete ingest is the same error in a new place.

The ladder escalated 60s -> 600s -> 1200s on the assumption that a longer silence buys a
bigger allowance from KAP's WAF. That assumption was never measured.

Measured now, over 66 windows in the run's own log - counting how many disclosures got
through between one block and the next, bucketed by how long we had just slept:

    silence      windows    disclosures passed before the next block
    < 2 min         36                 42.9
    2-8 min         11                 57.0
    10 min           7                 57.1
    20 min          12                 49.6

It is flat. A 20-minute sleep buys the same ~50 disclosures as a 90-second one. The WAF's
budget is roughly "~50 requests, then block", and it refills within about two minutes, so
nearly every 10- and 20-minute sleep was spent waiting for an allowance we already had.
Worst-case sleep per chunk was 60+600+1200 = 31 minutes; the run was averaging 20.9 min per
month, almost all of it asleep.

Not shortened to zero: in the two observed runs of 3+ consecutive short sleeps, the 4th and
5th windows fell to 30 and 20 disclosures (n=1 each - thin, but pointing the wrong way), so
hammering without pause may still draw a penalty. The ladder keeps its shape and loses its
tail: 60 -> 120 -> 240. Worst-case sleep per chunk 7 minutes.

This matters because the ingest is not converging. Over the last 8 hours roughly 40 month
attempts produced ONE SUCCESS - the rest went PARTIAL or FAILED to WAF drops - and the
sweep of the 127 non-SUCCESS months would itself have taken ~44 hours at the old rate,
while producing fresh PARTIALs as it went. The bottleneck was never KAP's tolerance. It
was our own sleep.

The WAF blocks per source IP - the block arrives as httpx.RemoteProtocolError and the
budget (~50 requests) refills with wall-clock time, not with a slower request rate. That
was measured over 66 blocks in one backfill: a 20-minute pause bought the same ~50 requests
as a 90-second one. It is exactly the limit a rotating IP pool defeats.

RateLimitedClient now optionally rotates over a pool of proxies:

  - the pool is read from KAP_PROXIES (comma-separated) or a gitignored proxies.txt, one
    URL per line. A proxy list is a credential and must never be committed - proxies.txt is
    added to .gitignore.
  - each IP gets its own httpx client, its own rate limiter, and its own cooldown clock:
    the WAF budget is per-IP, so a single global limiter would throttle the whole pool to
    one IP's rate and waste the pool.
  - policy is drain-then-rotate, not round-robin: requests are sequential so the pool does
    not parallelise - its value is avoiding the block stall. One IP serves at full speed
    until it throws RemoteProtocolError, then it is parked for 120s (the measured refill
    time) and the request retries on the next fresh IP. By the time the rotation comes back,
    the first IP has refilled.
  - only when every IP in the pool is cooling does RemoteProtocolError surface to the
    scraper, which defers and waits it out exactly as before.

With NO proxy configured the path is byte-for-byte the old one: a single direct client,
RemoteProtocolError surfaced immediately. Pinned by tests in both directions - the no-proxy
path unchanged, and a blocked IP rotating to a fresh one rather than failing the request.

This is the lever on the ETA. Single IP, the ingest was taking ~8-12h and the non-SUCCESS
sweep barely converged; an N-IP pool cuts the block stalls that were most of that time.

…ackfill

Shortening the backoff ladder to 60/120/240 had a consequence I did not follow through:
_run_chunk raises RuntimeError when a month exhausts all its backoffs, and that exception
propagated out of the main loop and abandoned every remaining month behind it. The very
first blocked month under the new ladder - 2022-01 - crashed the entire run.

It was latent before: with the old 20-minute tail a chunk almost never exhausted, so the
raise almost never fired. Shortening the ladder made exhaustion common and exposed it.

The month's scraper_runs record is already written by scraper.run() (PARTIAL, or FAILED if
the warmup GET never got through), so a later sweep collects it - that is the ledger's
entire purpose. The loop now catches the RuntimeError, logs chunk_abandoned, and continues.
Matches the existing TimeoutError branch, which already returns rather than raising.

Pinned by a test: a chunk that raises in the middle of the run must not stop the months
behind it from being attempted.

Left over from an earlier draft; ruff F401.

The measurement that drove the proxy work also points at a fix that needs no proxy at all.
Requests between one WAF block and the next ran to a median of 161 over 87 windows (p10 =
43 - noisy, not a clean bucket). We were spending the whole budget, hitting the wall, and
paying for it three times over: the backoff sleep, the PARTIAL month the dropped filings
leave behind, and the recovery sweep that PARTIAL forces. That sweep was the real reason the
ingest would not converge - 127 of 139 months were non-SUCCESS and each needed re-running.

So pace under the budget. After _PACE_EVERY (120) requests on an IP, pause _PACE_SLEEP_S
(120s) for the refill window on our own terms. The block mostly never happens, the month
finishes SUCCESS on the first pass, and the sweep disappears. Both knobs are env-configurable
(KAP_PACE_EVERY=0 disables); the counter is per-IP, so it composes with the proxy rotation -
each IP paces itself, and a rotation to a cooled IP resets its counter because the cooldown
already refilled it.

This is the honest version of the throughput fix: it respects the server's stated limit
rather than evading it, and it is strictly better for our own convergence because it removes
the sweep. Slower per burst (~40 req/min), far faster to a complete, all-SUCCESS ingest.

Pinned by a test: PACE_EVERY requests pass freely, the next one waits.

The pacer as first written lived on the RateLimitedClient instance - but the backfill builds
a fresh client for every month (async with RateLimitedClient() per chunk), while the WAF
budget is global per source IP across months. So the counter reset at every month boundary
and never accumulated. On the sparse recent months (~30-60 requests each, well under the
120 threshold) it therefore never fired at all, and a run of short months spent the shared
budget between them and blocked - exactly what the first restart showed: pace_pause=0 with a
WAF cooldown already logged.

Moved the counter to module level, keyed by proxy URL (None = direct), so it survives the
per-month client and still composes with the proxy pool (each IP paces independently; a
cooldown resets that IP's count because the wait already refilled it).

Pinned by a test that makes two requests through one client and two through a brand-new one
and asserts the pace-of-3 trips once across the boundary rather than restarting.

The backfill process has died silently more than once mid-run (last time during a 240s
backoff, log frozen, process gone). Each death cost data-collection time until the next
manual restart. backfill_supervisor.sh runs the backfill and re-launches it whenever it
exits, so a silent death recovers on its own.

It also does the PARTIAL sweep automatically. backfill_kap_insider.py fixes its todo list at
startup, so months that go PARTIAL during a run are only collected by the NEXT run - the
supervisor re-runs until the SUCCESS month count stops rising (plateau) or reaches 139, then
stops. This is the "re-run until SUCCESS stops climbing" loop, made autonomous.

Launched detached via PowerShell Start-Process so it does not share the fate of the shell
that started it. success_count() takes only the integer line from the query because
init_db() prints db_connected to stdout ahead of it.

… pace

Two things, both surfaced by wiring in a real proxy pool.

1. A dead proxy crashed the whole backfill. Webshare's free tier returned 402 Payment
   Required after briefly working, surfacing as httpx.ProxyError - which the per-chunk
   `except RuntimeError` did not catch, so it propagated and abandoned every remaining
   month. Widened to `except Exception`: the month's ledger record is already written by
   scraper.run() before it re-raises, so a sweep collects it. One bad month, whatever
   killed it, must never take the rest of the run with it. Test now covers a ProxyError
   mid-run alongside the RuntimeError case.

2. The pool did nothing for throughput as first written. The pacer prevents blocks, and
   the old rotation only advanced on a block, so a healthy IP was drained and then SLEPT
   on at its pace limit while nine fresh IPs sat idle. Reworked so reaching the pace limit
   PARKS the IP (cools it for the refill window) and rotates to the next fresh one, exactly
   like a block does. With one IP this is the same proactive pause as before; with ten, the
   others serve while one refills, so we almost never sleep. Unified single-IP and pool
   paths through one _acquire()/_request loop; the single-IP WAF-disconnect contract
   (surface immediately) is preserved and still tested.

The free proxies are 402 for now, so the run is back on the direct+pacer path. The pool
code is proven by tests and ready for a working proxy list.

…ever

Raising the per-IP rate limit barely moved throughput (measured 65 -> 72 disclosures/min),
which proved the bottleneck was not rate but round-trip LATENCY: each request to KAP through
a UK proxy is ~1s wall time and the sequential loop sat idle waiting for it. The fix is
concurrency, not a higher rate.

  - the ingest loop (both the first pass and each cooldown-retry pass) is now a bounded
    asyncio.gather via _ingest_batch, KAP_CONCURRENCY in flight at once (default 1, i.e.
    unchanged; the backfill runs it at 6). The shared mutable state - the deferred list and
    the _Counts increments - is safe because asyncio only switches at an await and none of
    those mutations span one.
  - the client's IP selection is now round-robin (advance the cursor on every acquire)
    instead of sticky drain-then-rotate. Sequentially this makes each IP's budget last n
    times longer in wall time; under concurrency it hands each in-flight request a different
    pool IP, which is what actually parallelises the work.

Verified the one thing that could have made this unsafe: KAP's session cookie is NOT
IP-bound - a cookie minted through one proxy is accepted on a request through another (tested
directly), so one shared cookie jar across the pool is fine.

Measured at concurrency 6 over the pool: ~100-125 disclosures/min against ~65 sequential,
zero WAF blocks, and SUCCESS climbing fast (29 -> 40 in minutes). Bounded deliberately - the
DB pool is 15 connections and a public disclosure server should not face an unbounded
fan-out. Tests cover the concurrency bound, deferred collection, and round-robin spread.

The backfill is complete - 139/139 months SUCCESS, 18,415 disclosures, 2,339 clusters, more
than double the 1,079 the earlier analysis ran on. That was collected to answer the one
question left open: does the insider-cluster result hold across regimes? It does not, and
the answer is sharper than "not tradeable".

Split by the cluster's own era (scripts/regime_split.py):

    regime        20d gross   t(gross)   20d net
    2015-2018       +2.36%      6.46      -1.00%   real, uncapturable (as before)
    2019-2020       +3.34%      3.34      -5.01%   COVID small-cap mania, N=198, cost 8.4%
    2021-2026       -0.67%     -1.00      -5.44%   the gross signal is GONE

In 2015-2018 the gross abnormal return was real and strong and died only to the spread. In
2021-2026 it has decayed to nothing (t=-1.0, indistinguishable from zero before costs). The
pooled full-sample figure (+1.58% gross at 20d, t=5.2, still EDGE_DETECTED) is carried
entirely by the early era and is misleading alone. Insiders' disclosed purchases predicted
abnormal returns in 2015-2018 - returns you still could not capture after the spread - and in
the regime that matters to a trader today they no longer predict them at all.

Full-sample net-of-cost also refreshed and is sharper than the earlier cut: net negative at
every horizon, significant even at 60 days (N=2,279: 20d gross +1.58%, net -2.61%, t=-7.76;
median round trip 2.33%).

README and METHODOLOGY updated: the regime item in §6 moves from "still open" to closed with
the decay finding; the ingest and quarantine items are updated to their final state (139/139;
3,795 of 18,415 quarantined, ~75% scanned images). scripts/regime_split.py makes the split
reproducible.

… edge

The wider search for a capturable edge, logged in docs/ALTERNATIVE_ALPHA.md. Every public
signal traded after the fact fails - insider clusters (net -2.6%), holding pairs (t collapses
out-of-sample), bonus anticipation (capturable window nil once conditioned on the public
board decision), tender wins (the +2.26% run-up is real at t=7.86 but pre-announcement),
named-insider persistence (top-quartile of 2015-2020 returns -12.3% OOS), share-class
arbitrage (nothing significant). The alpha is real and it is pre-announcement.

scripts/network_signal.py finds the one exception. Splitting disclosed insider BUYS by the
actor's POINT-IN-TIME connectivity (distinct companies traded so far) and by institution vs
individual:

    regime     institutional hub (>=3 cos)   everyone else
    2015-2018  net +22.3%  t=4.40            net -1.1%
    2019-2020  net  +5.3%  t=3.10            net -15.5%
    2021-2026  net  -7.3%  t=-2.59           net  -5.6%

Connected institutional accumulation was a real, point-in-time, net-of-cost edge through 2020
- the strongest result in the project and a validation of the "follow the connected actors"
thesis - and it decayed to negative in 2021-2026 with everything else. A coordinated PACK
(>=3 different insiders piling into one name in 20 days) is the opposite: it underperforms,
net -10% in 2021-2026 (t=-6.7) - the crowd, a contrarian tell, not the smart money.

No look-ahead: connectivity uses only past trades, returns are after the same Abdi-Ranaldo
cost model, entry is t+1 after public disclosure.

…n, not mean

An outside-eye audit of the network finding, requested and warranted. The +22% net MEAN for
connected-institutional buys is driven almost entirely by extreme outliers on a
near-untradeable instrument: ISKUR (İş Bankası founder shares) prints +873% on ~32,000 TRY of
daily volume - a 25k order is most of a day, so the return is uncapturable in principle. Strip
the top 10 events and the hub mean falls 13.8% -> 6.2%; 17 actors and 41 tickers carry the
whole thing.

The MEDIAN hub trade loses money after cost in every regime (-0.7% / +0.3% / -4.3%), and with
a liquidity filter (ADV >= 1M TRY, i.e. tradeable names) the median is negative everywhere and
the mean is insignificant even in the best period (t=1.33). Point-in-time connectivity and the
regime splits were correct; the error was trusting the mean on skewed illiquid data - the exact
trap this project exists to catch, caught here in my own just-committed claim.

network_signal.py now reports mean, median, AND liquid-only median, and uses real high/low for
the spread estimate (it was passing close as high/low, understating cost - immaterial here but
wrong). docs/ALTERNATIVE_ALPHA.md rewritten: there is NO exception. Every candidate fails once
held to non-overlapping/OOS, per-trade cost, the median rather than the outlier-inflated mean,
and a liquidity filter. The alpha is real and pre-announcement (tender run-up t=7.86) but
structurally uncapturable via public data - the honest, defensible finding.