fix: stop on rate limits, cut GitHub API volume, add CI smoke mode

[edmundmiller]

Stacked on #151 (base fix/rate-limit-preflight; retarget to main once #151 merges).

Diagnosis

Run 32260435568 (draft PR #151) showed the real failure mode. The core quota was healthy — 5000 → 4635 (365 calls consumed) through commit_stats + traffic_stats, with core reset at 14:51:02. But at 14:21:29 a 403 arrived carrying reset 14:28:48 — a different bucket, i.e. GitHub's secondary rate limit, triggered by the bursty sequential comment loop in issue_stats (the tools repo).

This was not primary-quota exhaustion, and it was not caused by commit_stats volume.

The damage came from error handling: every per-repo/per-issue handler catches requests.RequestException and continues, so the run fired ~2,500 doomed requests at ~20 req/s for 90 s and still went green.

Changes

• RateLimitError + diagnosable logging (_github.py): new raise_for_github_errors() treats a 403 as a rate limit when any of remaining 0, a Retry-After header, or a secondary rate limit body marker is present, logs one line with all four header values (resource, remaining, reset, retry_after), and raises RateLimitError. Post-retry 429s raise it too. Non-rate-limit 403s still fall through to raise_for_status().
• Propagation: except RateLimitError: raise added above each existing except requests.RequestException in github_pipeline.py (traffic_stats, contributor_stats, issue_stats issues + comments, pipelines website JSON + releases, commit_stats) and citations_pipeline.py. main()'s resource loop now catches RateLimitError explicitly and breaks, instead of string-matching "rate limit" in the message.
• Cheaper issue listing: issue_stats uses per_page=100 (was defaulting to 30) and a dlt.sources.incremental("updated_at") cursor feeding since=.
• RESTClient.paginate() replaces the manual Link-header loop, reusing dlt's retry-enabled session. Signature unchanged; the non-list passthrough contract (e.g. stats/contributors 202) is preserved.
• CI smoke mode: --max-repos N limits all resources to the first N org repos; the PR (non-runitor) branch of run_pipelines.yml passes --max-repos 25 for the github pipeline. Scheduled/push production runs are unchanged.
• pipeline/AGENTS.md: documents that GET /rate_limit is not a reliable preflight, that secondary limits exist with their own reset independent of the core bucket, and that RateLimitError must never be swallowed.

Verification

• uvx ruff check src/ — clean.
• Unit check (no network): rate-limit-shaped 403s (remaining 0; and Retry-After with remaining 4999) and a 429 all raise RateLimitError; a plain 403 does not.
• Smoke run --max-repos 3: all 7 resources green, 93,676 rows.
• Second identical run: issue_stats 4,395 → 0 rows and 16 min → 18 s, confirming the incremental since= cursor took effect.

Update: dlt exception wrapping

The first CI run went red even though the stop logic worked: dlt wraps exceptions raised inside a resource generator as PipelineStepFailed -> ResourceExtractionError -> RateLimitError, so main()'s except RateLimitError never fired and the run fell through to the generic handler (marking the resource failed and continuing into the same limit). Fixed in 7999f44 with find_rate_limit_error(), which walks the cause chain instead of string-matching the message. Documented in pipeline/AGENTS.md.

That red run did produce exactly the intended log: a single Rate limited (403). resource=core remaining=0 reset=2026-08-19 16:33:05+00:00 line followed by the run stopping, with zero Failed to get comments ... rate limit lines (the old behaviour produced ~2,500 doomed requests).

Final CI run on this PR: all four pipelines green. Smoke mode: limited to 25 repositories, all 7 github resources completed in 21 min, 545,702 rows, no rate-limit spam.