Repository navigation
ADR-079 P8 follow-up: PCK@20 3.0% → ≥35% requires more paired data + multi-room framing #645
Description
Status (after today's cog v0.0.1 ship — see PRs #642, #643, #644)
ADR-079 P7 (data collection), P8 (alignment + train + eval) and cog packaging end-to-end all ran today. The pipeline is validated and a signed cog-pose-estimation@0.0.1 binary is live at gs://cognitum-apps/cogs/{arm,x86_64}/, installed on cognitum-v0. The remaining work for a useful model is data-bound.
| Lever | Current (v0.0.1) | Target |
|---|---|---|
| Paired sample count | 1,077 | 30,000+ (multiple 30-min sessions × full-body framing) |
| Camera framing | torso-up at desk (avg n_visible 14.3/17) | full-body, varied movements, multiple rooms |
| Avg detection confidence | 0.476 | ≥ 0.7 |
| Training epochs | 400 (Candle CUDA, 2.1 s on RTX 5080) | 1000+ if needed (still seconds on the GPU) |
| PCK@20 | 3.0% | ≥ 35% |
| PCK@50 | 18.5% | ≥ 60% |
| MPJPE (normalized) | 0.093 | < 0.05 |
What the v0.0.1 numbers tell us
Per-joint PCK@50 ranks show the model is learning where the camera lets it:
r_hip 76.9% ← excellent (right side most consistently in frame)
r_knee 35.2%
l_hip 27.3%
l_elbow 26.4%
l_wrist 24.1%
l_knee 20.8%
r_shoulder 19.9%
...
nose 5.1% ← essentially random (face joints at desk-level zoom)
l_ankle 7.9%
r_ankle 9.3%
The asymmetry is a direct reading of the seated-at-desk camera framing — not a model defect. CSI at 56 subcarriers × 20 frames carries enough spatial info for proximal joints with consistent visibility; it doesn't carry enough for fine-grained extremities. More data won't fix that subcarrier-density bottleneck for fingertips / face, but multi-room full-body data will solve it for the 11 joints that today already show some signal.
Suggested data-collection plan
- 3 × 30-min sessions with the camera backed up so head→ankles fits in frame. Different rooms (or different times of day for the same room) to give the model spatial diversity. Vary movements: walk pattern, arm raises, sit/stand transitions, squats, reaches, lying down.
- Re-run
scripts/align-ground-truth.js(now streaming-loader-safe per fix(align): stream JSONL + support sensing_update format (unblocks ADR-079 P8) #641) to produce a multi-session paired set. - Train via the existing Candle pipeline on ruvultra's RTX 5080. Expected wall time: still well under a minute even for 30K samples / 1000 epochs.
- Re-evaluate. PCK@20 should approach the 35% target if the framing + variety land.
Optimizations available within the pipeline (do not require new data)
- LoRA cross-environment fine-tune (per ADR-079 P9). Today's encoder was random-initialized because the HF presence encoder's MLP architecture didn't match; with multi-room data we can train a real shared encoder first and then per-room LoRA adapters.
- Subcarrier attention weighting was already enabled (top-5: [33, 47, 50, 19, 16]).
- Stoer-Wagner min-cut multi-person separation enabled.
Artifacts shipped today (for context)
- Signed v0.0.1 binaries:
gs://cognitum-apps/cogs/arm/cog-pose-estimation-arm+.../x86_64/.... - Trained model:
models/wifi-densepose-pretrained.safetensors→pose_v1.safetensors(507 KB) +pose_v1.onnx(12 KB). - Benchmark log:
docs/benchmarks/pose-estimation-cog.md. - Live install:
/var/lib/cognitum/apps/pose-estimation/on cognitum-v0.
Acceptance criteria for closing this issue
- Multi-room paired dataset ≥ 30K samples at avg conf ≥ 0.7 produced.
- PCK@20 ≥ 35% on a held-out time window from a different session.
- PCK@50 ≥ 60%.
- Per-joint PCK@50 ≥ 30% for at least 13 of 17 joints (face joints can lag).
- Re-release
cog-pose-estimation@0.1.0with the new weights (no code change required — same Candle inference path, just better weights).
🤖 Generated with claude-flow
Activity
docs: repoint #640 references to #645 (original deleted, replaced) (#646
docs(adr): ADR-103 — learned multi-person counter (SOTA path) (#693)
feat(cog-person-count): train count_v1.safetensors — honest v0.0.1 (A…
feat(cog-person-count): release v0.0.1 — signed binaries on GCS, live…
feat(cog-person-count): wire `run` subcommand — v0.0.1 fully function…
feat(cog-person-count): v0.0.2 — K-fold validated, label smoothing + …
feat(cog-person-count): v0.0.2 — K-fold + label-smoothing + temperatu…
research(R8): RSSI-only person count retains 95% of full-CSI accuracy (…
feat(tools/ruview-mcp): M2 — wire real inference via cog health (#706)
ruvnet commentedon Jun 11, 2026
Status update (2026-06-10) — forensic recheck, root causes, and a new collection session in progress
Work on ADR-152 §2.2 measurement (b) (benchmarking against the external WiFlow-STD model) forced a full audit of the paired-data pipeline this issue tracks. Three findings:
1. The original "92.9% PCK@20" figure is retracted (it was already disclaimed in the PR #535 audit; the forensic recheck now explains why it appeared). The surviving 69-sample April holdout (
models/wiflow-real/eval-holdout.jsonl) shows: the trainer'spck()used an absolute 0.2 image-unit threshold (not torso-normalized,scripts/train-wiflow-supervised.js:1123), the model emitted a constant pose (prediction std 0.0000), and the subject was nearly static — a trivial mean-pose predictor scores 100% under that protocol. Torso-normalized PCK@20 on the same holdout: 19.1%. This also directly answers the external reproducibility report in #509. Stale doc citations were removed (c54ec22da,42b261f80); full forensics inbenchmarks/wiflow-std/RESULTS.md.2. The 3.0% PCK plateau this issue describes is data-bound, as suspected — and the window under-delivery has a root cause. ADR-079's ~9k windows/30 min prediction assumes ~100 Hz CSI; window yield is just
csi_frames / 20(the aligner is non-overlapping and not lossy beyond its confidence filter). The May session ran at an effective ~12 Hz → 1,077 windows. Today's node (post-#985 self-ping firmware) measures 53.4 Hz → ~160 windows/min, so a 40-min session yields ~6k windows.3. A ≥2k-window multi-pose collection session is running right now (40 min, camera + ESP32-S3, MediaPipe conf>0.5 framing guidance, varied actions matching the WiFlow-STD action set). If it clears the bar, measurement (b) proceeds on ruvultra: WiFlow-STD (2.23M params, reproduced at ~96% PCK@20 in-domain on its own 360k-window dataset) fine-tuned on our ESP32 data with an input adapter, plus a from-scratch ablation — which will give this issue its first honest baseline number under torso-PCK@20 on a temporal split.
The user-guide tutorial was rewritten accordingly (CSI-rate check before recording, session-length math, two-checkerboard calibration step from ADR-152 §2.1.3, pose-variety guidance, and the eval-protocol checklist incl. the pred-variance sanity check).
🤖 Generated with claude-flow
junedmrjuned5-rgb commentedon Jul 7, 2026
Jubed
ruvnet commentedon Jul 18, 2026
The acceptance criterion remains unmet: multi-room paired data and honest held-out torso-PCK, without constant-pose or metric leakage. Keeping open; an ESP32 smoke test cannot resolve this data/model limitation.
docs(adr-187): mark P1-P3 done, verify acceptance criteria
docs: repoint #640 references to ruvnet#645 (original deleted, replac…
docs(adr): ADR-103 — learned multi-person counter (SOTA path) (ruvnet…
feat(cog-person-count): train count_v1.safetensors — honest v0.0.1 (A…
feat(cog-person-count): release v0.0.1 — signed binaries on GCS, live…
feat(cog-person-count): wire `run` subcommand — v0.0.1 fully function…
feat(cog-person-count): v0.0.2 — K-fold + label-smoothing + temperatu…
research(R8): RSSI-only person count retains 95% of full-CSI accuracy (…
feat(tools/ruview-mcp): M2 — wire real inference via cog health (ruvn…
docs(adr-187): mark P1-P3 done, verify acceptance criteria