TAKE BACK THE HOUSE

See the table. Understand the shoe. Feel the decision.

An explainable computer-vision and dual-channel haptic training system built for two Raspberry Pi nodes.

Open the live demo · Run locally · Architecture · 3-minute pitch · Hardware

The idea

Take Back the House turns a blackjack training table into a real-time perception and decision loop:

1. A spotter node sees exposed cards through Camera Module 3.
2. A compact detector classifies the cards; temporal consensus prevents one card from being counted once per video frame.
3. An explainable policy maintains the Hi-Lo running/true count and recommends basic strategy.
4. Two deliberately different haptic channels encode play and shoe-state guidance.
5. A second receiver node authenticates the message and drives a DRV2605L motor.

The hackathon build is an honest training prototype. It runs immediately with a synthetic camera source, while the same interfaces support Picamera2, a custom Ultralytics checkpoint, and real DRV2605L hardware. It does not bundle or claim a trained card model that does not exist.

Run it in 60 seconds

The public GitHub Pages demo uses a browser-native simulation so all three interactive tabs work without a server. The local version below uses the full FastAPI engine and is the reference implementation for Raspberry Pi integration.

git clone https://github.com/kai405/TakeBacktheHouse.git
cd TakeBacktheHouse
uv sync --extra dev
uv run takeback-demo

Open http://localhost:8000. The dashboard begins with a live synthetic camera round. No camera, model weights, API key, or Raspberry Pi is required.

Use the three tabs:

• Live Lab — camera detections, running/true count, policy decisions, haptic codes, telemetry, and pitch controls.
• Trainer — choose the correct play and enter the running count. Accuracy grades both skills independently and together.
• Records — inspect count drift by true-count bucket and compare unaided answers with the decision pipeline's supported coverage.

Why the Trainer matters

A normal blackjack trainer grades only the move. That hides the hard part of a team-counting exercise: maintaining shoe state while still playing accurately. This trainer asks for both.

• Strategy accuracy checks hit, stand, double, split, or surrender against a transparent six-deck S17 policy.
• Count accuracy checks the user's running count after every exposed card.
• Combined accuracy scores both without blending in the system's own result.
• Mistakes surfaced shows how many submitted decisions the assistive loop would have corrected.
• Accuracy by true count reveals whether errors cluster when the shoe state changes.

The 100% shown under Pipeline coverage means every supported strategy/count output maps to an explanation and a haptic command. It is not a win-rate claim.

System architecture

flowchart LR
    Camera[Camera Module 3] --> Detector[52-class YOLO adapter]
    Detector --> Consensus[Temporal consensus tracker]
    Consensus --> Count[Hi-Lo shoe state]
    Consensus --> Strategy[Explainable S17 policy]
    Count --> Advisor[Confidence-gated unit policy]
    Strategy --> Encoder[Dual-channel haptic encoder]
    Advisor --> Encoder
    Encoder --> Auth[HMAC + sequence envelope]
    Auth --> WiFi[Wi-Fi / TCP JSONL]
    WiFi --> Receiver[Bettor Pi receiver]
    Receiver --> DRV[DRV2605L + ERM motor]
    Count --> UI[Live FastAPI dashboard]
    Strategy --> UI
    Advisor --> UI

Loading

The two-node split is real even when only one Pi is available for judging. Run the receiver on a laptop during the demo, then move the same process to a second Zero 2 W without changing the protocol.

What is implemented

Capability	Status	What that means
Hi-Lo running and true count	Working	Multi-deck shoe state, penetration, history, idempotent observations
Basic strategy	Working	Pair, soft, hard, surrender, split, and double rules for six-deck S17
Count-aware Trainer	Working	Real grading, streaks, errors, count buckets, and session records
Dashboard	Working	Responsive UI, WebSocket state, autoplay, scenario injection, reduced-motion support
Two-Pi transport	Working	Reconnectable TCP, versioned JSONL, HMAC-SHA256, freshness and replay checks
Haptic vocabulary	Working	Distinct play/unit patterns plus console and DRV2605L drivers
Temporal vision consensus	Working	Confidence gating, IoU association, stable-track emission, expiry
Synthetic camera	Working	Deterministic, no-hardware path used in the public demo
Picamera2 adapter	Implemented	Requires Raspberry Pi OS and Camera Module 3 to validate physically
Ultralytics detector adapter	Implemented	Requires separately trained 52-class weights; training/export path is included
Hardware integration	Adapter ready	DRV2605L code is present; final motor calibration requires the physical wearable

AI/ML usage

AI is central to the perception layer, not used as a label for deterministic math.

1. Card perception

YoloCardDetector accepts a custom Ultralytics checkpoint and converts bounding boxes into a shared RawDetection type. The included dataset descriptor uses 52 rank/suit classes, and ml/train.py trains and exports an ONNX artifact.

2. Temporal inference

Video models produce repeated and occasionally unstable boxes. TemporalConsensusTracker associates same-class detections by intersection-over-union and emits a card only after multi-frame agreement. This is what makes the count safe to update from a stream.

3. Confidence-aware decisions

Perception confidence propagates into the unit policy. Low-confidence input can hold or reduce a recommendation, but cannot escalate it. The deterministic basic-strategy policy stays explicit because explainability is more valuable than pretending a lookup policy is ML.

Read the limitations and evaluation plan in docs/MODEL_CARD.md.

Two-device demo

Terminal 1 — receiver node (laptop or bettor Pi):

export TBH_SIGNAL_SECRET="replace-with-a-shared-secret"
uv run takeback-bettor --driver console

Terminal 2 — synthetic spotter node:

export TBH_SIGNAL_SECRET="replace-with-a-shared-secret"
uv run takeback-spotter --demo --bettor-host 127.0.0.1 --rounds 10

On the Pi, replace --driver console with --driver drv2605l. See the exact wiring and Raspberry Pi setup in docs/HARDWARE.md.

Live camera path

The repository intentionally does not ship unverified weights. To supply your own:

uv sync --extra vision
uv run python ml/train.py --data ml/cards.yaml --device mps
uv run takeback-spotter \
  --model runs/takebackhouse/card-detector/weights/best.pt \
  --bettor-host BETTOR_PI.local

The camera path counts every stable face-up detection. Dealer/player zone calibration is listed as the next hardware milestone in the model card.

Haptic language

The design prevents two meanings from competing on one tactile channel.

Channel	Command	Pattern
Strategy / forearm	Hit	One crisp tap
Strategy / forearm	Stand	Sustained confirmation
Strategy / forearm	Double	Two even taps
Strategy / forearm	Split	Three sharp taps
Strategy / forearm	Surrender	Descending pair
Shoe state / torso	Enter	Rising pair
Shoe state / torso	Increase	Rapid triple
Shoe state / torso	Decrease	Slow double
Shoe state / torso	Hold	Soft confirmation
Shoe state / torso	Sit out	Long low pulse

Effect IDs, wire schema, validation, and security properties are documented in docs/SIGNAL_PROTOCOL.md.

Repository map

src/takebackhouse/
├── counter.py       # idempotent Hi-Lo state
├── strategy.py      # explainable S17 policy
├── betting.py       # confidence-gated training units
├── vision.py        # YOLO, Picamera2, temporal consensus
├── protocol.py      # signed, versioned envelopes
├── network.py       # asyncio spotter ↔ receiver transport
├── haptics.py       # pulse vocabulary + drivers
├── engine.py        # live-lab orchestration
├── trainer.py       # dual-accuracy trainer and analytics
├── dashboard.py     # FastAPI + WebSocket application
├── spotter.py       # spotter process
├── bettor.py        # receiver process
└── static/          # responsive judge-facing interface

API surface

Route	Purpose
GET /api/state	Live perception/count/policy snapshot
POST /api/demo/round	Advance the synthetic table
POST /api/demo/hit	Add a player card
POST /api/demo/expose	Add any face-up table card
GET /api/trainer/state	Current hand, session score, and records
POST /api/trainer/answer	Grade play plus running-count answer
POST /api/trainer/next	Deal the next training hand
WS /ws	Push live dashboard updates

FastAPI also exposes interactive schema documentation at /docs.

Verification

uv run pytest
uv run ruff check src tests ml
node --check src/takebackhouse/static/app.js

The suite covers counting, duplicate protection, hard/soft/pair strategy, confidence gating, protocol tampering and expiry, temporal consensus, haptic mappings, engine lifecycle, Trainer records, and HTTP flows.

Rubric fit

Criterion	Evidence in this repo
Innovation — 25	Two-role perception loop; body-separated haptic vocabulary; count + strategy dual-skill Trainer
Technical execution — 25	Modular typed Python; two-node protocol; replay defense; adapters; 39 tests; CI
AI usage — 20	Real detector interface, training/export path, temporal inference, confidence propagation, honest model card
UX/UI — 15	Three polished responsive views; live cards; count visualization; keyboard controls; records by true count
Business potential — 15	Reusable perception-to-haptics platform for training and accessible tactile guidance

Responsible use

This project is a simulation and training prototype. Casino rules and local law can restrict electronic assistance and coordinated play. Do not deploy it in a venue or context where it is prohibited. The broader product thesis is a transparent visual-to-haptic decision-support platform for training and accessibility—not guaranteed gambling outcomes.

More documentation

• docs/ARCHITECTURE.md — components, data flow, failure handling
• docs/SIGNAL_PROTOCOL.md — authenticated wire and tactile protocol
• docs/HARDWARE.md — BOM, wiring, two-Pi setup, validation checklist
• docs/MODEL_CARD.md — ML scope, limitations, dataset and evaluation plan
• docs/PITCH.md — timed three-minute finalist pitch and demo choreography
• docs/BUSINESS.md — legitimate market wedge and expansion path

Built for the hackathon clock. Designed beyond it.