HandPoseDetection

Real-time hand pose detection for iOS using Apple's Vision framework. Detects Point, Peace, Pinch, and Hand poses with angle-based joint analysis, window-based temporal smoothing, and a protocol-oriented AVFoundation/Vision pipeline.

Screenshots & Demo

Live demo of real-time hand pose detection with skeleton overlay and smoothed pose labels:

Hand-Point-Detection-Demo-Video.mov

Features

• Real-time hand skeleton visualization with 21 joints
• Angle-based pose recognition (Point, Peace, Pinch, Hand)
• Window-based smoothing to reduce flickering between poses
• Protocol-oriented AVFoundation + Vision pipeline
• AsyncStream for continuous frame processing

Pose Detection

• Live camera preview with hand skeleton overlay
• Classified pose label (Point, Peace, Pinch, Hand)
• Dedicated states for no hand detected, incomplete hand, and camera errors

Installation

1. Clone the repository:

git clone https://github.com/YvonneG-Dev/HandPoseDetection.git

2. Open the project in Xcode
3. Run on a physical device (front camera required)

Architecture

The app follows MVVM, with a protocol-oriented pipeline for both vision capture and pose classification.

• VisionViewModel (@Observable) exposes a single handPoseState that drives a fully declarative PoseDetectionView, and is constructed via dependency injection for testability
• AVFoundationCameraController and AppleVisionHandPointDetector implement the CameraController and HandPointDetector protocols and communicate via delegate protocols, matching AVFoundation/Vision's native callback style
• VisionPipeline (protocol) bridges these delegate callbacks into a single AsyncStream<HandJointDetectionResult>, so everything above it only deals with async/await
• PoseClassificationPipeline (protocol) coordinates a PoseDetector and a PoseSmoother in sequence, decoupling gesture classification from temporal stabilization
• GeometryBasedPoseDetector classifies gestures via atan2-based joint angles — deterministic and unit-testable with no camera or Vision request involved
• WindowBasedPoseSmoother stabilizes the classified pose over a sliding window of recent frames, falling back to the last known joint positions when the current frame lacks them
• Dependency injection throughout — every layer (VisionPipeline, PoseClassificationPipeline, VisionViewModel) is constructed via a designated initializer accepting protocol types, with a convenience initializer wiring up production defaults
• Errors as stream values — camera and detection failures are modeled as result cases rather than thrown errors, since an AsyncThrowingStream would terminate the pipeline on the first bad frame

flowchart TD
    subgraph VisionGroup["Vision Capture"]
        CC["CameraController (Protocol)<br/><i>AVFoundationCameraController</i>"]
        HPD["HandPointDetector (Protocol)<br/><i>AppleVisionHandPointDetector</i>"]
        VP["VisionPipeline (Protocol)<br/><i>CameraVisionPipeline</i>"]
        CC --> VP
        HPD --> VP
    end

    subgraph ClassificationGroup["Pose Classification"]
        PCP["PoseClassificationPipeline (Protocol)<br/><i>SequentialPoseClassificationPipeline</i>"]
        PD["PoseDetector (Protocol)<br/><i>GeometryBasedPoseDetector</i>"]
        PS["PoseSmoother (Protocol)<br/><i>WindowBasedPoseSmoother</i>"]
        PCP --> PD
        PCP --> PS
    end

    VP -- "AsyncStream" --> VVM
    VVM --> PCP
    PCP -- "PoseState" --> VVM

    VVM["VisionViewModel"] ---> PDV["PoseDetectionView"]

    style VisionGroup fill:#e1f5fe
    style ClassificationGroup fill:#e8f5e9
    style VVM fill:#fce4ec
    style PDV fill:#fff3e0

Loading

Testing

The project includes unit tests for the core domain logic:

• GeometryBasedPoseDetectorTest — angle-based classification, including confidence thresholds and degenerate bounding boxes
• WindowBasedPoseSmootherTest — window stability, ambiguous windows, and joint continuity through outlier frames
• BoundingBoxTest — geometry edge cases, including degenerate and negative dimensions
• HandJointTest — the non-failable initializer, the failable initializer's nil case, bounding box construction from joint positions, and finger chain structure
• VisionViewModelTest — verifies VisionViewModel correctly wires the vision stream to pose classification, using mocked VisionPipeline and PoseClassificationPipeline

All dependencies are injected behind protocols, so classification, smoothing, and the ViewModel can each be tested without a camera or Vision request.

Known Limitations

The app tracks a single hand (maximumHandCount = 1) and targets iPhone in portrait orientation only. Pose classification is smoothed over a sliding window. Hand is a fallback state for a detected hand that matches none of the specific gestures, rather than a positively classified pose — the generic .hand fallback also applies no confidence threshold, as confidence is only relevant to specific gesture classifications rather than hand presence itself.

Requirements

• iOS 17+
• Xcode 15+
• Physical device (front camera required)

Note

This is a personal portfolio project. Pull requests will not be accepted, but feedback and suggestions via Issues are welcome.