AR-powered object detection and contextual understanding for physical systems.
Reality Layer transforms your iPhone camera into an intelligent analyzer. Point at a breaker panel, HVAC system, or other equipment and get real-time AR overlays identifying components, safety warnings, and diagnostic guidance—powered by Google's Gemini Pro Vision.
- Mac with Xcode 15+ installed
- iPhone with ARKit support (iPhone 6s or newer, iOS 15+)
- Python 3.11+ installed
- USB cable to connect iPhone to Mac
# 1. Clone the repository
git clone https://github.com/Dparent97/Reality-layer.git
cd Reality-layer
# 2. Start backend (creates venv, installs deps, runs in mock mode)
./backend/run.sh
# 3. Get your Mac's IP address (note this for step 5)
ipconfig getifaddr en0# 4. Open iOS project in Xcode
open RealityLayer.xcodeproj
# 5. In Xcode:
# - Select your connected iPhone as the build target
# - Update Config.swift: change localhost to your Mac's IP (from step 3)
# - Press Cmd+R to build and run
- Grant Permissions: Tap "Allow" for camera and AR access
- Point Camera: Aim at any object (breaker panel works great for demo)
- Tap "Analyze Scene": Wait 1-2 seconds for mock response
- See AR Labels: 3D text labels appear anchored in space
- Clear & Repeat: Tap the X button to clear and try again
Backend health check:
curl http://localhost:8000/health
# Expected: {"status":"healthy","version":"0.1.0","environment":"development"}Test analysis endpoint:
curl -X POST http://localhost:8000/analyze \
-F "file=@/path/to/any/image.jpg" \
-F "mock=true"┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ iPhone │────▶│ Backend │────▶│ Gemini │
│ AR Camera │◀────│ FastAPI │◀────│ Pro Vision │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
│ Camera frame │ Image + prompt │ JSON response
│ as JPEG │ │ with objects
▼ ▼ ▼
┌─────────────────────────────────────────────────────┐
│ AR Overlays: 3D labels anchored to detected │
│ objects with safety warnings and descriptions │
└─────────────────────────────────────────────────────┘
Use Cases:
- Electrical panel identification and safety warnings
- HVAC component labeling
- Industrial equipment documentation
- Educational overlays for physical systems
| Component | Technology | Purpose |
|---|---|---|
| iOS Client | Swift, SwiftUI, ARKit, RealityKit | Camera capture, AR rendering, user interface |
| Backend API | Python, FastAPI, Pydantic | Image processing, API gateway, mock mode |
| AI Service | Google Vertex AI (Gemini Pro Vision) | Multimodal image analysis and object detection |
See docs/ARCHITECTURE.md for detailed system design.
cd backend
# Create virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Run in development mode (mock enabled)
ENVIRONMENT=development DEBUG=true uvicorn main:app --reload
# Run tests
pytest -vEnvironment Variables:
| Variable | Default | Description |
|---|---|---|
GCP_PROJECT_ID |
(required for real AI) | Google Cloud Project ID |
ENVIRONMENT |
development | development, staging, production |
DEBUG |
false | Enable debug mode and API docs |
- Open
RealityLayer.xcodeprojin Xcode - Select your development team (Signing & Capabilities)
- Update
ios/Config.swiftwith backend URL for device testing - Build to physical device (Simulator doesn't support ARKit)
See ios/README.md for detailed iOS setup.
Analyze an image frame and return detected objects with AR positioning.
Request:
curl -X POST http://localhost:8000/analyze \
-H "Content-Type: multipart/form-data" \
-F "file=@image.jpg" \
-F "mock=true" # Optional: use mock modeResponse:
{
"session_id": "mock-abc12345",
"detected_objects": [
{
"id": "obj_1",
"label": "Breaker Panel",
"type": "label",
"position_2d": [0.5, 0.5],
"description": "Main distribution panel. Ensure main breaker is OFF before servicing."
},
{
"id": "obj_2",
"label": "Main Breaker (200A)",
"type": "warning",
"position_2d": [0.5, 0.2],
"description": "High Voltage! Do not touch terminals."
}
],
"system_context": "Electrical Distribution System",
"processing_time_ms": 1023.45
}Health check endpoint.
curl http://localhost:8000/healthInteractive API documentation (available when DEBUG=true).
See docs/DEMO_SCRIPT.md for:
- What to demonstrate
- Talking points
- Expected behavior
- Known limitations
Reality-layer/
├── backend/
│ ├── main.py # FastAPI application
│ ├── config.py # Environment configuration
│ ├── gemini_service.py # Gemini Vision integration
│ ├── run.sh # One-command startup script
│ └── requirements.txt # Python dependencies
├── ios/
│ ├── ContentView.swift # Main UI
│ ├── ARViewContainer.swift# AR rendering
│ ├── NetworkManager.swift # API client
│ ├── Config.swift # iOS configuration
│ └── README.md # iOS-specific docs
├── docs/
│ ├── ARCHITECTURE.md # System design
│ └── DEMO_SCRIPT.md # Demo guide
└── README.md # This file
# Check Python version
python --version # Should be 3.11+
# Try manual setup
cd backend
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --reload- Ensure backend is running:
curl http://localhost:8000/health - Get Mac's IP:
ipconfig getifaddr en0 - Update
ios/Config.swiftto use IP instead of localhost - Ensure Mac and iPhone are on same Wi-Fi network
- Check Mac firewall allows incoming connections on port 8000
- ARKit requires a physical device (not Simulator)
- Ensure camera permissions are granted
- iPhone must have A12 chip or newer
- Good lighting helps AR tracking
- Select project in Xcode
- Go to Signing & Capabilities tab
- Select your Apple Developer account under "Team"
- If no team, add Apple ID in Xcode → Settings → Accounts
- Real-time streaming analysis
- Offline caching of common patterns
- Custom prompt domains (industrial, automotive, etc.)
- Multi-user session sharing
- Voice-guided instructions
- Fork the repository
- Create a feature branch:
git checkout -b feature/amazing-feature - Commit your changes:
git commit -m 'Add amazing feature' - Push to the branch:
git push origin feature/amazing-feature - Open a Pull Request
MIT License - see LICENSE file for details.
Need help? Check the docs/ folder or open an issue on GitHub.