AgentCare for iOS

Chat with your medical records on your iPhone or iPad—an Apple HealthKit App powered by on-device LLM.

Connect to your Health Recrod thru Health App • Sync your data from Prociders EHR

Share your Healthkit record with the App

Chat with your health records • View medications and labs • Track trends over time

Architecture

📋 Quick Links:

• 👤 For Users:
  • User Guide - How to use AgentCare
  • User Consent & Health Records Setup - Connecting your health records
  • Privacy Policy - Your data stays on your device
• 👨‍💻 For Developers:
  • Developer Setup - Building from source
  • Contributing - How to contribute
  • Troubleshooting - Common issues

Features

• Natural Language Interface: Ask questions about your health records in plain English
• On-Device AI: Uses MLX Swift with Qwen2.5-3B-Instruct-4bit model for completely private, local inference
• HealthKit Integration: Direct access to clinical records (FHIR R4)
• Trend Visualization: Automatic chart generation for lab results and vital signs over time
• Smart Data Filtering: Intelligently prioritizes relevant information based on your question
• Privacy First: All processing happens on your device - no cloud, no data collection

Data Types Supported

• 💊 Medications and prescriptions
• ⚠️ Allergies and intolerances
• 🧪 Lab results and tests
• 💉 Immunizations and vaccines
• 🩺 Medical conditions and diagnoses
• 🏥 Procedures and treatments
• ❤️ Vital signs (blood pressure, heart rate, weight, etc.)

Requirements

• iOS 16.0+
• Xcode 15.0+
• Physical iPhone/iPad with Metal GPU (MLX does not work on simulator)
• ~3GB free storage for model download
• Health records synced to Apple Health app (see setup below)

User Consent & Health Records Setup

Understanding Your Data

AgentCare is completely privacy-first:

• ✅ All data stays on your device
• ✅ No cloud uploads or external API calls
• ✅ You control access via iOS permissions
• ✅ HealthKit data never leaves your device
• ✅ Revoke access anytime in iOS Settings

Before Using AgentCare

AgentCare reads your health records through Apple Health app. You need to connect your healthcare provider's records to Apple Health first.

Step 1: Connect Your Healthcare Provider to Apple Health

1. Open Apple Health App on your iPhone
2. Tap your profile picture (top right)
3. Tap Health Records
4. Tap Add Account or Get Started
5. Search for your healthcare provider (hospital, clinic, lab)
6. Sign in with your patient portal credentials (e.g., MyChart, Epic, Cerner)
7. Authorize Apple Health to download your records

Supported Providers: 800+ healthcare organizations in the US support Apple Health Records, including:

• Major hospital systems (Kaiser, Mayo Clinic, Cleveland Clinic)
• Lab services (Quest, LabCorp)
• Urgent care chains (CVS MinuteClinic)
• University health systems

What Gets Synced:

• Medications and prescriptions
• Allergies and reactions
• Lab results (A1C, cholesterol, etc.)
• Immunizations
• Medical conditions/diagnoses
• Procedures
• Vital signs

Data Format: Apple Health uses FHIR R4 (Fast Healthcare Interoperability Resources) - an international standard for health records.

Step 2: Grant AgentCare Permission

When you first launch AgentCare:

1. Permission Dialog Appears: iOS shows HealthKit authorization
2. Review Data Types: See exactly what AgentCare wants to read
3. Grant or Deny:
   • Allow All - Recommended for full functionality
   • Select Individual Types - Choose specific data types
   • Don't Allow - App won't work without health record access

What AgentCare Requests:

• ✅ Read-only access to clinical records (FHIR data)
• ❌ No write access - AgentCare cannot modify your health data
• ❌ No fitness data - Only clinical records, not steps/workouts

Step 3: Verify Connection

After granting permission:

1. Open AgentCare
2. Wait for model download (first time only)
3. Ask a test question: "What medications am I taking?"
4. If you see your actual medications → ✅ Connected!
5. If you see "No active medications" → Check permissions

Managing Permissions

View/Change Permissions:

1. iOS Settings → Privacy & Security → Health → agentcare
2. Toggle individual data types on/off
3. Changes take effect immediately

Revoke All Access:

1. iOS Settings → Privacy & Security → Health → agentcare
2. Toggle off all permissions OR delete the app

Your Rights

• Right to Access: You control what AgentCare can read
• Right to Revoke: Remove access anytime
• Right to Delete: Uninstall removes all local data
• Right to Know: Code is open-source - verify privacy claims yourself

Common Questions

Q: Does my healthcare provider know I'm using AgentCare? A: No. AgentCare reads data from Apple Health, not directly from your provider.

Q: Can AgentCare see health data I manually entered in Apple Health? A: No. AgentCare only reads clinical records (FHIR data) synced from healthcare providers, not manually-entered data.

Q: What if I don't have health records in Apple Health? A: You won't be able to use AgentCare. You must first connect your healthcare provider to Apple Health (see Step 1 above).

Q: Can AgentCare access health data from other apps (fitness trackers, etc.)? A: No. AgentCare only requests access to clinical records, not fitness/activity data.

Q: Will my health records be used to train AI models? A: No. All processing happens on-device. Your data never leaves your phone.

Data Flow

Here's how your health data flows (all on-device):

┌─────────────────────────────────────────────────────────────────┐
│                    YOUR HEALTHCARE PROVIDER                     │
│              (Hospital, Clinic, Lab, etc.)                      │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             │ FHIR R4 via Patient Portal
                             │ (You authorize this connection)
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                      APPLE HEALTH APP                           │
│                  (Built into iOS, Apple-managed)                │
│                                                                 │
│  Stores: Medications, Allergies, Labs, Conditions, etc.        │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             │ HealthKit API (Read-Only)
                             │ (You authorize this in iOS Settings)
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                         AGENTCARE                               │
│                    (This App, On-Device)                        │
│                                                                 │
│  1. Reads health records via HealthKit                         │
│  2. Processes with local LLM (MLX/Qwen)                        │
│  3. Generates natural language responses                        │
│  4. Never sends data off device                                │
└─────────────────────────────────────────────────────────────────┘

NO CLOUD ❌  |  NO SERVERS ❌  |  NO API CALLS ❌  |  NO DATA COLLECTION ❌

Key Points:

• Your provider → Apple Health: Direct FHIR connection (you authorize)
• Apple Health → AgentCare: iOS HealthKit permission (you control)
• AgentCare → Cloud: Never happens
• All AI processing: On your device using MLX

Setup

1. Clone the Repository

git clone https://github.com/yourusername/agentcare-ios.git
cd agentcare-ios/agentcare

2. Open in Xcode

open agentcare.xcodeproj

3. Configure Dependencies

The project uses Swift Package Manager for MLX Swift dependencies. These should resolve automatically when you open the project, but if needed:

1. In Xcode, go to File → Add Package Dependencies
2. Add the following package:
   • URL: https://github.com/ml-explore/mlx-swift-lm
   • Branch: main
3. Select both products:
   • MLXLLM
   • MLXLMCommon

The packages will be fetched and built automatically. This may take 2-5 minutes on first build.

4. Configure Signing & Capabilities

1. Select the agentcare target in Xcode
2. Go to Signing & Capabilities tab
3. Set your Team and Bundle Identifier

Required Capabilities (already configured in agentcare.entitlements):

• ✅ HealthKit - Read access to clinical records
• ✅ Health Records - FHIR R4 data access
• ✅ Siri - App Intents support (optional)
• ✅ App Groups - For Siri/Shortcuts communication

5. Update App Group (Optional)

If you want to use Siri/Shortcuts, update the app group identifier:

1. In Signing & Capabilities, click on App Groups
2. Change group.com.xxx.agentcare to your own identifier
3. Update the same in AskHealthQuestionIntent.swift:30

6. Build and Run

IMPORTANT: Must use a physical iPhone/iPad - MLX does not work on simulator.

1. Select your physical device in Xcode
2. Build and run (⌘R)
3. Grant HealthKit permissions when prompted

First Run

Before First Launch

Prerequisites:

1. ✅ Health records synced to Apple Health app (see "User Consent & Health Records Setup" above)
2. ✅ Physical iPhone/iPad connected to Xcode
3. ✅ At least 3GB free storage

Initial Setup Flow

1. HealthKit Permissions (First time only)

• iOS will show permission dialog
• Tap "Allow" to grant access to clinical records
• See "User Consent & Health Records Setup" section for details on what you're authorizing
• Without this permission, the app cannot function

2. Model Download (First time only)

• Automatic: Model downloads on first app launch (~1.9GB)
• Duration: 3-10 minutes depending on WiFi speed
• Location: Cached locally in app container (not in iCloud)
• Progress: Check Xcode console for download progress

3. Shader Compilation (First time only)

• First inference takes 10-30 seconds for one-time Metal shader compilation
• Subsequent responses will be 2-5 seconds

4. Ready to Use

• When you see "🎉 MLX ready for fast inference!" in console
• Ask your first question: "What medications am I taking?"

Console Logs to Monitor

🔄 Loading MLX model on physical device...
📥 Downloading model: mlx-community/Qwen2.5-3B-Instruct-4bit
📊 Download progress: 25%
✅ Model loaded, initializing ChatSession...
🔥 Warming up Metal shaders...
✅ Metal shaders compiled and cached!
🎉 MLX ready for fast inference!

Usage Examples

Basic Questions:

• "What medications am I taking?"
• "Do I have any allergies?"
• "What was my last A1C result?"

Trend Analysis:

• "Show me my A1C trend over time"
• "How has my blood pressure changed?"
• "Track my weight over the past year"

Follow-up Context:

• The app maintains conversation history for contextual follow-ups
• Example: "What about cholesterol?" after asking about lab results

Technology Stack

• MLX Swift: Apple's ML framework for local LLM inference
• HealthKit/FHIR: Direct access to iOS clinical records
• SwiftUI: Modern declarative UI
• Charts: Native trend visualization

Data Flow

User Question
    ↓
ChatViewModel (UI State)
    ↓
HealthAgent (Orchestrator)
    ├─→ Keyword Analysis → Tools Selection
    ├─→ ClinicalRecordsManager → HealthKit/FHIR → Health Records
    ├─→ Build Context (records + history)
    └─→ LLMService
         ├─→ Fresh ChatSession (stateless)
         ├─→ MLX Model (Qwen 3B, on-device)
         └─→ Generate Response
              ↓
         Natural Language Answer
              ↓
         Chart Generation (if trend question)
              ↓
ChatView (Display to User)

Project Structure

agentcare/
├── App/                    # App entry point
├── Models/                 # Data models (ChatMessage, ChartData)
├── Views/                  # SwiftUI views (ChatView, PermissionsView)
├── ViewModels/            # View models (ChatViewModel)
├── Services/              # Core services
│   ├── LLMService.swift          # MLX model management & generation
│   ├── HealthAgent.swift         # Tool calling & orchestration
│   ├── ClinicalRecordsManager.swift  # HealthKit/FHIR integration
│   └── SafetyFilter.swift        # Safety guardrails (optional)
├── Entities/              # AppIntent entities for Siri
├── Intents/               # Siri Shortcuts support
└── Resources/             # Entitlements, assets

Key Files:
- LLMService.swift: Stateless session management, conversation history
- HealthAgent.swift: Keyword-based tool selection, smart data filtering
- ChatViewModel.swift: Chat state management, message handling
- ChatView.swift: Message bubbles, charts, keyboard handling

Technical Details

Memory Management:

• Stateless generation (fresh session per query)
• Limited conversation history (1 Q&A pair for 3B model)
• Truncated prompts to prevent memory crashes on mobile
• Smart data filtering to reduce context size

Model:

• Qwen2.5-3B-Instruct-4bit (mlx-community)
• Quantized for mobile performance
• ~1.9GB disk space, ~2-3GB RAM during inference
• Alternative models supported: Change modelIdentifier in LLMService.swift:135
  • Smaller: mlx-community/Llama-3.2-1B-Instruct-4bit (~0.5GB, lower quality)
  • Larger: mlx-community/Qwen2.5-7B-Instruct-4bit (~4.5GB, better quality, more RAM)

Safety

This app provides educational information only about your documented health records. It does NOT:

• Diagnose medical conditions
• Recommend treatments or medications
• Advise on medication changes
• Replace healthcare providers

Always consult your healthcare provider for medical advice.

Privacy

Quick Summary:

• ✅ All data processing happens on-device
• ✅ No cloud services or API calls
• ✅ No data collection or telemetry
• ✅ HealthKit data never leaves your device
• ✅ Open source - verify privacy claims yourself

📄 Read Full Privacy Policy →

Troubleshooting

"MLX requires physical device" Warning

Problem: App shows mock responses instead of real AI Solution: MLX requires Metal GPU. Run on physical iPhone/iPad, not simulator.

Model Download Fails

Problem: Model download errors or hangs Solutions:

• Ensure stable WiFi connection (model is ~1.9GB)
• Check free storage space (need ~3GB)
• Restart app to retry download
• Check Xcode console for specific error messages

App Crashes After 3-4 Questions

Problem: Memory pressure from 3B model Current Status: Mitigated with stateless sessions and limited history If it still happens:

• Tap the clear/reset button (↻) in header
• Close and reopen the app
• Consider device with more RAM (8GB+ recommended)

First Inference Takes 30+ Seconds

Expected: First generation includes one-time Metal shader compilation After First Time: Subsequent generations should be 2-5 seconds If Always Slow: Check console for errors or memory warnings

HealthKit Permissions Denied

Problem: Can't access health records Solution:

1. Settings → Privacy & Security → Health → agentcare
2. Enable "Health Records" permission
3. Restart the app

No Health Records Found

Problem: App says "No medications found" or similar, even after granting permissions Common Causes:

1. No records in Apple Health

   • Open Apple Health app → Browse → Health Records
   • If empty, you need to connect your healthcare provider first
   • See "User Consent & Health Records Setup" section above

2. Records not synced yet

   • Apple Health syncs periodically (not real-time)
   • Force sync: Open Apple Health → tap provider → pull to refresh
   • Can take a few minutes for initial sync

3. Provider doesn't support Apple Health

   • Check: https://www.apple.com/healthcare/health-records/
   • Not all providers support FHIR integration

4. Wrong permission granted

   • AgentCare needs "Clinical Records" access, not just "Health Data"
   • Check iOS Settings → Health → agentcare → ensure clinical types are enabled

Build Errors with MLX Packages

Problem: Swift Package resolution fails Solutions:

1. File → Packages → Reset Package Caches
2. File → Packages → Update to Latest Package Versions
3. Clean build folder (⌘⇧K) and rebuild
4. Check network/firewall isn't blocking GitHub

Code Signing Issues

Problem: "Failed to register bundle identifier" Solution: Change bundle ID to your own unique identifier in project settings

License

MIT License - see LICENSE file for details

Contributing

Contributions welcome! Please open an issue or submit a pull request.

Acknowledgments

• MLX Swift - Apple's ML framework
• Qwen2.5 - Base language model
• Apple HealthKit team for FHIR integration