Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 48 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,49 @@ jobs:
build \
CODE_SIGNING_ALLOWED=NO

# Build Expo test app
build-expo:
name: Build Expo
runs-on: macos-15
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'

- name: Install dependencies
run: npm ci

- name: Build package
run: npm run prepare

- name: Install expo-test dependencies
run: cd expo-test && npm install

- name: Sync FluidAudio package
run: cd expo-test && npm run sync-fluidaudio

- name: Generate native project
run: cd expo-test && npx expo prebuild --platform ios --no-install

- name: Install CocoaPods
run: cd expo-test/ios && pod install

- name: Build iOS
run: |
xcodebuild \
-workspace expo-test/ios/expotest.xcworkspace \
-scheme expotest \
-configuration Debug \
-sdk iphonesimulator \
-destination 'platform=iOS Simulator,name=iPhone 16 Pro' \
build \
CODE_SIGNING_ALLOWED=NO

# Lint check
lint:
name: Lint
Expand Down Expand Up @@ -124,13 +167,15 @@ jobs:

- name: Verify exports
run: |
# Check that compiled files exist and export the expected symbols
node -e "
const pkg = require('./lib/commonjs/index.js');
const fs = require('fs');
const indexContent = fs.readFileSync('./lib/commonjs/index.js', 'utf8');
const expected = ['ASRManager', 'StreamingASRManager', 'VADManager', 'DiarizationManager', 'TTSManager', 'getSystemInfo', 'isAppleSilicon', 'cleanup'];
const missing = expected.filter(e => !pkg[e]);
const missing = expected.filter(e => !indexContent.includes(e));
if (missing.length) {
console.error('Missing exports:', missing);
process.exit(1);
}
console.log('All exports present');
console.log('All exports found in compiled output');
"
66 changes: 3 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,8 @@ React Native wrapper for [FluidAudio](../FluidAudio) - a Swift library for ASR,

## Requirements

- iOS 17.0+ / macOS 14.0+
- iOS 17.0+
- React Native 0.71+
- Apple Silicon (M1/M2/M3) - Intel Macs have limited support

## Installation

Expand All @@ -34,8 +33,6 @@ Then install pods:
cd ios && pod install
```

**Note:** Requires **arm64** architecture (Apple Silicon). Simulator builds only work on M1/M2/M3 Macs.

## Usage

### Basic Transcription
Expand Down Expand Up @@ -150,17 +147,11 @@ await tts.synthesizeToFile('Hello, world!', '/path/to/output.wav');
### System Information

```typescript
import { getSystemInfo, isAppleSilicon } from 'react-native-fluidaudio';
import { getSystemInfo } from 'react-native-fluidaudio';

const info = await getSystemInfo();
console.log(info.summary);
// e.g., "Apple M2 Pro, 16GB RAM, macOS 14.0"

if (await isAppleSilicon()) {
// Full ML model support available
} else {
// Intel Mac - some models may not work
}
// e.g., "Apple A17 Pro, iOS 17.0"
```

### Cleanup
Expand Down Expand Up @@ -196,67 +187,16 @@ await cleanup();

See [src/types.ts](./src/types.ts) for complete TypeScript definitions.


## Notes

### Model Loading

First initialization downloads and compiles ML models (~500MB total). This can take 20-30 seconds as Apple's Neural Engine compiles the models. Subsequent loads use cached compilations (~1 second).

### Intel Mac Support

Most ML models require Apple Silicon (ARM64). On Intel Macs:
- VAD works with CPU fallback
- ASR/Diarization may not work
- Use `isAppleSilicon()` to check before initializing

### TTS License

The TTS module uses ESpeakNG which is GPL licensed. Check license compatibility for your project.

## Architecture

This package supports both React Native architectures:

### New Architecture (Recommended)
- **TurboModules** for type-safe native module interface
- **JSI (JavaScript Interface)** for zero-copy audio buffer transfer
- **Codegen** for automatic type synchronization

Enable New Architecture in your app:
```bash
# iOS
cd ios && RCT_NEW_ARCH_ENABLED=1 pod install
```

### Legacy Architecture
- Falls back to Bridge-based modules automatically
- Audio data transferred as base64 strings
- Fully functional, slightly higher latency for large audio

### Zero-Copy API

When New Architecture is enabled, use `ArrayBuffer` methods for best performance:

```typescript
import { ASRManager, hasZeroCopySupport } from 'react-native-fluidaudio';

// Check if zero-copy is available
if (hasZeroCopySupport()) {
console.log('Using JSI zero-copy audio transfer');
}

const asr = new ASRManager();
await asr.initialize();

// Zero-copy transcription (New Architecture)
const audioBuffer = new ArrayBuffer(audioData.length);
const result = await asr.transcribeBuffer(audioBuffer, 16000);

// Legacy API still works (falls back automatically)
const result2 = await asr.transcribe(base64Audio, 16000);
```

## License

MIT
3 changes: 1 addition & 2 deletions __mocks__/react-native.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,8 @@ export const NativeModules = {
FluidAudioModule: {
getSystemInfo: jest.fn().mockResolvedValue({
isAppleSilicon: true,
isIntelMac: false,
platform: 'ios',
summary: 'Mock Apple Silicon Device',
summary: 'Mock iOS Device',
}),
initializeAsr: jest.fn().mockResolvedValue({
success: true,
Expand Down
1 change: 0 additions & 1 deletion __tests__/FluidAudio.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ describe('FluidAudio', () => {
const info = await getSystemInfo();

expect(info.isAppleSilicon).toBe(true);
expect(info.isIntelMac).toBe(false);
expect(info.platform).toBe('ios');
expect(mockModule.getSystemInfo).toHaveBeenCalledTimes(1);
});
Expand Down
1 change: 0 additions & 1 deletion __tests__/types.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ describe('Type Definitions', () => {
it('should have correct shape', () => {
const info: SystemInfo = {
isAppleSilicon: true,
isIntelMac: false,
platform: 'ios',
summary: 'Test device',
};
Expand Down
20 changes: 2 additions & 18 deletions example/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ Example React Native app demonstrating the `react-native-fluidaudio` package.

## Requirements

- macOS with Apple Silicon (M1/M2/M3/M4)
- Xcode 15+
- macOS with Xcode 15+
- Node.js 18+
- CocoaPods

Expand All @@ -29,23 +28,9 @@ npm start
npm run ios
```

## New Architecture

To enable TurboModules and JSI zero-copy:

```bash
# Clean and reinstall pods with New Architecture
cd ios
RCT_NEW_ARCH_ENABLED=1 pod install
cd ..

# Run the app
npm run ios
```

## Features Demonstrated

- **System Info**: Shows device capabilities and architecture detection
- **System Info**: Shows device capabilities
- **ASR**: Speech-to-text initialization
- **VAD**: Voice activity detection
- **Diarization**: Speaker identification
Expand All @@ -55,4 +40,3 @@ npm run ios

- First run downloads ~500MB of ML models
- Model compilation takes 20-30 seconds on first launch
- Requires Apple Silicon for full ML model support
41 changes: 41 additions & 0 deletions expo-test/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Learn more https://docs.github.com/en/get-started/getting-started-with-git/ignoring-files

# dependencies
node_modules/

# Expo
.expo/
dist/
web-build/
expo-env.d.ts

# Native
.kotlin/
*.orig.*
*.jks
*.p8
*.p12
*.key
*.mobileprovision

# Metro
.metro-health-check*

# debug
npm-debug.*
yarn-debug.*
yarn-error.*

# macOS
.DS_Store
*.pem

# local env files
.env*.local

# typescript
*.tsbuildinfo

# generated native folders
/ios
/android
70 changes: 70 additions & 0 deletions expo-test/App.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import { StatusBar } from 'expo-status-bar';
import { StyleSheet, Text, View } from 'react-native';
import { useEffect, useState } from 'react';
import { getSystemInfo } from '@fluidinference/react-native-fluidaudio';

export default function App() {
const [status, setStatus] = useState('Loading...');
const [info, setInfo] = useState(null);

useEffect(() => {
async function test() {
try {
setStatus('Calling getSystemInfo...');
const result = await getSystemInfo();

setInfo(result);
setStatus('Success!');
} catch (error) {
setStatus(`Error: ${error.message}`);
console.error('FluidAudio error:', error);
}
}
test();
}, []);

return (
<View style={styles.container}>
<Text style={styles.title}>FluidAudio Expo Test</Text>
<Text style={styles.status}>{status}</Text>
{info && (
<View style={styles.infoBox}>
<Text style={styles.infoText}>Platform: {info.platform}</Text>
<Text style={styles.infoText}>Apple Silicon: {info.isAppleSilicon ? 'Yes' : 'No'}</Text>
<Text style={styles.infoText}>Summary: {info.summary}</Text>
</View>
)}
<StatusBar style="auto" />
</View>
);
}

const styles = StyleSheet.create({
container: {
flex: 1,
backgroundColor: '#fff',
alignItems: 'center',
justifyContent: 'center',
padding: 20,
},
title: {
fontSize: 24,
fontWeight: 'bold',
marginBottom: 20,
},
status: {
fontSize: 16,
color: '#666',
marginBottom: 20,
textAlign: 'center',
},
infoBox: {
backgroundColor: '#f0f0f0',
padding: 20,
borderRadius: 10,
},
infoText: {
fontSize: 14,
marginBottom: 5,
},
});
31 changes: 31 additions & 0 deletions expo-test/app.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
{
"expo": {
"name": "expo-test",
"slug": "expo-test",
"version": "1.0.0",
"orientation": "portrait",
"icon": "./assets/icon.png",
"userInterfaceStyle": "light",
"newArchEnabled": false,
"splash": {
"image": "./assets/splash-icon.png",
"resizeMode": "contain",
"backgroundColor": "#ffffff"
},
"ios": {
"supportsTablet": true,
"bundleIdentifier": "com.fluidaudio.expotest",
"deploymentTarget": "17.0"
},
"android": {
"adaptiveIcon": {
"foregroundImage": "./assets/adaptive-icon.png",
"backgroundColor": "#ffffff"
},
"edgeToEdgeEnabled": true
},
"web": {
"favicon": "./assets/favicon.png"
}
}
}
Binary file added expo-test/assets/adaptive-icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added expo-test/assets/favicon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added expo-test/assets/icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added expo-test/assets/splash-icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions expo-test/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
import { registerRootComponent } from 'expo';

import App from './App';

// registerRootComponent calls AppRegistry.registerComponent('main', () => App);
// It also ensures that whether you load the app in Expo Go or in a native build,
// the environment is set up appropriately
registerRootComponent(App);
Loading
Loading