🎭 Khavee AI - VRM Avatar SDK

Transform VRM 3D avatars into interactive AI characters with expressions, animations, and voice.

Build immersive AI experiences with realistic 3D avatars that can talk, express emotions, and respond intelligently to users.

✨ Features

🎤 Real-time Voice Chat - OpenAI Realtime API with WebRTC (no backend needed)
👄 Automatic Lip Sync - MFCC-based phoneme detection syncs with AI speech
💬 Talking Animations - Auto-plays gestures during AI conversations
🎨 Facial Expressions - Control 30+ VRM expressions with smooth transitions
💃 Body Animations - Load and play Mixamo animations via simple URLs
🔍 RAG Support - Built-in vector search with Qdrant for knowledge bases
👁️ Natural Blinking - Randomized blinking for lifelike avatars
🛠️ Function Calling - OpenAI tools for custom functions and RAG
🎯 Simple API - URL-based animations, no complex setup
📦 Provider System - Plug-and-play OpenAI Realtime, Mock, or custom providers
🎭 Auto-Remapping - Mixamo animations work out of the box
💪 TypeScript - Full type safety and IntelliSense support
⚡ React Three Fiber - Built on the industry-standard 3D React framework

📦 Installation

Required Dependencies

First, install the core 3D rendering libraries:

npm install three @react-three/fiber @react-three/drei
# or
pnpm add three @react-three/fiber @react-three/drei
# or
yarn add three @react-three/fiber @react-three/drei

Install Khavee AI SDK

npm install @khaveeai/react @khaveeai/core
# or
pnpm add @khaveeai/react @khaveeai/core
# or
yarn add @khaveeai/react @khaveeai/core

Optional: Provider Packages

# For OpenAI Realtime API (voice chat + lip sync)
npm install @khaveeai/providers-openai-realtime

# For RAG (Retrieval-Augmented Generation)
npm install @khaveeai/providers-rag

# For development/testing (no API keys needed)
npm install @khaveeai/providers-mock

Peer Dependencies

The SDK requires these peer dependencies (most React projects already have them):

{
  "react": "^18.0.0 || ^19.0.0",
  "react-dom": "^18.0.0 || ^19.0.0",
  "three": "^0.160.0"
}

🚀 Quick Start

Basic VRM Avatar

import { Canvas } from '@react-three/fiber';
import { KhaveeProvider, VRMAvatar } from '@khaveeai/react';

export default function App() {
  return (
    <KhaveeProvider>
      <Canvas>
        <ambientLight intensity={0.5} />
        <directionalLight position={[10, 10, 5]} />
        
        <VRMAvatar 
          src="/models/character.vrm"
          position={[0, -1, 0]}
        />
      </Canvas>
    </KhaveeProvider>
  );
}

With Animations

const animations = {
  idle: '/animations/idle.fbx',        // Auto-plays on load
  walk: '/animations/walk.fbx',
  dance: '/animations/dance.fbx',
  talking: '/animations/talking.fbx',  // Played during AI speech
  gesture1: '/animations/gesture.fbx'  // Also played during speech
};

function App() {
  return (
    <KhaveeProvider>
      <Canvas>
        <VRMAvatar 
          src="/models/character.vrm"
          animations={animations}
          enableBlinking={true}              // Natural blinking
          enableTalkingAnimations={true}     // Gestures during speech
        />
      </Canvas>
    </KhaveeProvider>
  );
}

Note: Animations with 'talk', 'gesture', or 'speak' in the name are automatically played randomly when the AI is speaking.

With OpenAI Realtime (Voice Chat + Lip Sync)

"use client";
import { KhaveeProvider, VRMAvatar, useRealtime } from '@khaveeai/react';
import { OpenAIRealtimeProvider } from '@khaveeai/providers-openai-realtime';
import { Canvas } from '@react-three/fiber';
import { useMemo } from 'react';

function ChatInterface() {
  const { 
    isConnected, 
    connect, 
    disconnect, 
    sendMessage,
    conversation,
    chatStatus 
  } = useRealtime();

  return (
    <div>
      {!isConnected ? (
        <button onClick={connect}>🎤 Start Voice Chat</button>
      ) : (
        <div>
          <div>Status: {chatStatus}</div>
          <button onClick={() => sendMessage('Hello!')}>Say Hello</button>
          <button onClick={disconnect}>Disconnect</button>
          
          {/* Conversation history */}
          {conversation.map((msg, i) => (
            <div key={i}>{msg.role}: {msg.text}</div>
          ))}
        </div>
      )}
    </div>
  );
}

export default function App() {
  // Memoize provider to prevent recreation
  const realtime = useMemo(() => 
    new OpenAIRealtimeProvider({
      apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY!,
      voice: 'coral',
      instructions: 'You are a helpful AI assistant.',
    }), []
  );

  return (
    <KhaveeProvider config={{ realtime }}>  
      <Canvas>
        {/* Lip sync happens automatically! */}
        <VRMAvatar src="/models/character.vrm" />
      </Canvas>
      <ChatInterface />
    </KhaveeProvider>
  );
}

✨ Automatic Features:

Lip sync with MFCC phoneme detection
Talking animations during speech
Natural blinking
WebRTC connection (no backend needed)

🎨 Control Expressions

import { useVRMExpressions } from '@khaveeai/react';

function ExpressionControls() {
  const { setExpression, resetExpressions, setMultipleExpressions } = useVRMExpressions();

  return (
    <div>
      {/* Single expression */}
      <button onClick={() => setExpression('happy', 1)}>
        😊 Happy
      </button>
      
      {/* Partial intensity */}
      <button onClick={() => setExpression('happy', 0.5)}>
        🙂 Slightly Happy
      </button>
      
      {/* Multiple expressions */}
      <button onClick={() => setMultipleExpressions({
        happy: 0.8,
        surprised: 0.4
      })}>
        😲 Excited
      </button>
      
      {/* Reset all */}
      <button onClick={() => resetExpressions()}>
        😐 Neutral
      </button>
    </div>
  );
}

💃 Play Animations

import { useVRMAnimations } from '@khaveeai/react';

function AnimationControls() {
  const { animate, stopAnimation, currentAnimation } = useVRMAnimations();

  return (
    <div>
      <button onClick={() => animate('walk')}>
        🚶 Walk
      </button>
      <button onClick={() => animate('dance')}>
        💃 Dance
      </button>
      <button onClick={() => animate('idle')}>
        🧍 Idle
      </button>
      <button onClick={() => stopAnimation()}>
        ⏹️ Stop
      </button>
      
      <p>Current: {currentAnimation || 'none'}</p>
    </div>
  );
}

🗂️ Project Structure

your-project/
├── public/
│   ├── models/
│   │   └── character.vrm          # Your VRM model
│   └── animations/
│       ├── idle.fbx               # Mixamo animations
│       ├── walk.fbx
│       └── dance.fbx
├── src/
│   ├── app/
│   │   └── page.tsx               # Your main component
│   └── ...
└── package.json

📚 API Reference

Components

`<KhaveeProvider>`

Root provider that manages VRM state and optional LLM/TTS configuration.

<KhaveeProvider config={config}>
  {children}
</KhaveeProvider>

Props:

config? - Optional LLM/TTS provider configuration
children - React children

`<VRMAvatar>`

Renders a VRM 3D character with animations and expressions.

<VRMAvatar
  src="/models/character.vrm"
  animations={animations}
  position={[0, -1, 0]}
  rotation={[0, Math.PI, 0]}
  scale={[1, 1, 1]}
  enableBlinking={true}
  enableTalkingAnimations={true}
/>

Props:

src - URL to VRM model file (required)
animations? - Animation configuration (URLs to FBX files)
position? - 3D position [x, y, z] (default: [0, 0, 0])
rotation? - 3D rotation [x, y, z] (default: [0, Math.PI, 0])
scale? - 3D scale [x, y, z] (default: [1, 1, 1])
enableBlinking? - Enable natural blinking (default: true)
enableTalkingAnimations? - Enable gestures during AI speech (default: true)

Hooks

`useVRMExpressions()`

Control facial expressions with smooth transitions.

const { 
  expressions,           // Current expression values
  setExpression,         // Set single expression
  resetExpressions,      // Reset all to neutral
  setMultipleExpressions // Set multiple at once
} = useVRMExpressions();

Example:

setExpression('happy', 1);              // Full happiness
setExpression('happy', 0.5);            // Partial
setMultipleExpressions({                // Multiple
  happy: 0.8,
  surprised: 0.3
});
resetExpressions();                     // Reset all

`useVRMAnimations()`

Play and control body animations.

const { 
  animate,              // Play animation by name
  stopAnimation,        // Stop all animations
  currentAnimation      // Currently playing animation name
} = useVRMAnimations();

Example:

animate('walk');       // Play walk animation
animate('dance');      // Play dance animation
stopAnimation();       // Stop all

`useRealtime()`

Real-time voice chat with OpenAI Realtime API.

const { 
  isConnected,
  connect,
  disconnect,
  sendMessage,
  conversation,
  chatStatus,
  currentPhoneme,  // Current phoneme for lip sync
  interrupt        // Interrupt AI speech
} = useRealtime();

// Usage
await connect();              // Start voice chat
await sendMessage('Hello!');  // Send text message
interrupt();                   // Stop AI from speaking
await disconnect();            // End session

Chat Status Values:

stopped - Not connected
ready - Connected, waiting
listening - User is speaking
thinking - AI is processing
speaking - AI is responding

`useAudioLipSync()`

Analyze audio files for lip sync (separate from realtime).

const { 
  analyzeLipSync, 
  stopLipSync, 
  isAnalyzing,
  currentPhoneme 
} = useAudioLipSync();

// Usage
await analyzeLipSync('/audio/speech.wav', {
  sensitivity: 0.8,
  intensityMultiplier: 3.0
});

`useVRM()`

Access the raw VRM instance for advanced use cases.

const vrm = useVRM();

if (vrm) {
  console.log('VRM loaded:', vrm.meta.name);
}

`useKhavee()`

Access all SDK functionality at once.

const { 
  vrm,
  setExpression,
  animate,
  // ... all functions
} = useKhavee();

🎯 Common Patterns

Real-time Voice Chat with Expressions

function VoiceChat() {
  const { isConnected, connect, chatStatus } = useRealtime();
  const { setExpression } = useVRMExpressions();

  // Set expressions based on chat status
  useEffect(() => {
    if (chatStatus === 'listening') {
      setExpression('surprised', 0.3);
    } else if (chatStatus === 'thinking') {
      setExpression('neutral', 1);
    } else if (chatStatus === 'speaking') {
      setExpression('happy', 0.7);
    }
  }, [chatStatus]);

  return (
    <button onClick={connect} disabled={isConnected}>
      🎤 Start Voice Chat
    </button>
  );
}

Animation + Expression Combo

function DanceWithJoy() {
  const { animate } = useVRMAnimations();
  const { setExpression } = useVRMExpressions();

  const danceHappily = () => {
    animate('dance');
    setExpression('happy', 1);
  };

  return <button onClick={danceHappily}>Dance!</button>;
}

Text Input with Voice Response

function TextChat() {
  const { sendMessage, conversation, chatStatus } = useRealtime();
  const [input, setInput] = useState('');

  const handleSend = async () => {
    if (!input.trim()) return;
    await sendMessage(input);
    setInput('');
  };

  return (
    <div>
      <div className="messages">
        {conversation.map((msg, i) => (
          <div key={i}>
            <strong>{msg.role}:</strong> {msg.text}
          </div>
        ))}
      </div>
      
      <input 
        value={input}
        onChange={(e) => setInput(e.target.value)}
        onKeyPress={(e) => e.key === 'Enter' && handleSend()}
        disabled={chatStatus === 'speaking'}
      />
      <button onClick={handleSend}>Send</button>
    </div>
  );
}

🔌 Providers

OpenAI Realtime Provider (Recommended)

Real-time voice chat with automatic lip sync:

import { OpenAIRealtimeProvider } from '@khaveeai/providers-openai-realtime';
import { useMemo } from 'react';

function App() {
  const realtime = useMemo(() => 
    new OpenAIRealtimeProvider({
      apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY!,
      voice: 'coral',  // or: alloy, echo, sage, shimmer
      instructions: 'You are a helpful AI assistant.',
      temperature: 0.8,
      tools: []  // Optional: Add RAG or custom functions
    }), []
  );

  return (
    <KhaveeProvider config={{ realtime }}>  
      {/* Your app */}
    </KhaveeProvider>
  );
}

RAG Provider

Add knowledge base search to your AI:

// app/lib/rag.ts (server-side)
"use server";
import { RAGProvider } from '@khaveeai/providers-rag';

export async function searchKnowledgeBase(query: string) {
  const rag = new RAGProvider({
    qdrantUrl: process.env.QDRANT_URL!,
    qdrantApiKey: process.env.QDRANT_API_KEY,
    collectionName: process.env.QDRANT_COLLECTION!,
    openaiApiKey: process.env.OPENAI_API_KEY!,
  });
  return await rag.search(query);
}

// app/page.tsx (client-side)
"use client";
const realtime = new OpenAIRealtimeProvider({
  apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY!,
  tools: [
    {
      name: 'search_knowledge_base',
      description: 'Search the knowledge base',
      parameters: {
        query: { type: 'string', description: 'Search query', required: true }
      },
      execute: async (args) => await searchKnowledgeBase(args.query)
    }
  ]
});

Mock Provider (Development)

Perfect for testing without API keys:

import { MockLLM, MockTTS } from '@khaveeai/providers-mock';

const config = {
  llm: new MockLLM(),
  tts: new MockTTS(),
};

<KhaveeProvider config={config}>
  {/* Test your UI without API costs */}
</KhaveeProvider>

🎨 Where to Get Assets

VRM Models

VRoid Hub - Free VRM characters
VRoid Studio - Create your own
Booth.pm - Buy premium models

Mixamo Animations

Go to Mixamo
Select any animation
Download as FBX format
No skeleton, just animation
Use the URL in your animations config

Recommended Animations:

Idle → Breathing Idle
Walk → Walking
Dance → Hip Hop Dancing, Swing Dancing
Talk → Talking with Hands
Wave → Waving

🛠️ Troubleshooting

VRM not rendering?

Check these:

✅ VRM file is valid (test in VRoid Hub)
✅ Wrapped in <Canvas> from @react-three/fiber
✅ Wrapped in <KhaveeProvider>
✅ Lights added to scene (<ambientLight>, <directionalLight>)

Animations not playing?

Check these:

✅ FBX files are from Mixamo
✅ Downloaded as FBX (not BVH)
✅ "Without Skin" option selected
✅ URLs are correct and accessible
✅ Animation name matches config key

Expressions not working?

Check these:

✅ VRM model has expression support
✅ Expression names are correct (check VRM in VRoid Hub)
✅ Values between 0 and 1
✅ Called inside component wrapped by <KhaveeProvider>

LLM/Voice not working?

Check these:

✅ Provider configured in <KhaveeProvider config={...}>
✅ API keys are valid

📖 Full Documentation

API Reference - Complete API docs
Examples - Working examples
Function Documentation - All functions documented
IntelliSense Guide - IDE integration guide

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details.

📄 License

🌟 Examples

Check out our example app to see:

✅ Expression controls
✅ Animation panel
✅ LLM chat integration
✅ Voice synthesis
✅ Combined interactions

💬 Support

📧 Email: support@khaveeai.com
💬 Discord: Join our community
🐛 Issues: GitHub Issues

Built with ❤️ by the Khavee AI Team

Name		Name	Last commit message	Last commit date
Latest commit History 42 Commits
.github/workflows		.github/workflows
packages		packages
public		public
src/app		src/app
.gitignore		.gitignore
.npmignore		.npmignore
FINAL_SUMMARY.md		FINAL_SUMMARY.md
IMPLEMENTATION_SUMMARY.md		IMPLEMENTATION_SUMMARY.md
README.md		README.md
eslint.config.mjs		eslint.config.mjs
next.config.ts		next.config.ts
package.json		package.json
pnpm-lock.yaml		pnpm-lock.yaml
pnpm-workspace.yaml		pnpm-workspace.yaml
postcss.config.mjs		postcss.config.mjs
tsconfig.json		tsconfig.json
tsconfig.packages.json		tsconfig.packages.json

SolveServeSolution/khaveeai-sdk

Folders and files

Latest commit

History

Repository files navigation

🎭 Khavee AI - VRM Avatar SDK

✨ Features

📦 Installation

Required Dependencies

Install Khavee AI SDK

Optional: Provider Packages

Peer Dependencies

🚀 Quick Start

Basic VRM Avatar

With Animations

With OpenAI Realtime (Voice Chat + Lip Sync)

🎨 Control Expressions

💃 Play Animations

🗂️ Project Structure

📚 API Reference

Components

<KhaveeProvider>

<VRMAvatar>

Hooks

useVRMExpressions()

useVRMAnimations()

useRealtime()

useAudioLipSync()

useVRM()

useKhavee()

🎯 Common Patterns

Real-time Voice Chat with Expressions

Animation + Expression Combo

Text Input with Voice Response

🔌 Providers

OpenAI Realtime Provider (Recommended)

RAG Provider

Mock Provider (Development)

🎨 Where to Get Assets

VRM Models

Mixamo Animations

🛠️ Troubleshooting

VRM not rendering?

Animations not playing?

Expressions not working?

LLM/Voice not working?

📖 Full Documentation

🤝 Contributing

📄 License

🌟 Examples

💬 Support

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

`<KhaveeProvider>`

`<VRMAvatar>`

`useVRMExpressions()`

`useVRMAnimations()`

`useRealtime()`

`useAudioLipSync()`

`useVRM()`

`useKhavee()`

Packages