Real-time AI-powered scam call detection system using multi-modal analysis of audio features, speech patterns, and conversation context. The system combines an Android mobile application with a Python backend to provide live scam detection during phone calls.
This project uses a 4-model ensemble approach to detect scam calls in real-time:
- Phoneme CNN - Analyzes speech phonetic patterns
- Urgency Detector - Measures pitch, energy, and speech rate anomalies
- Repetition CNN - Detects repetitive keyword patterns
- Sentence Transformer - Tracks conversation stages via intent classification
┌─────────────────┐ WebSocket ┌──────────────────┐
│ Android App │ ◄────────────────────────► │ Python Backend │
│ (Kotlin) │ 4-sec audio chunks │ (FastAPI) │
└─────────────────┘ └──────────────────┘
│ │
│ ▼
│ ┌────────────────┐
│ │ 4 AI Models │
│ │ - Phoneme CNN │
│ │ - Urgency MLP │
│ │ - Repetition │
│ │ - Stage Track(Transcribing Audio) │
│ └────────────────┘
│ │
│ ▼
│ ┌────────────────┐
│ │ Risk Score │
│ ◄────────────────────────────────────│ (0.0 - 1.0) │
│ Risk Assessment └────────────────┘
▼
┌─────────────────┐
│ User Alert │
│ ✅ SAFE │
│ 🟡 LOW RISK │
│ 🟠 MODERATE │
│ 🔴 HIGH RISK │
│ 🚨 SCAM ALERT │
└─────────────────┘
- Real-time Audio Processing - 4-second sliding window analysis
- Multi-Model Fusion - Combines 4 AI models for comprehensive detection
- WebSocket API - Live audio streaming from mobile clients
- Voice Activity Detection - Filters silence and non-speech segments
- Conversation Stage Tracking - Monitors call progression patterns
- Risk Assessment - Continuous risk scoring with 5 alert levels
Hackathon-Projects/
├── app/ # Android application
│ ├── src/
│ │ └── main/
│ │ ├── java/com/example/audio/
│ │ │ ├── MainActivity.kt
│ │ │ ├── Model/ # Data layer
│ │ │ ├── View/ # UI screens
│ │ │ └── ViewModel/ # Business logic
│ │ └── AndroidManifest.xml
│ └── build.gradle.kts
│
├── Backend/ # Python backend server
│ ├── models/ # Trained AI models
│ ├── whisper.cpp/ # Speech transcription
│ ├── newServer.py # FastAPI server
│ ├── config.py # Configuration
│ ├── feature_extraction.py # Audio preprocessing
│ ├── audio_pipeline.py # Processing pipeline
│ └── pyproject.toml
│
├── dataset/ # Training data
│ ├── NORMAL_CALLS/
│ └── SCAM_CALLS/
│
├── models/ # Trained models
│ ├── best_phoneme_model.tflite
│ └── Prosody_model/
│
├── notebooks/ # Training notebooks
│ ├── phoneme_layer.ipynb
│ └── prosody_layer.ipynb
│
└── training_config.json # Model training config
Backend:
- Python 3.12.6+
- Whisper.cpp (for speech transcription)
- TensorFlow/Keras
- FastAPI
Android:
- Android Studio
- Kotlin
- Gradle 8.13+
- Min SDK 26 (Android 8.0)
- Target SDK 36
- Navigate to Backend directory:
cd Backend- Install dependencies:
uv init
uv sync- Build Whisper.cpp:
cd whisper.cpp
make- Download Whisper model:
wget https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-tiny.en.bin -P models/- Place trained models in
models/directory:
best_phoneme_model.kerasbest_repetition_model.kerasggml-tiny.en.bin
- Start the server:
python newServer.pyServer starts on http://localhost:8000
- Open project in Android Studio:
# Open the root directory in Android Studio- Sync Gradle dependencies:
- Android Studio will automatically sync dependencies
- Configure backend URL:
- Update WebSocket URL in the app to point to your backend server
- Build and run:
./gradlew assembleDebug
# Or use Android Studio's Run buttonGET /health
Returns server status.
WS /ws/audio
Input: 4-second int16 PCM audio chunks (128KB at 16kHz)
Output: Risk assessment string
✅ SAFE- Risk < 0.15🟡 LOW RISK- Risk 0.15-0.35🟠 MODERATE- Risk 0.35-0.55🔴 HIGH RISK- Risk 0.55-0.80🚨 SCAM ALERT- Risk > 0.80
SR = 16000 # Audio sample rate
WINDOW_SEC = 4 # Analysis window size
VAD_MODE = 2 # Voice activity detection sensitivity
MIN_SPEECH_S = 1.0 # Minimum speech duration
# Model fusion weights
W_STAGE = 0.60 # Stage tracking weight
W_REP = 0.20 # Repetition detection weight
W_PHONEME = 0.10 # Phoneme analysis weight
W_URGENCY = 0.10 # Urgency detection weight
# Risk calculation
ALPHA = 0.7 # Running risk smoothing
BETA = 0.3 # New risk weight
THRESHOLD = 0.45 # Scam threshold{
"sequence_length": 301,
"feature_dim": 39,
"n_mfcc": 13,
"sample_rate": 16000,
"chunk_duration": 3.0,
"scam_threshold": 0.75
}The Android app captures audio during phone calls and sends 4-second chunks to the backend via WebSocket.
The backend filters out silence and non-speech segments using WebRTC VAD.
- MFCC Features - Mel-frequency cepstral coefficients for phoneme analysis
- Prosody Features - Pitch, energy, speech rate for urgency detection
- Repetition Features - Keyword pattern detection
- Transcription - Speech-to-text using Whisper
All 4 models run in parallel:
- Phoneme CNN analyzes speech patterns
- Urgency detector measures vocal stress
- Repetition CNN identifies keyword patterns
- Sentence transformer tracks conversation stages
Weighted combination of all model outputs:
risk = 0.60 × stage_risk + 0.20 × repetition + 0.10 × phoneme + 0.10 × urgency
Risk score is mapped to one of 5 alert levels and sent back to the Android app.
The system tracks 5 conversation stages common in scam calls:
- GREETING - Initial contact (Risk: 0.10)
- AUTHORITY - Claims to be from bank/official (Risk: 0.25)
- PROBLEM - Creates urgency/fear (Risk: 0.50)
- URGENCY - Demands immediate action (Risk: 0.70)
- DATA_REQUEST - Asks for OTP/password (Risk: 0.95)
Training notebooks are available in the notebooks/ directory:
phoneme_layer.ipynb- Train phoneme CNN modelprosody_layer.ipynb- Train urgency detection model
Dataset structure:
dataset/
├── NORMAL_CALLS/ # Legitimate call recordings
└── SCAM_CALLS/ # Scam call recordings
- FastAPI - Web framework
- TensorFlow/Keras - Deep learning models
- Librosa - Audio processing
- Sentence Transformers - Text embeddings
- WebRTC VAD - Voice activity detection
- Whisper.cpp - Speech-to-text
- Uvicorn - ASGI server
- Kotlin - Programming language
- Jetpack Compose - UI framework
- Room - Local database
- Retrofit - HTTP client
- Coroutines - Async programming
- TensorFlow Lite - On-device inference
- Zego UIKit - Call functionality
- Latency - ~500ms per 4-second window
- Accuracy - Multi-model ensemble approach
- Real-time - Processes audio as it streams
- Scalability - Thread pool for concurrent connections
- Audio is processed in real-time and not stored
- Encrypted communication via WebSocket
- Local processing option with TensorFlow Lite
- No personal data collection
- Requires stable internet connection for backend processing
- Works best with clear audio (minimal background noise)
- English language optimized (Hindi support in progress)
- Requires 16kHz audio sample rate
- Multi-language support (Hindi, regional languages)
- On-device processing with TFLite models
- Historical call analysis dashboard
- Community-driven scam pattern database
- Integration with telecom providers
- Automatic call blocking
Contributions are welcome! Please follow these steps:
- Fork the repository
- Create a feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
This project is licensed under the MIT License.
- Arya Gupta - All rounder (System Design + Contribution in all other aspects)
- Srijan Sarkar - Android App Development
- Shinjan Saha - AI/ML Development
- Pritam Paul - AI/ML Developement
- Whisper.cpp for speech transcription
- Sentence Transformers for text embeddings
- TensorFlow team for deep learning framework
- Research paper: "Spam Detection in Voicemails" (ASA Meeting 2021)
For questions or support, please open an issue in the repository.
Note: This is a research project built in DoubleSlash Hackathon, Jadavpur. Always verify suspicious calls through official channels.