DeepSea-AI Pipeline

A comprehensive pipeline for analyzing environmental DNA (eDNA) sequences using deep learning and phylogenetic methods.

Project Structure

./deepsea-ai/
├── frontend/          # React app
├── backend/           # FastAPI/Express server  
├── pipeline/          # ML scripts + models
│   ├── scripts/       # Python pipeline scripts
│   ├── models/        # Pre-trained models
│   └── data/          # Raw and processed data
├── shared/            # Shared TypeScript schemas
├── docker-compose.yml # Container orchestration
└── README.md          # This file

Quick Start

Installation

cd deepsea-ai
pip install -r pipeline/scripts/requirements.txt

Single Command Execution

python pipeline/run_pipeline.py --input pipeline/data/raw/deepsea-sample.fasta

Advanced Usage

python pipeline/run_pipeline.py --input your_sequences.fasta --output-dir pipeline/data/processed --method transformer --device cuda --k 6

Pipeline Stages

1. Preprocessing & QC - Quality filtering, trimming, deduplication
2. K-mer Tokenization - Convert sequences to k-mer features
3. Embeddings Generation - Create dense representations using DNABERT or SVD
4. VAE Clustering - Unsupervised discovery using Variational Autoencoders
5. Phylogenetic Visualization - Build similarity graphs and phylogenetic trees
6. Taxonomy Assignment - Hybrid workflow with representative BLAST queries
7. Report Generation - Comprehensive analysis reports and visualizations

Options

• --input: Input FASTA/FASTQ file (required)
• --output-dir: Output directory (default: pipeline/data/processed)
• --method: Embedding method - transformer or svd (default: transformer)
• --device: Device for transformer - cpu or cuda (default: cpu)
• --k: K-mer size (default: 6)
• --skip-stages: Skip specific stages (e.g., --skip-stages 1 2)
• --blast-db: Local BLAST database path for taxonomy assignment
• --online-blast: Use online BLAST (slower but no local DB needed)

Output Files

• cleaned_sample.fasta - Preprocessed sequences
• dedup_counts.csv - Sequence counts and metadata
• embeddings.npy - Dense sequence embeddings
• clusters.csv - Cluster assignments
• novelty_scores.csv - Novelty detection results
• umap_2d.npy - 2D projections for visualization
• phylo_graph.json - Similarity graph
• phylo_simple.nwk - Phylogenetic tree
• phylo_tree_colored.png - Tree visualization
• cluster_taxonomy.csv - Taxonomy assignments for cluster representatives
• sequence_taxonomy.csv - Full taxonomy mapping for all sequences
• analysis_report.md - Comprehensive analysis summary
• taxonomy_abundance.png - Taxonomic composition plots
• novelty_analysis.png - Novelty detection visualizations
• phylogenetic_diversity.png - Phylogenetic diversity plots

Individual Stage Execution

You can also run individual stages:

# Stage 1: Preprocessing
python pipeline/scripts/preprocess.py pipeline/data/raw/input.fasta pipeline/data/processed

# Stage 2: Tokenization  
python pipeline/scripts/kmer_tokenize.py pipeline/data/processed/cleaned_sample.fasta pipeline/data/processed --k 6

# Stage 3: Embeddings
python pipeline/scripts/get_embeddings.py pipeline/data/processed/cleaned_sample.fasta pipeline/data/processed/embeddings.npy --method transformer

# Stage 4: Clustering
python pipeline/scripts/vae_cluster.py

# Stage 5: Phylogenetics
python pipeline/scripts/phylo_vis.py

# Stage 6: Taxonomy Assignment
python pipeline/scripts/run_taxonomy.py --blast-db pipeline/data/reference/16S_ribosomal_RNA
# OR for online BLAST (slower):
python pipeline/scripts/run_taxonomy.py --online

# Stage 7: Generate Reports
python pipeline/scripts/biodiversity_analysis.py

# Visualization
python pipeline/scripts/view_tree.py

Development

Frontend (React)

cd frontend
npm install
npm run dev

Backend (Node.js/TypeScript)

cd backend
npm install
npm run dev

Full Stack with Docker

docker-compose up --build

Features

• ✅ Robust error handling and validation
• ✅ Reproducible results with fixed random seeds
• ✅ Memory-efficient processing for large datasets
• ✅ Support for both CPU and GPU acceleration
• ✅ Comprehensive logging and progress tracking
• ✅ Flexible configuration options
• ✅ Web interface for pipeline management
• ✅ Real-time progress monitoring