An autonomous multi-agent system that keeps a GitHub learning-resources repository fresh, well-organised, and continuously improving. Each run generates a strategic plan from prior memory, analyses the target repo, discovers new high-quality content, updates existing files in-place, validates quality, requests human approval by email, opens a Pull Request, and analyses its own performance — all without manual intervention. Cross-run memory and an adaptive ML scorer mean every run starts smarter than the last.
- How It Works
- Pipeline State Machine
- Agents
- Persistent Memory & Self-Improvement
- Monitoring API
- Project Structure
- Tech Stack
- Configuration
- Local Setup
- Deployment (Railway)
- Topic Taxonomy
- Quality Gate
- v3 Changes
RepoSentinel runs as a one-shot pipeline. Each run:
- Loads memory — reads prior topic coverage, quality scores, source statistics, and semantic topic history from the DB so the run starts with full context from all previous runs.
- Plans —
PlannerAgentanalyses prior memory with an LLM to produce a strategic run plan: which topics to focus on, which to skip (recently covered at high quality), and how to weight each search source. - Analyses the target GitHub repository against a curated DS/ML/AI topic taxonomy to identify content gaps and stale material. The run plan's focus areas and skip list guide prioritisation.
- Scouts the web — Tavily search, arXiv papers, YouTube tutorials, and GitHub repositories — for fresh content. Source allocation is weighted by historical relevance scores and adjusted by the
MLScorer, which learns from each run. - Synthesises discovered content into Markdown files, running all files concurrently. For files that already exist, the LLM receives the current file content and merges new resources into it in-place rather than creating a new file.
- Validates the generated files through a 5-check quality gate (formatting, link validity, code syntax, duplication, completeness).
- Requests approval by sending a summary email (now includes total token usage) and waiting for a human approve/reject response.
- Publishes the approved files by creating a new branch and opening a Pull Request on the target repository.
- Updates memory — writes topic coverage, source quality, and per-agent timing back to the DB.
- Analyses performance —
PostRunAnalystAgentruns unconditionally after every completed or failed run, producing a performance summary, insights, recommendations, and anomaly flags stored in the DB.
INITIALIZED
│
▼
LOAD_MEMORY ─────────────────────────────────── Prior knowledge + semantic topics from DB
│
▼
PLANNING ──────────────────────────────────── PlannerAgent
│ LLM-generated run strategy
│ (focus areas, skip list, source weights)
▼
ANALYZING ──────────────────────────────────── RepoAnalystAgent
│ Gap analysis vs taxonomy
│ (parallel freshness scoring)
▼
SCOUTING ──────────────────────────────────── ContentScoutAgent
│ Tavily · arXiv · YouTube · GitHub
│ (MLScorer-adjusted allocation)
▼
SYNTHESIZING ────────────────────────────────── SynthesisEngineAgent
│ In-place updates + new files
│ (parallel per-file synthesis)
▼
VALIDATING ──────────────────────────────────── QAValidatorAgent
│ 5-check quality gate
├── PASS (score ≥ threshold) ──────────────► AWAITING_APPROVAL
│ │
├── FAIL + retries remaining ──────────────► SYNTHESIZING (retry)
│
└── FAIL + no retries left ────────────────► FAILED ──────────┐
AWAITING_APPROVAL │
│ │
┌────────┴────────┐ │
APPROVED REJECTED/TIMEOUT
│
▼
PUBLISHING ─────────── PublisherAgent
│ Branch + PR
▼
UPDATE_MEMORY ──────── MemoryUpdateAgent
│ Write learnings to DB
▼
POST_ANALYZING ─────── PostRunAnalystAgent ◄──┘
│ Performance insights
▼ (runs on failure too)
COMPLETED
| Agent | Role |
|---|---|
| PlannerAgent | (new in v3) Opens every run by querying prior memory and calling the LLM to produce a strategic plan: topic focus areas, a skip list of recently well-covered topics, per-source weighting, and a rationale. Falls back gracefully to a balanced default if the LLM call fails. |
| RepoAnalystAgent | Walks the repo tree, scores content freshness in parallel via asyncio.gather, maps files against the taxonomy, and loads prior knowledge from the DB to prioritise genuinely missing or low-quality topics. Receives the run plan to align gap analysis with the current strategy. |
| ContentScoutAgent | Runs parallel searches across Tavily, arXiv, YouTube, and GitHub. Allocates result slots using the MLScorer-adjusted source weights from the run plan. Filters blocked/paywalled domains, deduplicates, and scores relevance with the LLM. |
| SynthesisEngineAgent | Converts discovered items into Markdown files concurrently. For existing files, reads the current content from GitHub and sends it to the LLM with a merge prompt (using a higher token budget) rather than always creating new files. |
| QAValidatorAgent | Runs 5 checks per file (see Quality Gate). Auto-fixes minor formatting issues. Produces a 0–100 composite score and a PASS/FAIL decision. |
| PublisherAgent | Creates a timestamped branch, commits all approved files, and opens a Pull Request with a structured description and change manifest. |
| MemoryUpdateAgent | Post-publish agent that writes topic coverage, source quality statistics, and per-agent timing back to PostgreSQL so future runs start smarter. |
| PostRunAnalystAgent | (new in v3) Runs unconditionally after every completed or failed run. Uses the LLM to analyse quality scores, token usage, agent timings, and the original run plan to produce a performance summary, actionable insights, recommendations, and anomaly flags. |
RepoSentinel maintains persistent DB tables across runs:
| Table | What it stores |
|---|---|
repo_knowledge |
Per-topic: last covered date, best quality score, file paths. Used to skip recently-covered high-quality topics and prioritise weak ones. |
topic_history |
Append-only log of every topic touched per run, with action and quality score. Also drives get_all_semantic_topics() for richer context in the planning stage. |
source_quality |
Running EMA of relevance scores per source (Tavily, arXiv, YouTube, GitHub). Drives adaptive slot allocation — better sources get more result slots over time. |
run_metrics |
Per-run KPIs: quality score, files created/updated, topics covered, total tokens, per-agent timing. Surfaced via the /metrics API. |
agent_outputs |
Raw output JSON for every agent per run, now including per-agent total_tokens. Used to restore state when resuming approved runs. |
MLScorer (v3): At startup, MLScorer.load_from_db() reads source_quality and topic_history to build lightweight in-memory weights. ContentScoutAgent uses these to adjust raw relevance scores before filtering, making source selection increasingly accurate over time without any external ML dependencies.
This creates a compound feedback loop: each run's results improve the next run's strategic plan, gap analysis, source allocation, and in-place update decisions.
The FastAPI service (api.py) exposes monitoring endpoints alongside the approval webhook:
| Endpoint | Description |
|---|---|
GET /health |
Liveness check — verifies DB connectivity and returns the last run's status. |
GET /metrics?limit=20 |
KPIs for the last N completed runs (quality score, file counts, per-agent timing). |
GET /runs?limit=20 |
Paginated list of pipeline runs with status. |
GET /runs/{run_id} |
Full detail for a single run including all per-agent outputs. |
GET /knowledge |
Current topic knowledge state — all topics with quality scores, coverage dates, and source quality breakdown. |
GET /approve?token=…&action=approve |
Human approval webhook (existing). |
reposentinel/
├── src/reposentinel/
│ ├── agents/
│ │ ├── orchestrator.py # LangGraph StateGraph — main DAG (v3)
│ │ ├── planner.py # Strategic run planner (v3)
│ │ ├── repo_analyst.py # Gap analysis + parallel freshness scoring
│ │ ├── content_scout.py # Multi-source discovery + MLScorer integration
│ │ ├── synthesis_engine.py # Parallel LLM synthesis + in-place updates
│ │ ├── qa_validator.py # 5-check quality gate
│ │ ├── publisher.py # Branch + PR via GitHub API
│ │ ├── memory_update.py # Post-publish memory persistence
│ │ ├── post_run_analyst.py # Post-run performance analysis (v3)
│ │ └── base.py # BaseAgent interface (+ set_state_store)
│ ├── config/
│ │ ├── settings.py # Pydantic-settings (all env vars)
│ │ └── taxonomy.yaml # DS/ML/AI topic tree (editable)
│ ├── schemas/
│ │ └── core.py # All Pydantic models and enums
│ ├── services/
│ │ ├── github_client.py # PyGithub wrapper
│ │ ├── state_store.py # PostgreSQL run state + schema DDL
│ │ ├── memory_store.py # Persistent cross-run memory
│ │ ├── ml_scorer.py # Adaptive relevance scorer (v3)
│ │ └── email_gateway.py # SMTP send + approval polling
│ ├── api.py # FastAPI — approval webhook + monitoring endpoints
│ └── utils/
│ └── logging.py # structlog JSON logging
├── tests/ # pytest unit tests (all LLM/HTTP mocked)
├── Dockerfile # Multi-stage build
├── railway.toml # Railway deployment config
├── pyproject.toml # Poetry dependencies
└── .env # Local environment variables (not committed)
| Layer | Technology |
|---|---|
| Language | Python 3.11+ |
| Agent framework | LangGraph 0.2.x, LangChain 0.3.x |
| LLM | Google Gemini (gemini-2.5-pro) via langchain-google-genai |
| Web search | Tavily Python SDK |
| Academic papers | arXiv REST API |
| Video content | YouTube Data API v3 |
| Code repositories | GitHub Search API (PyGithub) |
| Data validation | Pydantic v2, pydantic-settings |
| Database | PostgreSQL (Railway) — psycopg2-binary |
| SMTP (Gmail / any provider) | |
| Logging | structlog (JSON to stdout) |
| Containerisation | Docker (multi-stage, python:3.11-slim) |
| Deployment | Railway (manual trigger or cron via dashboard) |
| CI / linting | ruff, mypy, bandit, pre-commit |
All settings are loaded from environment variables (or a .env file locally). Copy .env and fill in your values:
# Pipeline
GLOBAL_TIMEOUT_MINUTES=45
MAX_RETRIES=2
QUALITY_THRESHOLD=75 # 0–100, files scoring below this trigger a retry
# Target repository
GITHUB_REPO_URL=https://github.com/your-org/your-learning-repo
TARGET_BRANCH=main
GITHUB_PAT=ghp_... # Personal Access Token with repo write scope
# Content search
TAVILY_API_KEY=tvly-...
YOUTUBE_API_KEY=AIza...
MAX_RESULTS_PER_SOURCE=5
RELEVANCE_THRESHOLD=0.7 # 0.0–1.0, items below this are discarded
# Email / approval
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=you@gmail.com
SMTP_PASSWORD=your_app_password
RECIPIENT_EMAIL=approver@example.com
APPROVAL_TTL_HOURS=72 # Pipeline moves to REJECTED after this window
# LLM
PROVIDER=gemini
MODEL_NAME=gemini-2.5-pro # or gemini-1.5-flash for faster/cheaper runs
GEMINI_API_KEY=AIza...
TEMPERATURE=0.0
MAX_TOKENS_PER_CALL=4096
# Database
DATABASE_URL=postgresql://user:pass@host:5432/reposentinel
# Memory & self-improvement
MEMORY_ENABLED=true # set to false to disable cross-run learning
SYNTHESIS_MAX_TOKENS=8192 # token budget for in-place merge prompts
LOG_RETENTION_DAYS=90 # delete completed/failed run records older than this
SEARCH_RETENTION_DAYS=30 # delete stale content inventory cache older than thisPrerequisites: Python 3.11+, Poetry, PostgreSQL (or Docker), MailHog (optional, for local email testing).
# 1. Clone and install dependencies
git clone https://github.com/your-org/reposentinel.git
cd reposentinel
poetry install
# 2. Copy and fill in environment variables
cp .env.example .env # edit with your API keys
# 3. (Optional) Start local services with Docker Compose
docker-compose up -d # starts PostgreSQL + MailHog
# 4. Run the pipeline once
python -m reposentinel.agents.orchestratorRun tests:
poetry run pytest --cov=src/reposentinelLint and type-check:
poetry run ruff check src/
poetry run mypy src/RepoSentinel is deployed on Railway as a one-shot service.
- Fork / push this repo to GitHub.
- Create a new Railway project and link the GitHub repo.
- Add a PostgreSQL service to the project (Railway provides one click).
- Set all environment variables from the Configuration section in the Railway service's Variables tab. Set
DATABASE_URLto the Railway Postgres connection string. - Deploy — Railway builds the Docker image automatically.
- To trigger a run: go to the service in Railway dashboard → Deployments → Deploy (or configure a Cron Job schedule in the Railway dashboard settings).
The container runs the pipeline once and exits. restartPolicyType = "NEVER" in railway.toml ensures it does not restart automatically.
DB tables are created automatically on first run (via initialise_schema() at startup). The API service also creates them on boot, so hitting any /health or /knowledge endpoint before a pipeline run is safe.
The gap analysis is driven by src/reposentinel/config/taxonomy.yaml. The current taxonomy covers 12 topic areas with 60+ subtopics:
| # | Topic Area |
|---|---|
| 1 | Python for Data Science (NumPy, Pandas, Matplotlib, etc.) |
| 2 | Statistics & Mathematics |
| 3 | Machine Learning (Supervised, Unsupervised, Ensemble, etc.) |
| 4 | Deep Learning (CNNs, Transformers, PyTorch, TensorFlow) |
| 5 | Natural Language Processing (BERT, LLMs, RAG, etc.) |
| 6 | Computer Vision (Object Detection, GANs, ViT) |
| 7 | Generative AI & Foundation Models (LangChain, Agents, VectorDBs) |
| 8 | MLOps & Production ML (MLflow, FastAPI, Docker, CI/CD) |
| 9 | Data Engineering (Spark, Airflow, Kafka, dbt) |
| 10 | Reinforcement Learning (DQN, Policy Gradients, RLHF) |
| 11 | AI Ethics & Responsible AI (Fairness, SHAP, Governance) |
| 12 | Specialised Domains (Time Series, RecSys, Graph NNs) |
To add or remove topics, edit taxonomy.yaml — no code changes required.
Every generated file passes through five checks before the pipeline requests human approval:
| Check | Weight | What it validates |
|---|---|---|
| Formatting | 10 | Trailing newline, no excessive blank lines, H1 heading present |
| Link validity | 25 | All Markdown links respond with HTTP 2xx/3xx (403/429 bot-blocks are not penalised) |
| Code syntax | 25 | Python code blocks inside Markdown parse without SyntaxError |
| Duplication | 20 | Word-overlap with existing repo content is below 85% |
| Completeness | 20 | LLM estimates whether the planned topics are covered (target ≥ 70%) |
The composite score (0–100) must reach QUALITY_THRESHOLD (default 75) to proceed. If it fails, the pipeline retries synthesis up to MAX_RETRIES times before moving to FAILED.
PlannerAgent— added as the first pipeline stage afterload_memory. Queries prior knowledge and uses the LLM to generate a strategic run plan (focus areas, skip list, source weights). Gracefully falls back to a balanced default if the LLM call fails.PostRunAnalystAgent— added as the final stage, running after both successful and failed runs. Produces a structured performance report (outcome, efficiency, insights, recommendations, anomalies) stored inagent_outputs.
MLScorer— loaded at startup viaload_from_db(). Reads historicalsource_qualityandtopic_historyto build weighted scoring factors. Injected intoContentScoutAgentviaset_ml_scorer()to adjust raw relevance scores before filtering.
- New
PipelineStatefields:run_plan,post_run_analysis,total_tokens total_tokensis accumulated across every agent node and included in approval emails and the post-run analysisPipelineStatusgained two new values:PLANNINGandPOST_ANALYZING
state_store.save_agent_output()now storestotal_tokensper agent outputstate_store.sum_tokens_for_run(run_id)— sums token usage for a run from DB (used when resuming approved runs)state_store.cleanup_old_records(log_retention_days, search_retention_days)— called at startup to prune stale run records and content inventory cachememory_store.get_all_semantic_topics()— aggregatestopic_historyby topic for richer planning contextmemory_store.initialise_schema()— no-op stub; schema is fully managed byStateStoreBaseAgent.set_state_store()— concrete method added to the base class; lets the orchestrator wire DB access into any agent post-construction- New settings:
LOG_RETENTION_DAYS(default 90) andSEARCH_RETENTION_DAYS(default 30)