feat(analytics): GitHub-backed code source registry for codebase analytics

Summary

Add a Code Source Registry to codebase analytics so the system can pull code from GitHub (or other VCS) repositories using stored credentials, rather than relying on local filesystem paths. Supports multiple repos/projects, customer code analysis, automation reuse, and optional knowledge base ingestion.

Problem

The current /analytics/codebase page requires a local file path (/opt/autobot/...). In distributed infrastructure you cannot rely on local folders — code must be pulled from the authoritative source (GitHub) with proper credentials. Multiple projects/customers also need project-switching support.

Design

Code Source Registry

Each registered source is stored persistently (Redis or DB):

Field	Description
`id`	UUID
`name`	Display name ("AutoBot Production", "Customer A CRM")
`source_type`	`github` \| `local`
`repo`	`owner/repo` (GitHub mode)
`branch`	e.g. `Dev_new_gui`
`credential_name`	References existing secrets store entry
`clone_path`	`/opt/autobot/data/code-sources/<id>/`
`last_synced`	ISO timestamp
`status`	`configured` \| `syncing` \| `ready` \| `error`

Backend API

GET    /api/analytics/codebase/sources           — list all sources
POST   /api/analytics/codebase/sources           — register new source
GET    /api/analytics/codebase/sources/{id}      — get source details
PUT    /api/analytics/codebase/sources/{id}      — update source config
DELETE /api/analytics/codebase/sources/{id}      — remove source
POST   /api/analytics/codebase/sources/{id}/sync — clone/pull now (async job)

POST /api/analytics/codebase/index updated to accept source_id in addition to root_path (backward compatible).

Clone Strategy

Shallow clone: git clone --depth=1 --branch <branch> https://<token>@github.com/<repo> <clone_path>
Subsequent syncs: git pull if clone path exists, else re-clone
Clone path: /opt/autobot/data/code-sources/<source-id>/
Token injected from secrets store (never logged or exposed)
Reuse: any automation step can resolve source_id → local path

Frontend

Project selector dropdown in analytics header (replaces plain text path input)
Source management panel: list, add, edit, delete, sync sources
Status indicators: last synced time, sync button, error state
Add Source modal: name, type toggle (GitHub/Local), repo, branch, credential selector
Existing analytics UI unchanged — just feeds from selected source

Knowledge Base (opt-in)

Optional checkbox after indexing: "Add to knowledge base"
OFF by default — prevents junk knowledge ingestion
When enabled: passes indexed files through existing knowledge pipeline

Credentials

Uses existing SecretType.TOKEN / SecretType.API_KEY from api/secrets.py
No new secret storage needed
Credential dropdown in Add Source modal lists compatible secrets

Implementation Tasks

Notes

All paths under /opt/autobot/data/code-sources/ — never hardcode dev machine paths
GitRepoSync in skills/sync/git_sync.py already handles clone logic — extend it for persistent paths and private repos
Backward compatible: existing root_path in IndexCodebaseRequest still works
Same clone path reused by automation workflows (reference by source_id or resolved path)

Additional Requirement: Per-Source Access Control

Code sources need to support multi-user sharing. A source registered by one user can be shared with specific other users or made accessible to all users.

Access Model

Each CodeSource record gets an access field:

Field	Description
`owner_id`	User who created the source
`access`	`private` \| `shared` \| `public`
`shared_with`	List of user IDs (for `shared` access)

Rules

private — only the owner can see, sync, and analyze this source (default)
shared — owner can grant specific users read/sync access
public — all authenticated users can use this source (admin-set only)

UI

Source management panel shows an access badge per source (Private, Shared, Public)
Owner can click "Share" → pick users from a list → grant/revoke access
Shared users see the source in their project selector but cannot edit/delete it
Credential is resolved server-side — shared users do NOT see the underlying GitHub token

API additions

POST   /api/analytics/codebase/sources/{id}/share
  body: { user_ids: [...], access: "shared" | "private" | "public" }

GET    /api/analytics/codebase/sources
  — already filters by: owned by me OR shared with me OR public

This also means the /sources list endpoint must filter by the requesting user's identity.

Add to implementation tasks:

Backend: access + shared_with fields on CodeSource model
Backend: Filter GET /sources list by user access
Backend: POST /sources/{id}/share endpoint
Frontend: Access badge on source list
Frontend: Share modal with user picker

Additional Requirement: Analysis Job Queue

Currently indexing.py rejects a second index request if one is already running (status: already_running). Instead, new requests should be queued and executed automatically after the running job finishes.

Behaviour

Only one analysis job runs at a time (unchanged)
Additional POST /api/analytics/codebase/index requests are queued, not rejected
When a job finishes (completed, failed, or cancelled), the next queued job starts automatically
Queue is FIFO per source — if the same source is already queued, deduplicate (don't add it twice)
Queue survives in-process (no persistence needed for now — restart clears it)

Queue State

running: { task_id, source_id, started_at }
queue:   [ { source_id, queued_at, requested_by }, ... ]

API additions/changes

GET /api/analytics/codebase/index/queue
  → { running: {...} | null, queue: [...], queue_length: N }

DELETE /api/analytics/codebase/index/queue/{source_id}
  → remove a source from the queue (cancel before it starts)

POST /api/analytics/codebase/index response changes:

If nothing running → { status: "started", task_id: "..." } (same as today)
If another job running → { status: "queued", position: 2, message: "Queued behind current job" } (NEW — no longer an error)

Frontend

Progress/status area shows both the running job AND the queue
"X jobs queued" indicator
Ability to remove a source from the queue before it starts
When a queued job starts, the UI transitions automatically (same polling mechanism)

Implementation notes

Auto-dequeue logic goes in the existing _create_cleanup_callback — when a task finishes, pop the next item from the queue and start it
Queue stored as asyncio.Queue or a plain deque with the existing _tasks_lock
Deduplication check on enqueue: if source_id already in queue, return existing position instead of adding again

Add to implementation tasks:

Backend: In-memory FIFO queue (deque) with deduplication by source_id
Backend: Auto-start next job in _create_cleanup_callback
Backend: GET /index/queue status endpoint
Backend: DELETE /index/queue/{source_id} cancel-before-start endpoint
Backend: Update POST /index response — queued status instead of error
Frontend: Queue status display (running + queued jobs)
Frontend: Remove-from-queue button

Additional Requirement: Repo Sync / Update

The POST /api/analytics/codebase/sources/{id}/sync endpoint already covers initial clone, but needs to be explicit about update behaviour:

If clone path exists → git pull --rebase (pick up latest commits on the configured branch)
If clone path missing → full git clone --depth=1
If branch changed since last sync → re-clone (shallow clone does not support branch switch via pull)
Sync result stored on the source: last_synced, last_commit_sha, status

Auto-sync options (future, not MVP)

Add auto_sync flag and sync_interval_minutes to CodeSource model so sources can be configured to pull on a schedule. MVP = manual sync only.

Sync-then-analyse flow

Common workflow: "update the repo, then run analysis." Support chaining in one request:

POST /api/analytics/codebase/sources/{id}/sync-and-index

Equivalent to: sync → on completion → enqueue index job. Returns a sync_task_id and a queued_index_position so the frontend (and skill) can track both steps.

Add to implementation tasks:

Backend: git pull --rebase path in sync handler (detect existing clone)
Backend: sync_and_index chained endpoint
Frontend: "Update & Analyse" button alongside existing "Sync" and "Analyse" buttons

Additional Requirement: Code Analytics Skill (AutoBot Chat Integration)

A builtin skill so users can trigger code analysis conversationally and receive results back in chat.

Skill: `code-analysis`

Location: autobot-backend/skills/builtin/code_analysis.py

Follows the same pattern as CodeReviewSkill (BaseSkill + SkillManifest).

Actions

Action	Description
`analyze_source`	Run analysis on a registered source by name or ID
`analyze_repo`	Register-and-analyze an ad-hoc repo URL (no pre-registration needed)
`get_results`	Fetch results for a completed analysis job
`list_sources`	List available code sources the user has access to

Example conversation

User:  Analyse the AutoBot Production repo and tell me about code quality.

AutoBot:  I'll run a code analysis on AutoBot Production. Syncing the repo first...

          ✓ Repo synced (branch: Dev_new_gui, commit: a1b2c3d)
          ⏳ Analysis queued (position 1 in queue)

          [30 seconds later...]

          ✅ Analysis complete — here's the summary:

          📊 Code Quality Report — AutoBot Production
          ─────────────────────────────────────────
          Files analysed:   1,247
          Lines of code:    89,432
          Languages:        Python (68%), TypeScript (28%), Other (4%)

          🔴 Issues requiring attention (23)
            • 8 functions exceed 65-line limit
            • 12 bare except clauses
            • 3 hardcoded IP addresses

          🟡 Warnings (41)
            • 29 missing UTF-8 encoding declarations
            • 12 direct Redis connections (use get_redis_client)

          🟢 Health score: 74/100

          Full report available at /analytics/codebase
          (source: AutoBot Production, job: abc123...)

Ad-hoc repo (no pre-registration)

User:  Run analysis on github.com/customerX/api-service branch main,
       use the github_token_customerX credential.

AutoBot:  Creating a temporary code source for customerX/api-service...
          [follows same sync → queue → results flow]

Skill parameters

SkillManifest(
    name="code-analysis",
    tools=["analyze_source", "analyze_repo", "get_results", "list_sources"],
    triggers=["manual"],  # chat-invoked only
    config={
        "default_source_id": SkillConfigField(type="string", default=None),
        "include_in_knowledge_base": SkillConfigField(type="bool", default=False),
        "report_detail_level": SkillConfigField(
            type="string", default="summary",
            choices=["summary", "detailed", "full"]
        ),
    }
)

Implementation notes

Skill polls /api/analytics/codebase/index/status/{task_id} (existing endpoint) until complete
Results fetched from /api/analytics/codebase/stats and /report (existing endpoints)
Skill formats the JSON report into a human-readable chat message
include_in_knowledge_base=True triggers knowledge ingestion post-analysis
For ad-hoc repos: creates a private source scoped to the requesting user, auto-cleans after analysis (or keeps if user says "save this source")

Add to implementation tasks:

Backend: skills/builtin/code_analysis.py — CodeAnalysisSkill
Backend: analyze_source action (resolve source → sync if stale → queue index → poll → format result)
Backend: analyze_repo action (register temp source → same flow)
Backend: list_sources action (proxy to registry filtered by user access)
Backend: Result formatter (JSON stats → readable chat markdown)
Backend: Register skill in skill registry

feat(analytics): add code source registry frontend (#1133)

feat(analytics): add code source registry backend (#1133)

✅ Implemented in commits eba722b (frontend) + 95844d0 (backend) on Dev_new_gui.

Changes:

Backend:

source_models.py — CodeSource Pydantic model with GitHub/local types, status, access control
source_storage.py — Redis CRUD layer (analytics DB) with visibility-filtered listing
endpoints/sources.py — CRUD + async sync (clone/pull) + share endpoints at /sources
endpoints/queue.py — GET /index/queue, DELETE /index/queue/{source_id}
scanner.py — Added _index_queue: deque for FIFO job queuing
endpoints/indexing.py — Accepts source_id, queues requests when busy, auto-dequeues on completion
router.py — Registered sources + queue routers

Frontend:

SourceManager.vue — Source list panel with CRUD, status/access badges, queue display, auto-refresh
AddSourceModal.vue — Create/edit modal with GitHub/local toggle, credential picker, access selector
ShareSourceModal.vue — Access control dialog with user picker for shared mode
AnalyticsHeader.vue — Source selector dropdown + Manage Sources button (backward-compatible)
CodebaseAnalytics.vue — Wired source management, source_id in index POST, KB opt-in post-index

Verification:

✅ All pre-commit checks passed (black, isort, flake8, bandit, no hardcoded values)
✅ No print()/console.* violations
✅ Function length compliant
✅ Redis client uses canonical get_redis_client(database="analytics")

fix(slm): fix the two CI reds blocking the merge train (#16229, #16231)

feat(analytics): GitHub-backed code source registry for codebase analytics #1133

Description

Summary

Problem

Design

Code Source Registry

Backend API

Clone Strategy

Frontend

Knowledge Base (opt-in)

Credentials

Implementation Tasks

Notes

Activity

mrveiss commented on Feb 22, 2026

Additional Requirement: Per-Source Access Control

Access Model

Rules

UI

API additions

mrveiss commented on Feb 22, 2026

Additional Requirement: Analysis Job Queue

Behaviour

Queue State

API additions/changes

Frontend

Implementation notes

mrveiss commented on Feb 22, 2026

Additional Requirement: Repo Sync / Update

Auto-sync options (future, not MVP)

Sync-then-analyse flow

mrveiss commented on Feb 22, 2026

Additional Requirement: Code Analytics Skill (AutoBot Chat Integration)

Skill: code-analysis

Actions

Example conversation

Ad-hoc repo (no pre-registration)

Skill parameters

Implementation notes

mrveiss commented on Feb 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Skill: `code-analysis`