DejaVec — Idea Archaeology

Your idea has been here before.

DejaVec is a high-fidelity semantic concept search and genealogy timeline engine. It is built to disprove the myth of the "unique startup idea" by analyzing raw concepts and tracing their direct intellectual ancestors—patents filed, research papers published, corporate memos written, or startups launched—across every domain and technology era since the 1920s.

Unlike traditional keyword search tools, DejaVec encodes raw text into high-dimensional coordinate spaces to understand conceptual shape, allowing it to identify similar ideas even when they are described using completely different words.

---

System Architecture

graph TD
    subgraph Frontend [frontend/dejavec.html - Vanilla CSS & JS]
        UI[Search / Excavation UI]
        Anim[Cinematic Loading Animation]
        TML[Timeline Generator]
    end

    subgraph Backend [FastAPI Server - backend/main.py]
        API[FastAPI Router]
        Model[sentence-transformers<br>all-mpnet-base-v2]
    end

    subgraph Storage [Qdrant In-Process DB - backend/qdrant_db/]
        Coll[Collection: 'ideas']
        HNSW[HNSW Dense Index]
        Payload[Payload Storage]
    end

    UI -->|1. POST /search| API
    UI -->|2. POST /add| API
    API -->|Generate 768-dim vector| Model
    API -->|Query / Upsert| Coll
    Coll -->|HNSW Graph Traverse| HNSW
    Coll -->|Read/Write Payload| Payload
    API -->|Ranked Ancestors + Similarity| UI
    TML -->|Render timeline dynamically| UI

Loading

---

Premium UI Features

We have built a suite of production-grade premium UI upgrades designed specifically to maximize usability and visual depth during evaluations:

1. Centroid-Based Query Intent Classification

• Engine: Zero-shot domain categorization using cosine similarity against pre-computed 768-dimensional centroids across 13 domains, calibrated via contrast-enhanced Softmax.
• Presentation: Renders a clean monospaced banner displaying predicted domain intent and confidence metrics (e.g. Social Architecture & Urban Planning) to make the vector model's mathematical process visible.

2. 2D Dimensionality Projection Mapping

• Engine: Abstract mathematical projection mapping that compresses the query's 768-dimensional space into coordinates [X, Y] on-the-fly.
• Presentation: Highlights coordinates and spatial directions in real-time, demonstrating clustering concepts visually.

3. Explainable Similarity Archaeology Reports

• Engine: Generates detailed, structured reasoning cards describing exactly why a historical precursor matched, including domain overlap, shared themes, and temporal deltas.
• Presentation: Replaces standard similarity decimals with actual semantic intelligence, showing real academic and technical value.

4. Clickable Source Reference Links with Smart Redirection

• Engine: Connects matching cards to specific places on the internet:
  • Patents: Automatically routes through to the official Google Patents document viewer (e.g. https://patents.google.com/patent/US6978307B2/en).
  • Concepts / Papers / Studies / Startups: Cleaned titles route through Wikipedia's exact-match Special Search engine, immediately opening the specific historic post or article page.
• Presentation: Directs users immediately from simple timelines to authoritative resources across the web.

5. Reload-Free "Surprise Me" Discovery Cycler

• Behavior: State-aware cycler dynamically selects a completely different query from the current input and runs it instantly on the timeline without any page reloads, guaranteeing fresh, rapid discovery.

---

Technical Specifications & Stack

1. Vector Embedder Layer (backend/embedder.py)

• Model: sentence-transformers/all-mpnet-base-v2
• Vector Dimensions: 768-dimensions
• Distance Metric: Cosine Similarity (calibrated dot product matching)
• Hardware: In-process CPU embedding engine.

2. Dense Indexing & Vector Store (backend/qdrant_store.py)

• Engine: qdrant-client local storage.
• Index Structure: Hierarchical Navigable Small World (HNSW) graph, guaranteeing sub-millisecond similarity scans.
• Decay Rank Equation: Scores are adjusted adaptively based on unit-normalized semantic similarity, temporal year deltas, and domain category overlaps.

3. API Service Layer (backend/main.py)

• Security & Guardrails: Thread-safe Token-Bucket rate limiter (100 capacity, 1/s refill) and strict parameters checking (extra = "forbid") to shield against parameter injection.
• Observability: Detailed /metrics endpoint measuring uptime, query counts, and database query latency.

---

API Endpoint Schemas

1. GET /health

Returns server status and live vector store collection counts.

• Response Payload:

  {
    "status": "ok",
    "collection_size": 135
  }

2. GET /metrics

Exposes system health, request counts, and real-time CPU/DB average latencies.

• Response Payload:

  {
    "uptime_seconds": 1245.5,
    "total_searches": 142,
    "total_adds": 3,
    "average_embedding_latency_ms": 320.15,
    "average_db_query_latency_ms": 1.45,
    "rate_limiter_capacity": 100,
    "current_rate_limiter_tokens": 98.4
  }

3. POST /search

Performs raw semantic queries against the dense vector space.

• Request Payload:

  {
    "query": "string describing an idea",
    "source_type": "patent | paper | startup | memo | study | all",
    "year_from": 1940,
    "year_to": 2026,
    "limit": 8
  }

• Response Payload:

  {
    "query": "string describing an idea",
    "query_insights": {
      "projection_2d": [2.71, -2.27],
      "intent_category": "Ephemeral Communication & Social Media",
      "semantic_direction": "Semantically aligned with ephemeral message structures..."
    },
    "results": [
      {
        "id": 1779109230159,
        "similarity": 0.7981,
        "score": 0.8123,
        "title": "US6978307B2 — Time-limited electronic communication system",
        "description": "...",
        "year": 2004,
        "source_type": "patent",
        "domain": "software",
        "confidence_label": "Strong Echo",
        "similarity_note": "User added idea",
        "source_url": "https://patents.google.com/patent/US6978307B2/en",
        "explanation": {
          "summary": "This result matches due to shared structural patterns...",
          "temporal_distance": 22,
          "domain_overlap_score": 1.0
        }
      }
    ]
  }

4. POST /add

Creates new concepts in real-time, embeds them immediately, and upserts them to the database.

• Request Payload:

  {
    "title": "string",
    "description": "string",
    "year": 2026,
    "source_type": "patent | paper | startup | memo | product | study",
    "domain": "string",
    "source_url": "string | null"
  }

• Response Payload:

  {
    "status": "success",
    "message": "Idea committed to vector space successfully",
    "id": 1779109230159,
    "collection_size": 135,
    "inserted_item": { ... },
    "closest_ancestors": [ ... ]
  }

---

Vector DB Local Locking Behavior

Because Qdrant local mode is highly optimized and runs completely inside your process, it acts similarly to a local SQLite database. It establishes an exclusive write lock (backend/qdrant_db/.lock) to prevent concurrent processes from accessing or corrupting the local database files.

Seeding Workflow (backend/seed.py)

If you ever want to reset, populate, or bulk-ingest entries from ideas.json:

1. Stop the Uvicorn server (releasing the .lock file).
2. Run the seed script from the root directory:

   ./venv/bin/python backend/seed.py

3. Start the Uvicorn server again to host the API.

---

Project Structure Overview

DejaVec/
├── .gitignore               # Excludes python cache, virtual env, and database locks
├── README.md                # Structural / running documentation
├── requirements.txt         # Clean list of python dependencies
├── backend/                 # Backend module folder
│   ├── __init__.py          # Python package initializer
│   ├── main.py              # API server (serves frontend/dejavec.html at /)
│   ├── qdrant_store.py      # Qdrant queries & custom decays
│   ├── embedder.py          # sentence-transformers models loader
│   ├── seed.py              # Ingestion pipeline
│   ├── ideas.json           # Raw concept dataset source
│   └── qdrant_db/           # Local Qdrant binary data folder
└── frontend/                # Standalone static asset directory
    └── dejavec.html         # SPA client UI

---

Running the Application Locally

1. Install dependencies

pip install -r requirements.txt

2. Start the backend API server

./venv/bin/python -m uvicorn main:app --app-dir backend --port 8000

Once the server is running, the SPA frontend client is natively served at http://127.0.0.1:8000/.