Obsidian Vault Search — a local-first semantic search trio

English · 繁體中文

Three tools that turn your Obsidian vault into a semantically searchable, AI-chattable knowledge base — running entirely on your own machine. No cloud indexing, no notes leaving your computer for search, no per-query API bill for retrieval.

Built by a physician to study from a vault of thousands of clinical notes, but domain-agnostic: point it at any vault of markdown notes (research, law, engineering, a personal wiki) and it works.

Two ways to use it (one shared index)

This repo ships two front-ends over the same local LanceDB index + retrieval core — use either or both:

• vault-search-obsidian — the Obsidian plugin (Search / Related Notes / Chat in the sidebar), talking to a small local API server. → Quickstart
• vault-search-mcp — an MCP server exposing the same search as tools (vault_search, vault_similar, vault_stats, textbook_search) to Claude Code or any MCP client. No Obsidian required. → Quickstart

Both reuse the same server/ core (indexer · scoring · Personalized-PageRank · knowledge graph), so you index once and query from wherever you like.

---

The three-piece set (三套組)

Tool	What it does	Backed by
🔍 Vault Search	Type a question in natural language (any language) and get the most semantically relevant note sections — not keyword matches.	LanceDB vector search + local embeddings
🔗 Related Notes	A live sidebar that, as you read or select text, surfaces related notes you forgot you wrote — surfacing hidden links across your vault.	Same index, "find similar" mode
💬 Vault Chat	Chat with your notes (RAG). Three modes: Vault (notes only, 100% local & free), Hybrid (notes + AI knowledge), Free (open chat + literature search).	Local LLM or your Claude subscription

All three live in the Obsidian sidebar. The same index also exposes an MCP server, so coding agents like Claude Code can search your vault as a tool.

Screenshots

🔍 Vault Search	🔗 Related Notes

💬 Vault Chat	👁 Context preview

---

Why it exists

Built-in Obsidian search is lexical — it finds the words you typed. But you rarely remember the exact words in a note you wrote months ago. Semantic search finds notes by meaning: search "how to manage a flaccid bladder after spinal injury" and it finds your note titled Neurogenic Bladder even if those exact words never appear.

The design priority is local-first and private:

• Embeddings run locally via Ollama (bge-m3, bilingual, CPU-friendly). Your note text is never sent anywhere to be indexed or searched.
• Vault-mode chat is fully local and free — a local LLM answers using retrieved notes, zero tokens billed.
• Hybrid/Free chat optionally shells out to the claude CLI so you can use a Claude subscription you already pay for, instead of wiring up (and paying per-token for) a raw API key.

---

How it works

                    ┌─────────────────────────────────────────┐
   Obsidian  ◄────► │  Plugin (sidebar): Search · Related · Chat│
                    └───────────────────┬─────────────────────┘
                                        │ HTTP (localhost:3789)
                                        ▼
                    ┌─────────────────────────────────────────┐
                    │  api_server.py  (FastAPI)                 │
                    │   /search  /similar  /chat  /reindex      │
                    └──────┬───────────────────────┬───────────┘
                           │                        │
                  embed query                 RAG context
                           ▼                        ▼
        ┌────────────────────────┐     ┌──────────────────────────┐
        │ Ollama  (bge-m3)        │     │ Vault mode → local LLM    │
        │ → vector               │     │ Hybrid/Free → `claude` CLI │
        └───────────┬────────────┘     └──────────────────────────┘
                    ▼
        ┌────────────────────────┐
        │ LanceDB vector index   │  ◄── indexer.py  (chunks notes by heading,
        │ (~/.vault-search)      │       embeds, stores; incremental updates)
        └────────────────────────┘

   Same index + scoring is also served to coding agents via mcp_server.py (MCP).

• indexer.py walks your vault, splits each note into chunks by heading, embeds them with bge-m3, and stores them in a local LanceDB table. Re-runs are incremental (content-hash cache).
• scoring.py reranks raw vector hits with optional path weighting (trust some folders more), recency (newer notes rank higher), and relation boosts.
• ppr.py ranks notes by Personalized PageRank over the [[wiki-link]] graph — a HippoRAG / LinearRAG-style random-walk-with-restart. It powers Related Notes and multi-hop query expansion (seeding the walk on the top semantic hits), surfacing relevant notes that pure vector similarity misses. On the author's vault this lifted sparse-bridge recall@10 from ~60% to ~84% versus naive 1-hop / shared-link expansion.
• api_server.py serves the plugin: semantic search, find-similar, single-note reindex, and chat (with retrieval-augmented context).
• mcp_server.py exposes the same search to MCP clients (e.g. Claude Code) as vault_search, vault_similar, vault_stats.
• Optional add-ons: a second long-form reference corpus (textbook_indexer.py, parent-child chunking) and a knowledge graph of [[wikilinks]] + entities (graph_builder.py).

---

Prerequisites

Requirement	Notes
Python 3.10+	For the server.
Ollama running locally	Provides embeddings (and, optionally, local chat). bge-m3 runs fine on CPU — a GPU only speeds up bulk indexing.
Obsidian	To use the sidebar plugin. The server can also be used headless via the MCP/HTTP API.
Claude CLI (optional)	Only for Hybrid/Free chat. Vault-mode chat uses a local model instead.
A machine that stays on (optional)	If you want search/chat available to your phone or other devices, run the server on an always-on PC/NAS. For single-machine use, just start it when you open Obsidian.

Pull the embedding model once:

ollama pull bge-m3
# optional, for zero-cost local Vault-mode chat:
ollama pull gemma2:9b

---

Installation

Step 0 — shared core (do this once, for either front-end)

git clone https://github.com/drpwchen/vault-search.git
cd vault-search
pip install -r requirements.txt

cp env.example .env          # then edit .env — set VAULT_PATH to your vault

cd server
python indexer.py            # build the index (incremental on later runs)

ollama pull bge-m3 first if you haven't (see Prerequisites). Now pick one or both front-ends below.

vault-search-obsidian — the Obsidian plugin

Start the local API server, then install the plugin:

python server/api_server.py                       # serves http://localhost:3789
cp -r plugin "<YOUR_VAULT>/.obsidian/plugins/vault-search-plugin"

In Obsidian: Settings → Community plugins → enable "Vault Semantic Search". Open its settings and confirm API Server URL is http://localhost:3789. Three ribbon icons appear: 🔍 Search, 💬 Chat, 🔗 Related Notes.

The plugin ships a data.json.example. Obsidian writes its real data.json on first run; never commit a data.json that contains an API key.

vault-search-mcp — use it from Claude Code (or any MCP client)

No Obsidian, no API server — just register the MCP server against the index you built in Step 0:

claude mcp add vault-search -- python "/abs/path/to/server/mcp_server.py"

Now Claude Code (or any MCP client) can call vault_search, vault_similar, vault_stats, and — if you indexed a reference corpus — textbook_search, all backed by the same local index, Personalized-PageRank expansion, and reranking.

---

Configuration

Everything is driven by environment variables (see env.example for the full annotated list). The only required one is VAULT_PATH. Highlights:

Variable	Default	Purpose
VAULT_PATH	— (required)	Absolute path to your vault.
VAULT_SEARCH_DATA_DIR	~/.vault-search	Where indexes/caches/logs live.
OLLAMA_HOST	http://localhost:11434	Ollama endpoint.
VAULT_SEARCH_EMBEDDING_MODEL	bge-m3	Vault embedding model.
VAULT_SEARCH_API_PORT	3789	API server port.
CLAUDE_CMD	claude	CLI used for Hybrid/Free chat.
VAULT_SEARCH_CHAT_MODEL	gemma2:9b	Local model for Vault-mode chat.
VAULT_SEARCH_PERSONA / _LANGUAGE	generic	Customize the assistant's role & reply language.
VAULT_SEARCH_PATH_WEIGHTS	{}	Boost/demote folders in ranking.
VAULT_SEARCH_EXCLUDE_PATTERNS	(none)	Hide derivative notes (drafts, scratch) from search by default.

---

Optional add-ons

Secondary reference corpus (textbooks / manuals)

If you keep a large corpus of long-form reference docs (converted PDFs, manuals) separate from your notes, index it with parent-child chunking for precise hits with surrounding context:

export VAULT_SEARCH_TEXTBOOK_PATH=/path/to/reference-md
ollama pull qwen3-embedding:0.6b
cd server && python textbook_indexer.py

Adds a textbook_search tool (MCP + HTTP). Optionally rank trusted sources higher with a boost file — see examples/source_boost.example.json and VAULT_SEARCH_SOURCE_BOOST.

Knowledge graph (wikilinks + entities)

cd server
python graph_builder.py            # parse [[wikilinks]] into an adjacency graph
python graph_builder.py --ner      # optional: scispaCy/NER entity extraction

When present, the wiki-link graph drives the Personalized PageRank retrieval in ppr.py (Related Notes + query expansion), and search results are enriched with linked notes and extracted relations. The NER step needs scispacy (see requirements.txt). PPR itself is pure-Python and needs only the wiki-link graph (Phase A) — no NER required.

---

Adapting it to your setup (alternatives)

This project was built around an always-on PC with a GPU + a Claude subscription, but none of that is required. Pick the row that matches you:

You don't have…	Do this instead
A GPU	Nothing — bge-m3 embeds on CPU. Indexing is slower for huge vaults but search is instant. Smaller models (nomic-embed-text) are even lighter; set VAULT_SEARCH_EMBEDDING_MODEL (and _DIM) accordingly and re-index.
A Claude subscription	Use Vault mode for chat — it runs entirely on a local Ollama model (VAULT_SEARCH_CHAT_MODEL), zero cost. Set CLAUDE_CMD= empty to disable Claude-backed modes.
An always-on machine	Just start api_server.py when you open Obsidian (or wrap it in a login item / systemd --user / Task Scheduler entry). Search only needs the server while you're using it.
A second device to reach it	Run the server on a home PC or NAS, set the plugin's API Server URL to that host, and set an API key (VAULT_API_KEY env on the server + matching key in plugin settings) so only you can query it.
Want to avoid running a server at all	Use just the MCP server with Claude Code from the terminal — no FastAPI process, no plugin.

Want a cloud embedder instead of Ollama? The embedding call is isolated in indexer.py / api_server.py (client.embed(...)); swap it for any provider and keep the same LanceDB pipeline.

---

Security notes

• The API server supports an API key (X-API-Key). Set VAULT_API_KEY (env) or ~/.vault-search/api_key.txt on the server and the matching key in plugin settings. Always set one if the server is reachable beyond localhost.
• .env, plugin/data.json, and the whole ~/.vault-search/ data dir are git-ignored. Never commit secrets or your index.
• Hybrid/Free chat runs the claude CLI as a subprocess with a deny-list of tools (no file write, no shell) — it can only read the context you pass it and call read-only literature tools.

---

Repository layout

server/      indexer · scoring · ppr · api_server · mcp_server   (core three-piece set)
             textbook_indexer · graph_builder              (optional add-ons)
             config.py                                      (all settings, env-driven)
plugin/      main.js · manifest.json · styles.css           (Obsidian plugin)
examples/    source_boost.example.json
docs/images/ screenshots
env.example  copy to .env

Author

Built by Dr. P.W. Chen — a physician who needed to actually find things in his own notes. 🌐 drpwchen.com · 🐙 github.com/drpwchen

License

MIT — see LICENSE. Contributions and issues welcome.

🧋 Support

If this tool saves you time, you can buy me a boba — it keeps the servers running 🧡

---

🌱 Start here if you're new to AI agents ／ AI agent 新手起點

This tool is one piece of my personal AI workflow. If you want to learn how to use AI agents like Claude Code from zero (no programming background needed), I wrote a beginner series (in Traditional Chinese):

這個工具是我個人 AI 工作流的一部分。想從零開始學怎麼用 Claude Code 這類 AI agent（不需要程式背景），可以從我的入門系列開始：

1. 從零開始：安裝、看懂 GitHub、跑起你的第一個工具
2. 怎麼跟 AI agent 講話：心法、元技能與規則檔
3. 自動化流程不是設計出來的，是長出來的

Full map of my tools and posts ／ 所有工具與文章的全貌 → drpwchen.com/map