feat(search): hybrid Fuse + embedding ranking with better relevance #307

dfordp · 2025-10-25T07:53:59Z

🧠 Summary

Closes #282

This PR upgrades the documentation search experience by introducing a hybrid ranking system that combines Fuse.js fuzzy search (local metadata-based) with backend embedding similarity scores (semantic relevance).
The result is a much more accurate and natural search experience, balancing instant responsiveness with deeper semantic understanding.

✨ What’s Changed

Fuse.js configuration tuned
- Looser threshold (0.25) and ignoreLocation: true for flexible word order
- Added support for excerpt field to provide lightweight context
- Enabled includeScore for hybrid ranking
Backend integration upgraded
- Backend /searchDocUtil now returns both fileKey and similarityScore (cosine similarity)
- Frontend uses these scores to rank results more intelligently
Hybrid ranking system added
- Merges Fuse.js and embedding results using a weighted blend (0.7 * backend + 0.3 * fuse)
- Deduplicates and sorts final results by combined relevance score
Improved user experience
- Faster initial results from Fuse.js, enhanced later by semantic backend results
- More accurate top matches, even with word order or phrasing variations
- Debug info (rank indicators) added for easier tuning

🔍 Why This Matters

Previous search relied heavily on metadata-only matching (title, description, keywords).
Word order and token matching limited recall, and semantic meaning wasn’t considered.
The new system integrates vector-based semantic similarity to better understand user intent while preserving fast local results.
No external dependencies (like Algolia) — still fully self-hosted and open.

🧪 Testing Instructions

Run the website locally (pnpm dev or yarn dev).
Open the search modal and try queries such as:
- “authentication guide”
- “get started with API”
- “token expiration issue”
Observe that:
- Results appear instantly (Fuse.js).
- After a short delay, more relevant results rise to the top (backend embeddings).
- Rankings feel semantically aware (not just keyword-based).

🧩 Affected Areas

apps/website/src/components/DocPage/Search/SearchView.tsx
apps/backend/src/controllers/search.controller.ts
@utils/AI/askDocQuestion/askDocQuestion (used indirectly)

🚀 Future Improvements

Cache embedding results for frequent queries
Experiment with different weight ratios (e.g., 0.6 / 0.4)
Add a contributor debug panel to visualize Fuse vs embedding scores

xenophobic-xenomorph-ts0o88484 · 2025-10-26T23:13:54Z

The preview deployment failed. 🔴

Open Build Logs

Last updated at: 2025-10-26 23:19:48 CET

aymericzip · 2025-10-27T00:02:45Z

build not pass

@intlayer/backend:build:ci: src/controllers/search.controller.ts(33,36): error TS2339: Property 'similarity' does not exist on type 'VectorStoreEl'.

aymericzip

code good to me, can merge once the build pass

aymericzip · 2025-10-27T00:04:53Z

apps/website/src/components/DocPage/Search/SearchView.tsx

+    const fuseScore = normalizeFuse(fuseItem.score);
+    const backendScore = backendMap.get(doc.docKey);
+    const combinedScore = backendScore
+      ? 0.7 * backendScore + 0.3 * fuseScore


avoid magic numbers

aymericzip · 2025-10-27T00:06:29Z

apps/website/src/components/DocPage/Search/SearchView.tsx

-      const backendResultUrls = new Set(
-        backendResults.map((doc) => doc.docKey)
+      const backendDocsWithScore =
+        (searchDocData?.data ?? []).map((d: any) => ({


restore abbreviations to plain variable

aymericzip · 2025-10-27T00:07:15Z

apps/website/src/components/DocPage/Search/SearchView.tsx

+            {results.map((result, i) => (
              <li key={result.url}>
                <SearchResultItem doc={result} onClickLink={onClickLink} />
+                <p className="text-gray-400 text-xs">Rank #{i + 1}</p>


debug only
to remove

aymericzip · 2025-10-27T00:09:06Z

apps/website/src/components/DocPage/Search/SearchView.tsx

+  allDocs: DocMetadata[]
+): DocMetadata[] {
+  const normalizeFuse = (score?: number) => 1 - Math.min((score ?? 1) / 0.5, 1); // invert Fuse score
+  const normalizeBackend = (score: number) => Math.min(score / 1.0, 1); // already cosine-like


no need to divide by 1

…back

dfordp · 2025-10-27T07:09:38Z

fixed the build errors and code review requests

aymericzip

good to me

aymericzip · 2025-10-30T21:30:03Z

Hey @dfordp , thanks it works well! It's much more precise to search a doc using fuse

however, as showed in the screenshot, when fuse.js do not retrieve result, the backend result are not showed anymore
It display no result, but should show the docs send from the backend

dfordp · 2025-10-31T04:12:21Z

Hey @aymericzip, good catch
I’ve added a fallback so that when Fuse returns no matches, we now display backend-only semantic results instead.
This ensures relevant docs from embeddings still show up even if there’s no local fuzzy match.

Thanks for spotting this edge case 🙌

…gration

…kend ranking

dfordp

i feel the changes are all as per the requirement

aymericzip · 2025-10-31T11:40:03Z

aymericzip · 2025-10-31T11:41:39Z

apps/website/src/components/DocPage/Search/SearchView.tsx

-            {results.map((result) => (
-              <li key={result.url}>
-                <SearchResultItem doc={result} onClickLink={onClickLink} />
+            {results.map((r) => (


no abreviation

aymericzip · 2025-10-31T11:42:44Z

apps/backend/src/utils/AI/askDocQuestion/askDocQuestion.ts

+    ...r,
+    similarity:
+      selection.find(
+        (s) => s.fileKey === r.fileKey && s.chunkNumber === r.chunkNumber


no abreviation

aymericzip · 2025-10-31T11:43:42Z

apps/website/src/components/DocPage/Search/SearchView.tsx

+
+type BackendDocResult = { fileKey: string; similarityScore: number };
+
+function mergeHybridResults(


prefer arrow function

aymericzip · 2025-10-31T11:44:46Z

apps/website/src/components/DocPage/Search/SearchView.tsx

  const inputRef = useRef<HTMLInputElement>(null);
  const searchQueryParam = useSearchParams().get('search');
  const [results, setResults] = useState<DocMetadata[]>([]);
+  const [currentQuery, setCurrentQuery] = useState<string | null>(


probably not necessary

use search

aymericzip · 2025-10-31T11:46:26Z

Hey @dfordp

I gave another review,

Please make sure the flow has been tested and works for next submission

dfordp · 2025-11-01T10:09:53Z

this is an update on the same

also included the workspace changes

…endering

dfordp

I think this fixes your required changes

xenophobic-xenomorph-ts0o88484 · 2025-11-01T13:28:31Z

The preview deployment failed. 🔴

Open Build Logs

Last updated at: 2025-11-04 01:38:52 CET

dfordp added 2 commits October 25, 2025 13:21

feat(search): hybrid Fuse + embedding ranking with better relevance

c1b968d

feat(search): hybrid Fuse + embedding ranking with better relevance

432a343

aymericzip reviewed Oct 27, 2025

View reviewed changes

dfordp added 2 commits October 27, 2025 11:37

Merge branch 'main' into main

62cc8c8

refactor(search): clean up hybrid ranking logic and apply review feed…

98ded71

…back

aymericzip approved these changes Oct 30, 2025

View reviewed changes

dfordp closed this Oct 31, 2025

dfordp force-pushed the main branch from 98ded71 to 88599b8 Compare October 31, 2025 07:13

dfordp added 2 commits October 31, 2025 12:45

merge from main

b5381a3

feat(search): add backend-only fallback and clean up hybrid Fuse inte…

8444797

…gration

dfordp reopened this Oct 31, 2025

feat(ai-search): include similarity scores in returned chunks for bac…

4b1d8fc

…kend ranking

dfordp commented Oct 31, 2025

View reviewed changes

aymericzip requested changes Oct 31, 2025

View reviewed changes

aymericzip marked this pull request as draft October 31, 2025 11:47

dfordp marked this pull request as ready for review November 1, 2025 10:09

feat(bun-seearch): include bun workspace fixes and search view docs r…

2f15f6f

…endering

dfordp commented Nov 1, 2025

View reviewed changes

dfordp closed this Nov 1, 2025

dfordp force-pushed the main branch from 2f15f6f to d6d22ee Compare November 1, 2025 13:03

feat(rebase): rebase from main

9d82c00

dfordp reopened this Nov 1, 2025

aymericzip force-pushed the main branch from fb0b0ec to 9cdf9c5 Compare November 4, 2025 02:57

aymericzip force-pushed the main branch from ae09ec7 to b2f7317 Compare November 17, 2025 13:32


		type BackendDocResult = { fileKey: string; similarityScore: number };

		function mergeHybridResults(

Uh oh!

feat(search): hybrid Fuse + embedding ranking with better relevance #307

Are you sure you want to change the base?

feat(search): hybrid Fuse + embedding ranking with better relevance #307

Conversation

dfordp commented Oct 25, 2025

🧠 Summary

✨ What’s Changed

🔍 Why This Matters

🧪 Testing Instructions

🧩 Affected Areas

🚀 Future Improvements

Uh oh!

xenophobic-xenomorph-ts0o88484 bot commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aymericzip commented Oct 27, 2025

Uh oh!

aymericzip left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dfordp commented Oct 27, 2025

Uh oh!

aymericzip left a comment

Choose a reason for hiding this comment

Uh oh!

aymericzip commented Oct 30, 2025

Uh oh!

dfordp commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dfordp left a comment

Choose a reason for hiding this comment

Uh oh!

aymericzip commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aymericzip commented Oct 31, 2025

Uh oh!

dfordp commented Nov 1, 2025

Uh oh!

dfordp left a comment

Choose a reason for hiding this comment

Uh oh!

xenophobic-xenomorph-ts0o88484 bot commented Nov 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xenophobic-xenomorph-ts0o88484 bot commented Oct 26, 2025 •

edited

Loading

dfordp commented Oct 31, 2025 •

edited

Loading

xenophobic-xenomorph-ts0o88484 bot commented Nov 1, 2025 •

edited

Loading