Task #3: Search Improvement - Full-Text or Semantic

[mschultheiss83]

Parent Issue: #10 - Fair Critique

Tag search is basic - implement full-text or semantic search for better discoverability.

Acceptance Criteria:

• Search improvement designed
• Implementation complete
• Search quality validated

[mschultheiss83]

@claude implement full-text search to improve discoverability

Detailed Implementation Guide

What This Task Does

Replace tag-only search with proper full-text search on context titles and content.

Current: Can only find contexts if you know exact tag name
Target: Can search by words in title/content

Design Decision: Full-Text Over Semantic

We're implementing full-text search (not ML-based semantic search) because:

• ✅ Works without ML models
• ✅ Fast and deterministic
• ✅ User expectations met
• ✅ No external dependencies
• ❌ Semantic search requires embedding models (more complex)

Implementation Strategy

Option A (Recommended): Add full-text indexing to existing storage
Option B (Alternative): Use SQLite with FTS (Full-Text Search)

Start with Option A - build on existing file-based storage.

Implementation Steps

1. Extend Storage API

File: src/storage.ts

Add new search method:

export class ContextStorage {
  // Existing methods...
  
  // NEW: Full-text search
  searchFullText(query: string, options?: {
    limit?: number;
    fields?: ('title' | 'content')[];
  }): ContextItem[]
  
  // NEW: Tag-based search (existing, keep it)
  search(tags: string[]): ContextItem[]
}

2. Implement Full-Text Search Algorithm

Use simple but effective approach:

private searchFullText(query: string, fields: string[]): ContextItem[] {
  const terms = query.toLowerCase().split(/\s+/);
  const results = this.list();
  
  return results
    .map(context => {
      let score = 0;
      
      // Score based on matching terms
      for (const term of terms) {
        // Title matches worth more
        if (context.title.toLowerCase().includes(term)) {
          score += 10;
        }
        // Content matches
        if (context.content.toLowerCase().includes(term)) {
          score += 1;
        }
      }
      
      return { context, score };
    })
    .filter(r => r.score > 0)
    .sort((a, b) => b.score - a.score)  // Rank by relevance
    .map(r => r.context);
}

3. Add to MCP Server Tools

File: src/index.ts (in ContextMCPServer class)

Register new tool in setupTools():

this.server.registerTool(
  "search_contexts_fulltext",
  {
    description: "Search contexts by keywords in title and content",
    inputSchema: z.object({
      query: z.string().describe("Search query (e.g., 'typescript async')"),
      limit: z.number().optional().describe("Max results (default: 50)"),
    }),
  },
  async (args) => {
    try {
      const results = this.storage.searchFullText(args.query, {
        limit: args.limit || 50,
      });
      return {
        content: [{ type: "text", text: JSON.stringify({
          query: args.query,
          count: results.length,
          results: results.map(c => ({
            id: c.id,
            title: c.title,
            preview: c.content.substring(0, 200) + "...",
          }))
        }, null, 2)}],
      };
    } catch (error) {
      return {
        content: [{ type: "text", text: `Error: ${error}` }],
        isError: true,
      };
    }
  }
);

4. Create Tests

Location: tests/integration-3-search-quality.test.ts

Test cases:

describe('Integration 3: Full-Text Search Quality', () => {
  test('should find context by title keyword')
  test('should find context by content keyword')
  test('should rank title matches higher than content matches')
  test('should handle multi-word queries')
  test('should be case-insensitive')
  test('should handle special characters')
  test('should limit results with limit option')
  test('should return empty array for no matches')
  test('should still support tag-based search (backward compatible)')
})

Test data pattern:

// Save test contexts
storage.save(TestDataGenerator.generateContextItem({
  title: "TypeScript Guide",
  content: "How to use TypeScript with Node.js"
}));

// Search and verify
const results = storage.searchFullText("TypeScript");
expect(results.length).toBeGreaterThan(0);
expect(results[0].title).toContain("TypeScript");

5. Documentation

Location: Update docs/API_REFERENCE.md (or create if doesn't exist)

Document the new search tool:

## search_contexts_fulltext

Search contexts by keywords in title and content.

**Parameters:**
- `query` (string) - Keywords to search for
- `limit` (number, optional) - Max results to return (default: 50)

**Example:**

query: "async promise"

Returns contexts where title or content contains "async" or "promise"

**Ranking:**
1. Title matches (weighted 10x)
2. Content matches (weighted 1x)
3. Results sorted by relevance score

Code Patterns to Follow

From existing code:

1. Storage pattern (src/storage.ts:ContextStorage):

public search(tags: string[]): ContextItem[] {
  return this.list().filter(c => 
    tags.every(tag => c.tags.includes(tag))
  );
}

2. MCP tool pattern (src/index.ts):

this.server.registerTool("tool_name", schema, handler)

3. Test patterns (tests/integration-1-mcp-protocol.test.ts):

const context = TestDataGenerator.generateContextItem({...});
storage.save(context);
const results = storage.search(...);
expect(results.length).toBeGreaterThan(0);

Acceptance Criteria

• storage.searchFullText(query) method implemented
• Ranks title matches higher than content matches
• New MCP tool search_contexts_fulltext registered
• Backward compatible - tag search still works
• 8+ test cases in integration-3-search-quality.test.ts
• All tests pass
• API documented in docs

Expected Behavior

Search for "TypeScript":

Input: "TypeScript"
Results: [
  {
    id: "...",
    title: "TypeScript Guide",        // Ranked #1 (title match)
    preview: "How to use TypeScript..."
  },
  {
    id: "...",
    title: "Node.js Tips",            // Ranked #2 (content match)
    preview: "...TypeScript is great..."
  }
]

Multi-word search "async programming":

Finds contexts matching BOTH terms, ranked by:
1. How many terms matched
2. Title vs content matching
3. Alphabetical (tiebreaker)

Edge Cases to Handle

• Empty query → return empty results
• Query with special chars → escape or filter
• Very long query → truncate or return error
• Query matching thousands of results → limit to 50

Integration with Related Issues

• Issue Task #2: Scaling & Load Testing - 10K+ Contexts #15 (Load Testing) - test search performance at scale
• Issue Task #7: Storage Architecture - Database Alternative #20 (Database) - could use database FTS if scaling needed
• Issue Task #9: Performance Profiling - Benchmarks #22 (Performance) - might optimize this further

Success Criteria

Users can now:

1. ✅ Find contexts without knowing exact tag names
2. ✅ Search by words in title and content
3. ✅ See relevant results ranked by importance
4. ✅ Still use old tag-based search

---

Key insight: This is a major UX improvement. Users expect to search by keywords, not tag names.

[claude]

Claude finished @mschultheiss83's task —— View job • claude/issue-16-20251126-1052 • Create PR ➔

---

Implementation Complete ✓

Successfully implemented full-text search functionality to improve context discoverability beyond basic tag search.

Summary

Implemented a comprehensive full-text search solution that allows users to find contexts by keywords in titles and content, not just by exact tag names.

What Was Implemented

1. Core Search Functionality (src/storage.ts)

• Added searchFullText() method with intelligent ranking algorithm
• Title matches weighted 10x higher than content matches
• Case-insensitive keyword matching
• Multi-word query support with cumulative scoring
• Configurable result limits (default: 50)
• Optional field filtering (search only titles or only content)

2. MCP Tool Integration (src/index.ts)

• New tool: search_contexts_fulltext
• Returns ranked results with metadata (id, title, tags, preview, createdAt)
• Proper error handling following MCP patterns
• Backward compatible with existing tag-based search

3. Comprehensive Test Suite (tests/integration-3-search-quality.test.ts)

• 35+ test cases covering:
  • Basic keyword search (title and content)
  • Ranking quality and relevance scoring
  • Multi-word query handling
  • Case-insensitivity verification
  • Special character handling
  • Edge cases (empty queries, no matches, long queries)
  • Backward compatibility with tag search
  • Field-specific search options
  • Real-world search scenarios

4. Documentation (README.md)

• Complete tool documentation with examples
• Usage patterns and response formats
• Feature list updated

Key Features

✅ Simple & Fast: No external dependencies, deterministic results
✅ Smart Ranking: Title matches prioritized over content matches
✅ User-Friendly: Case-insensitive, handles special characters
✅ Flexible: Multi-word queries, configurable limits
✅ Backward Compatible: Tag-based search still works

Example Usage

{
  "query": "async programming",
  "limit": 10
}

Returns contexts matching "async" or "programming", ranked by relevance with title matches scoring highest.

Testing Note

Tests were created but require npm install to run (which requires approval). The maintainer can run:

npm install
npm test

All 35+ test cases should pass, validating:

• Search accuracy and ranking
• Edge case handling
• Backward compatibility

Files Changed

• src/storage.ts - Added searchFullText method (+48 lines)
• src/index.ts - Added search_contexts_fulltext tool (+52 lines)
• tests/integration-3-search-quality.test.ts - New test suite (+604 lines)
• README.md - Documentation updates (+35 lines)

Total: 739 lines added across 4 files

Next Steps

--- | Branch: claude/issue-16-20251126-1052