fix(kb-uploads): created knowledge, chunks, tags services and use redis for queueing docs in kb

[waleedlatif1]

Summary

created knowledge, chunks, tags services and use redis for queueing docs in kb since doing it in-memory in serverless was not the right method

Type of Change

• Bug fix
• New feature

Testing

Tested manually, see below. Added/updated unit tests

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Screen.Recording.2025-08-26.at.6.00.25.PM.mov

Screen.Recording.2025-08-26.at.10.11.54.PM.mov

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
docs	Ready	Preview	Comment	Aug 27, 2025 6:00am
sim	Ready	Preview	Comment	Aug 27, 2025 6:00am

[greptile-apps]

Greptile Summary

This PR implements a comprehensive refactoring of the knowledge base system, moving from in-memory document processing to a Redis-based queueing architecture for better serverless compatibility. The changes create dedicated service layers for knowledge base operations, chunks, and tags, following proper separation of concerns principles.

Key architectural improvements:

• Service Layer Architecture: Extracted complex database operations from API routes into dedicated service modules (@/lib/knowledge/service.ts, @/lib/knowledge/documents/service.ts, @/lib/knowledge/chunks/service.ts, @/lib/knowledge/tags/service.ts)
• Redis Queue Implementation: Added DocumentProcessingQueue class with Redis-backed job queuing and fallback to in-memory processing for document processing workflows
• File Type Support Expansion: Added parsers for DOC, TXT, and MD files with comprehensive UTF-8 sanitization across all parsers to prevent PostgreSQL encoding issues
• Multipart Upload Support: Implemented batch presigned URL generation and multipart uploads for both S3 and Azure Blob storage to handle large file uploads efficiently
• Enhanced Upload UX: Added real-time progress tracking, file-specific status indicators, and improved error handling in the knowledge base creation modal

Code organization improvements:

• Moved document processing utilities from @/lib/documents/ to @/lib/knowledge/documents/ for better domain organization
• Consolidated embedding utilities into @/lib/embeddings/utils
• Added comprehensive TypeScript interfaces for all knowledge base operations
• Implemented proper validation for file uploads with centralized file type checking

Infrastructure changes:

• Added word-extractor dependency for DOC file parsing
• Updated upload strategies to use batch processing (reduced from 15 to 8 concurrent files)
• Implemented retry mechanisms with exponential backoff for external API calls
• Added comprehensive UTF-8 text sanitization utilities

The refactoring maintains API compatibility while dramatically improving scalability, maintainability, and reliability for knowledge base operations in serverless environments.

Confidence score: 3/5

• This PR requires careful review due to significant architectural changes and complex Redis queue implementation
• Score reflects concerns about error handling in async streaming logic, potential memory leaks in infinite loops, and some missing test coverage for new service layers
• Pay close attention to the Redis queue implementation, service layer error handling, and the removal of some existing test files without replacement

_{66 files reviewed, 25 comments}

_{Edit Code Review Bot Settings | Greptile}