[ROADMAP][1.6] Migrate from LlamaIndex UI to Assistant UI

[MODSetter]

Feature Description

Replace the current LlamaIndex-based chat UI with a modern Assistant UI implementation. The LlamaIndex UI is no longer actively maintained, and we need a more flexible, maintained solution for the new deep agent chat interface.

Target Deployment

• SurfSense Cloud (hosted version)
• Self-hosted version

Problem Statement

The current LlamaIndex UI:

• Is not actively maintained
• Limits customization for our specific use cases
• Doesn't support the advanced features we need (multiplayer, real-time updates)
• May have compatibility issues with future updates

Proposed Solution

1. Evaluate alternatives: Vercel AI SDK UI, custom implementation
2. Design new interface: Modern, Cursor-inspired design
3. Implement streaming: Using Vercel Streaming Protocol
4. Add rich features: Code blocks, citations, tool status, thinking indicators
5. Ensure accessibility: Keyboard navigation, screen reader support

UI Components Needed

• Message list with virtualization (for performance)
• Input area with rich formatting support
• Tool execution status indicators
• Citation/source cards
• Thinking/reasoning display
• File attachment support
• Message actions (copy, regenerate, etc.)

Design Inspiration

• Cursor's chat interface
• ChatGPT's conversation UI
• Linear's clean design aesthetic

Benefits

• Maintained, actively developed foundation
• Full control over UI customization
• Better support for advanced features
• Improved performance and accessibility
• Foundation for multiplayer features

Use Case Examples

1. User interacts with deep agent seeing real-time tool execution
2. User views and navigates cited sources inline
3. User sees agent's reasoning/thinking process
4. User can edit and regenerate messages

Implementation Considerations

• This may require frontend changes (major UI overhaul)
• This may require backend changes (streaming protocol)
• This may require database changes
• This may affect existing features (complete chat UI replacement)

Files Likely Affected

• surfsense_web/components/chat/ - All chat components
• surfsense_web/hooks/use-chat.ts - Chat hooks
• surfsense_web/app/dashboard/[slug]/chats/ - Chat pages
• surfsense_backend/app/routes/ - Streaming endpoints

Acceptance Criteria

• New chat UI renders messages correctly
• Streaming works smoothly without flickering
• Tool execution shows real-time status
• Citations display inline with navigation
• Keyboard shortcuts work (send, new line, etc.)
• Mobile responsive design
• Dark/light mode support
• Performance acceptable for long conversations

Technical Notes

• Use Vercel AI SDK for streaming protocol handling
• Consider react-window or @tanstack/virtual for message virtualization
• Implement optimistic updates for better UX
• Use Framer Motion for smooth animations
• Follow existing Tailwind/shadcn patterns

Related Issues

• Depends on: Issue 1.5 (Deep agent implementation)