feat: Add Docs Chat Widget with RAG-powered Q&A (openclaw#7908)

* feat: add docs chat prototype and related scripts

- Introduced a minimal documentation chatbot that builds a search index from markdown files and serves responses via an API.
- Added scripts for building the index and serving the chat API.
- Updated package.json with new commands for chat index building and serving.
- Created a new Vercel configuration file for deployment.
- Added a README for the docs chat prototype detailing usage and integration.

* feat: enhance docs chat with vector-based RAG pipeline

- Added vector index building and serving capabilities to the docs chat.
- Introduced new scripts for generating embeddings and serving the chat API using vector search.
- Updated package.json with new commands for vector index operations.
- Enhanced README with instructions for the new RAG pipeline and legacy keyword pipeline.
- Removed outdated Vercel configuration file.

* feat: enhance chat widget with markdown rendering and style updates

- Integrated dynamic loading of markdown rendering for chat responses.
- Implemented a fallback for markdown rendering to ensure consistent display.
- Updated CSS variables for improved theming and visual consistency.
- Enhanced chat bubble and input styles for better user experience.
- Added new styles for markdown content in chat bubbles, including code blocks and lists.

* feat: add copy buttons to chat widget for enhanced user interaction

- Implemented copy buttons for chat responses and code blocks in the chat widget.
- Updated CSS styles for improved visibility and interaction of copy buttons.
- Adjusted textarea height for better user experience.
- Enhanced functionality to allow users to easily copy text from chat bubbles and code snippets.

* feat: update chat widget styles for improved user experience

- Changed accent color for better visibility.
- Enhanced preformatted text styles for code blocks, including padding and word wrapping.
- Adjusted positioning and styles of copy buttons for chat responses and code snippets.
- Improved hover effects for copy buttons to enhance interactivity.

* feat: enhance chat widget styles for better responsiveness and scrollbar design

- Updated chat panel dimensions for improved adaptability on various screen sizes.
- Added custom scrollbar styles for better aesthetics and usability.
- Adjusted chat bubble styles for enhanced visibility and interaction.
- Improved layout for expanded chat widget on smaller screens.

* feat: refine chat widget code block styles and copy button functionality

- Adjusted padding and margin for preformatted text in chat responses for better visual consistency.
- Introduced a compact style for single-line code blocks to enhance layout.
- Updated copy button logic to skip short code blocks, improving user experience when copying code snippets.

* feat: add resize handle functionality to chat widget for adjustable panel width

- Implemented a draggable resize handle for the chat widget's sidebar, allowing users to adjust the panel width.
- Added CSS styles for the resize handle, including hover effects and responsive behavior.
- Integrated drag-to-resize logic to maintain user-set width across interactions.
- Ensured the panel resets to default width when closed, enhancing user experience.

* feat: implement rate limiting and error handling in chat API

- Added rate limiting functionality to the chat API, allowing a maximum number of requests per IP within a specified time window.
- Implemented error handling for rate limit exceeded responses, including appropriate headers and retry instructions.
- Enhanced error handling for other API errors, providing user-friendly messages for various failure scenarios.
- Updated README to include new environment variables for rate limiting configuration.

* feat: integrate Upstash Vector for enhanced document retrieval in chat API

- Implemented Upstash Vector as a cloud-based storage solution for document chunks, replacing the local LanceDB option.
- Added auto-detection of storage mode based on environment variables for seamless integration.
- Updated the chat API to utilize the new retrieval mechanism, enhancing response accuracy and performance.
- Enhanced README with setup instructions for Upstash and updated environment variable requirements.
- Introduced new scripts and configurations for managing the vector index and API interactions.

* feat: add create-markdown-preview.js for markdown rendering

- Introduced a new script for framework-agnostic HTML rendering of markdown content.
- The script includes various parsing functions to handle different markdown elements.
- Updated the chat widget to load the vendored version of @create-markdown/preview for improved markdown rendering.

* docs: update README for Upstash Vector index setup and environment variables

- Enhanced instructions for creating a Vector index in Upstash, including detailed settings and important notes.
- Clarified environment variable requirements for both Upstash and LanceDB modes.
- Improved formatting and organization of setup steps for better readability.
- Added health check and API endpoint details for clearer usage guidance.

* feat: add TRUST_PROXY environment variable for IP address handling

- Introduced the TRUST_PROXY variable to control the trust of X-Forwarded-For headers when behind a reverse proxy.
- Updated the README to document the new environment variable and its default value.
- Enhanced the getClientIP function to conditionally trust proxy headers based on the TRUST_PROXY setting.

* feat: add ALLOWED_ORIGINS environment variable for CORS configuration

- Introduced the ALLOWED_ORIGINS variable to specify allowed origins for CORS, enhancing security and flexibility.
- Updated the README to document the new environment variable and its usage.
- Refactored CORS handling in the server code to utilize the ALLOWED_ORIGINS setting for dynamic origin control.

* fix: ensure complete markdown rendering in chat widget

- Added logic to flush any remaining buffered bytes from the decoder, ensuring that all text is rendered correctly in the assistant bubble.
- Updated the assistant bubble's innerHTML to reflect the complete markdown content after streaming completes.

* feat: enhance DocsStore with improved vector handling and similarity conversion

- Added a constant for the distance metric used in vector searches, clarifying the assumption of L2 distance.
- Updated the createTable method to ensure all chunk properties are correctly mapped during table creation.
- Improved the similarity score calculation by providing a clear explanation of the conversion from L2 distance, ensuring accurate ranking of results.

* chore: fix code formatting

* Revert "chore: fix code formatting"

This reverts commit 6721f5b.

* chore: format code for improved readability

- Reformatted code in serve.ts to enhance readability by adjusting indentation and line breaks.
- Ensured consistent style for function return types and object properties throughout the file.

* feat: Update API URL selection logic in chat widget

- Enhanced the API URL configuration to prioritize explicit settings, defaulting to localhost for development and using a production URL otherwise.
- Improved clarity in the code by adding comments to explain the logic behind the API URL selection.

* chore: Update documentation structure for improved organization

- Changed the path for the "Start Here" page to "start/index" for better clarity.
- Reformatted the "Web & Interfaces" and "Help" groups to use multi-line arrays for improved readability.

* feat: Enhance markdown preview integration and improve chat widget asset loading

- Wrapped the markdown preview functionality in an IIFE to expose a global API for easier integration.
- Updated the chat widget to load the markdown preview library dynamically, checking for existing instances to avoid duplicate loads.
- Adjusted asset paths in the chat widget to ensure correct loading based on the environment (local or production).
- Added CORS headers in the Vercel configuration for improved API accessibility.

* fix: Update chat API URL to include '/api' for correct endpoint access

- Modified the chat configuration and widget files to append '/api' to the API URL, ensuring proper endpoint usage in production and local environments.

* refactor: Simplify docs-chat configuration and remove unused scripts

- Removed outdated scripts and configurations related to the docs-chat feature, including build and serve scripts, as well as the associated package.json and README files.
- Streamlined the API URL configuration in the chat widget for better clarity and maintainability.
- Updated the package.json to remove unnecessary scripts related to the now-deleted functionality.

* refactor: Update documentation structure for improved clarity

- Changed the path for the "Start Here" page from "start/index" to "index" to enhance navigation and organization within the documentation.

* chore: Remove unused dependencies from package.json and pnpm-lock.yaml

- Deleted `@lancedb/lancedb`, `@upstash/vector`, and `openai` from both package.json and pnpm-lock.yaml to streamline the project and reduce bloat.

* chore: Clean up .gitignore by removing obsolete entries

- Deleted unused entries related to the docs-chat vector database from .gitignore to maintain a cleaner configuration.

* chore: Remove deprecated chat configuration and markdown preview script

- Deleted the `create-markdown-preview.js` script and the `docs-chat-config.js` file to eliminate unused assets and streamline the project.
- Updated the `docs-chat-widget.js` to directly reference the markdown library from a CDN, enhancing maintainability.

* chore: Update markdown rendering in chat widget to use marked library

- Replaced the deprecated `create-markdown-preview` library with the `marked` library for markdown rendering.
- Adjusted the script loading mechanism to fetch `marked` from a CDN, improving performance and maintainability.
- Enhanced the markdown rendering function to ensure security by disabling HTML pass-through and opening links in new tabs.

* Delete docs/start/index.md

Author: BunsDev