Ingest with recursive crawler #2849
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
* feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency
- Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin.
…Framework, and refactor Azure OpenAI embedder
…to ingest configuration
…t and update allowedDomains format
… processLangchainDocument
TheSonOfThomp
force-pushed
the
a/ingest-web-loader
branch
from
25bb9ae to
684c776
Compare
…ructor and update data sources
|
Size Change: -105 B (-0.01%) Total Size: 1.53 MB
ℹ️ View Unchanged
|
tsck
reviewed
May 8, 2025
There was a problem hiding this comment.
Got an unresolved merge conflict here.
There was a problem hiding this comment.
oh weird. how'd it let me commit that
tsck
approved these changes
May 9, 2025
TheSonOfThomp
added a commit
that referenced
this pull request
May 9, 2025
commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 9, 2025
commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 9, 2025
commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 9, 2025
commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 9, 2025
* feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * mv ui & server * Refactor code structure for improved readability and maintainability * feat(ingest): add UMD configuration and update TypeScript settings * Updates package config * feat(chatbot): integrate Azure OpenAI services and update environment variable loading * feat(chatbot): refactor chatbot initialization and update server configuration * fix: update ingest script path and improve TypeScript configuration * feat(ingest): refactor Azure OpenAI embedder integration and remove unused constructor * fix: update mongodb-chatbot-server and mongodb-rag-core dependencies to latest versions * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> * Delete webSourceConstructor.ts * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 21, 2025
* feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 21, 2025
* feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * mv ui & server * Refactor code structure for improved readability and maintainability * feat(ingest): add UMD configuration and update TypeScript settings * Updates package config * feat(chatbot): integrate Azure OpenAI services and update environment variable loading * feat(chatbot): refactor chatbot initialization and update server configuration * fix: update ingest script path and improve TypeScript configuration * feat(ingest): refactor Azure OpenAI embedder integration and remove unused constructor * fix: update mongodb-chatbot-server and mongodb-rag-core dependencies to latest versions * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> * Delete webSourceConstructor.ts * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 21, 2025
* feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
TheSonOfThomp
added a commit
that referenced
this pull request
May 21, 2025
* feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * mv ui & server * Refactor code structure for improved readability and maintainability * feat(ingest): add UMD configuration and update TypeScript settings * Updates package config * feat(chatbot): integrate Azure OpenAI services and update environment variable loading * feat(chatbot): refactor chatbot initialization and update server configuration * fix: update ingest script path and improve TypeScript configuration * feat(ingest): refactor Azure OpenAI embedder integration and remove unused constructor * fix: update mongodb-chatbot-server and mongodb-rag-core dependencies to latest versions * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> * Delete webSourceConstructor.ts * Squashed commit of the following: commit 3eec08f Author: Adam Thompson <2414030+TheSonOfThomp@users.noreply.github.com> Date: Fri May 9 10:46:31 2025 -0400 Ingest with recursive crawler (#2849) * feat(ui): initialize chatbot UI with React and Vite (#2841) * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ui): initialize chatbot UI with React and Vite - Added package.json for UI package with scripts and dependencies. - Created App.css for styling the main application layout. - Implemented App.tsx as the main component integrating the MongoDB Chatbot UI. - Added a React SVG asset for branding. - Created index.css for global styles and theming. - Set up main.tsx as the entry point for the React application. - Added vite-env.d.ts for Vite type definitions. - Configured TypeScript with tsconfig.json and tsconfig.node.json for the UI package. - Created vite.config.ts for Vite configuration with React plugin. * fix: update license to Apache-2.0 and refactor import statements for consistency * feat(ingest): add initial configuration and data sources for chatbot ingestion * update configs * feat(ingest): implement data sources for LeafyGreen UI and MongoDB Chatbot Framework * chore: update pnpm workspace configuration to include 'apps/*' directory * feat(ingest): add data sources for LeafyGreen UI and MongoDB Chatbot Framework, and refactor Azure OpenAI embedder * feat(ingest): add MongoDB Design website data source and integrate into ingest configuration * Delete package-lock.json * reset ui * feat(crawler): export LoadedPageContents and ProcessSingleUrl interfaces * feat(crawler): refactor crawling logic to use processLangchainDocument and update allowedDomains format * feat(crawler): update recursive crawl logic to process documents with processLangchainDocument * feat(crawler): update package.json exports and refactor lodash imports * feat(crawler): implement createWebSourceConstructor and update ingest configuration * Update ingest.config.ts * feat(crawler): replace createWebSourceConstructor with webSourceConstructor and update data sources * fix(crawler): handle invalid URLs in newURL and recursive crawl logic * Resolve tsconfig.json --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com> --------- Co-authored-by: Terrence Keane <terrence.keane@mongodb.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Update ingest to use
recursiveCrawlFromBaseURLfrom@lg-tools/crawlerUpdate
recursiveCrawlFromBaseURLto accept callback instead of baking in the document processing