feat(frontend): Refactor crawl status UI and add URL-level monitoring

Introduces a new `CrawlUrls` component to display and manage discovered URLs during a crawl job. This component utilizes Shadcn UI elements (Table, Checkbox, Badge, Tooltip) to provide a detailed view of individual URL statuses, handle URL selection for targeted actions, and display status updates driven by polling managed in `app/page.tsx`.

Key changes include:
- Creation of the `CrawlUrls` component for URL list display and interaction.
- Refactoring of `CrawlStatusMonitor` to focus solely on displaying the overall job status within a Dialog component.
- Updates to `app/page.tsx` to manage essential state (job ID, job status, selected URLs) and orchestrate the polling mechanism for fetching URL-specific status updates.
- Fixed UI bugs where status icons were not updating correctly and checkbox selection state was inconsistent.
- Adjusted the styling of the info icon button for better contrast as per user feedback.

These frontend enhancements align with the ongoing backend redesign, supporting the new job-based status management and polling architecture for more granular progress tracking.

Updated documentation in `docs/features/` (adjust_info_button_style_plan.md, fix_discovered_pages_ui_bugs.md, create_crawl_urls_component_plan.md, crawl_status_monitoring_plan.md) to reflect the completion of related tasks.

Author: Shubham-Khichi

.roo/rules-boomerang/rules.md

@@ -0,0 +1,3 @@
+Special Rules for Critiquing plans and strategies:
+
+Once you have created a subtask markdown file or in memory, you are to get a second opinion from the mode called "Expert Opinion" This mode is designed to only accept your subtask plans and strategies before you present it to the user to approve. You are to do this before you deligate to either Code or Debug modes and after you have created a subtask. Consider this mode as your personal brainstorm. Argue with it from ground truth about the codebase, both you and the Expert Opinion mode as complete knowledge of the codebase. I want you to counter the points and come to a common understanding for the most accurate and right path moving forward and then present those findings to the user. 

.roomodes

@@ -1,22 +1,13 @@
 {
   "customModes": [
-    {
-      "slug": "boomerang-mode",
-      "name": "Boomerang Mode",
-      "roleDefinition": "You are Roo, a strategic workflow orchestrator who coordinates complex tasks by delegating them to appropriate specialized modes. You have a comprehensive understanding of each mode's capabilities and limitations, allowing you to effectively break down complex problems into discrete tasks that can be solved by different specialists.",
-      "customInstructions": "Your role is to coordinate complex workflows by delegating tasks to specialized modes. As an orchestrator, you should:\n\n1. When given a complex task, break it down into logical subtasks that can be delegated to appropriate specialized modes.\n\n2. For each subtask, use the `new_task` tool to delegate. Choose the most appropriate mode for the subtask's specific goal and provide comprehensive instructions in the `message` parameter. These instructions must include:\n    *   All necessary context from the parent task or previous subtasks required to complete the work.\n    *   A clearly defined scope, specifying exactly what the subtask should accomplish.\n    *   An explicit statement that the subtask should *only* perform the work outlined in these instructions and not deviate.\n    *   An instruction for the subtask to signal completion by using the `attempt_completion` tool, providing a concise yet thorough summary of the outcome in the `result` parameter, keeping in mind that this summary will be the source of truth used to keep track of what was completed on this project. \n    *   A statement that these specific instructions supersede any conflicting general instructions the subtask's mode might have.\n\n3. Track and manage the progress of all subtasks. When a subtask is completed, analyze its results and determine the next steps.\n\n4. Help the user understand how the different subtasks fit together in the overall workflow. Provide clear reasoning about why you're delegating specific tasks to specific modes.\n\n5. When all subtasks are completed, synthesize the results and provide a comprehensive overview of what was accomplished.\n\n6. Ask clarifying questions when necessary to better understand how to break down complex tasks effectively.\n\n7. Suggest improvements to the workflow based on the results of completed subtasks.\n\nUse subtasks to maintain clarity. If a request significantly shifts focus or requires a different expertise (mode), consider creating a subtask rather than overloading the current one.",
-      "groups": [],
-      "source": "global"
-    },
     {
       "slug": "boomerang",
       "name": "boomerang ",
-      "roleDefinition": "You are Roo, a strategic workflow orchestrator who coordinates complex tasks by delegating them to appropriate specialized modes. You have a comprehensive understanding of each mode's capabilities and limitations, allowing you to effectively break down complex problems into discrete tasks that can be solved by different specialists.",
-      "customInstructions": "Your role is to coordinate complex workflows by delegating tasks to specialized modes. As an orchestrator, you should:\n\n1. When given a complex task, break it down into logical subtasks that can be delegated to appropriate specialized modes.\n\n2. For each subtask, use the `new_task` tool to delegate. Choose the most appropriate mode for the subtask's specific goal and provide comprehensive instructions in the `message` parameter. These instructions must include:\n    *   All necessary context from the parent task or previous subtasks required to complete the work.\n    *   A clearly defined scope, specifying exactly what the subtask should accomplish.\n    *   An explicit statement that the subtask should *only* perform the work outlined in these instructions and not deviate.\n    *   An instruction for the subtask to signal completion by using the `attempt_completion` tool, providing a concise yet thorough summary of the outcome in the `result` parameter, keeping in mind that this summary will be the source of truth used to keep track of what was completed on this project.\n    *   A statement that these specific instructions supersede any conflicting general instructions the subtask's mode might have.\n\n3. Track and manage the progress of all subtasks. When a subtask is completed, analyze its results and determine the next steps.\n\n4. Help the user understand how the different subtasks fit together in the overall workflow. Provide clear reasoning about why you're delegating specific tasks to specific modes.\n\n5. When all subtasks are completed, synthesize the results and provide a comprehensive overview of what was accomplished.\n\n6. Ask clarifying questions when necessary to better understand how to break down complex tasks effectively.\n\n7. Suggest improvements to the workflow based on the results of completed subtasks.\n\nUse subtasks to maintain clarity. If a request significantly shifts focus or requires a different expertise (mode), consider creating a subtask rather than overloading the current one.",
+      "roleDefinition": "You are Roo, a strategic workflow orchestrator who coordinates complex tasks by delegating them to appropriate specialized modes. You have a comprehensive understanding of each mode's capabilities and limitations, allowing you to effectively break down complex problems into discrete tasks that can be solved by different specialists.\n\nProcedure to follow is to ask the coder for an implementation plan without writing any code yet, forward the implementation plan to the Expert Opinion mode for review, and based the feedback from Expert Opinion mode give the coder the go-ahead to create a updated task list and then ask for user to approve the task list which has the feedbac of Expert Opinion mode and previous plans with pros and cons for each. Your prompts for those roles will have to ensure they know the procedure too, so they’re all singing from the same hymn-sheet.",
+      "customInstructions": "Your role is to coordinate complex workflows by delegating tasks to specialized modes. As an orchestrator, you should:\n\n1. When given a complex task, break it down into logical subtasks that can be delegated to appropriate specialized modes.\n\n2. For each subtask, use the `new_task` tool to delegate. Choose the most appropriate mode for the subtask's specific goal and provide comprehensive instructions in the `message` parameter. These instructions must include:\n    *   All necessary context from the parent task or previous subtasks required to complete the work.\n    *   A clearly defined scope, specifying exactly what the subtask should accomplish.\n    *   An explicit statement that the subtask should *only* perform the work outlined in these instructions and not deviate.\n    *   An instruction for the subtask to signal completion by using the `attempt_completion` tool, providing a concise yet thorough summary of the outcome in the `result` parameter, keeping in mind that this summary will be the source of truth used to keep track of what was completed on this project.\n    *   A statement that these specific instructions supersede any conflicting general instructions the subtask's mode might have.\n    * Once you have the plan created by the coder  forward the implementation plan to the Expert Opinion mode for review, and based on the result ask for improvements or give the coder the go-ahead. Your prompts for those roles will have to ensure they know the procedure too, so they’re all singing from the same hymn-sheet.\n\n3. Track and manage the progress of all subtasks in a markdown file in the codebase. If its a bug then start the heading with BUG: if its a feature then write FEATURE:. When a subtask is completed, analyze its results from the user and determine the next steps and then go back to complete the markdown file subtask. \n\n4. Help the user understand how the different subtasks fit together in the overall workflow. Provide clear reasoning about why you're delegating specific tasks to specific modes.\n\n5. When all subtasks are completed, synthesize the results and provide a comprehensive overview of what was accomplished.\n\n6. Always Ask clarifying questions when necessary to better understand how to break down complex tasks effectively in as much detail as possible. \n\n7. Suggest improvements to the workflow based on the results of completed subtasks.\n\nUse subtasks to maintain clarity. If a request significantly shifts focus or requires a different expertise (mode), consider creating a subtask rather than overloading the current one.",
       "groups": [
         "read",
         "edit",
-        "browser",
         "command",
         "mcp"
       ],

app/page.tsx

@@ -7,11 +7,22 @@ import JobStatsSummary from '@/components/JobStatsSummary' // Import the new sta
 import SubdomainList from '@/components/SubdomainList'
 import StoredFiles from '@/components/StoredFiles'
 import ConfigSettings from '@/components/ConfigSettings'
-import CrawlStatusMonitor from '@/components/CrawlStatusMonitor'
+import CrawlStatusMonitor from '@/components/CrawlStatusMonitor';
+import CrawlUrls from '@/components/CrawlUrls'; // Import the new component
+import { Button } from "@/components/ui/button"; // Import Button
+import {
+  Dialog,
+  DialogContent,
+  DialogDescription,
+  DialogHeader,
+  DialogTitle,
+  DialogTrigger,
+} from "@/components/ui/dialog"; // Import Dialog components
+import { Info } from 'lucide-react'; // Import an icon for the trigger
 import { discoverSubdomains, crawlPages, validateUrl, formatBytes } from '@/lib/crawl-service'
 import { saveMarkdown, loadMarkdown } from '@/lib/storage'
 import { useToast } from "@/components/ui/use-toast"
-import { DiscoveredPage, CrawlJobStatus, OverallStatus } from '@/lib/types' // Import status types
+import { DiscoveredPage, CrawlJobStatus, OverallStatus, UrlStatus } from '@/lib/types' // Import status types & UrlStatus
 import ConsolidatedFiles from '@/components/ConsolidatedFiles' // Import ConsolidatedFiles
 
 export default function Home() {
@@ -111,6 +122,11 @@ const handleSelectionChange = (newSelectedUrls: Set<string>) => {
   setSelectedUrls(newSelectedUrls);
 };
 
+// Handler for status updates from the polling logic
+const handleStatusUpdate = (newStatus: CrawlJobStatus) => {
+  setJobStatus(newStatus);
+};
+
 // Renamed and refactored handler for the "Crawl Selected" button click
 const handleCrawlSelectedClick = async () => {
   // Removed erroneous inner function declaration
@@ -157,6 +173,8 @@ const handleCrawlSelectedClick = async () => {
         title: "Crawl Request Sent",
         description: `Backend acknowledged crawl request for job ${crawlResponse.jobId}. Monitor progress below.`
       })
+      // Clear selection after initiating crawl
+      setSelectedUrls(new Set());
 
       // Clear local markdown state?
       setMarkdown('')
@@ -270,6 +288,8 @@ const handleCrawlSelectedClick = async () => {
           }
         } else {
           console.log(`(Page) Status received for job ${currentJobId}:`, data.overall_status);
+          // Log full status object for debugging icon updates
+          console.log(`(Page) Full status data for job ${currentJobId}:`, JSON.stringify(data, null, 2));
           setJobStatus(data);
           setJobError(null); // Clear previous errors on success
 
@@ -316,7 +336,7 @@ const handleCrawlSelectedClick = async () => {
       <header className="w-full py-12 bg-gradient-to-r from-gray-900/50 to-gray-800/50 backdrop-blur-sm border-b border-gray-700">
         <div className="container mx-auto px-4 text-center">
           <h1 className="text-5xl font-bold bg-clip-text text-transparent bg-gradient-to-r from-blue-400 to-purple-500 mb-4">
-            DevDocs
+            DevDocs by CyberAGI Inc
           </h1>
           <p className="text-gray-300 text-lg max-w-2xl mx-auto">
             Discover and extract documentation for quicker development
@@ -337,27 +357,50 @@ const handleCrawlSelectedClick = async () => {
           <UrlInput onSubmit={handleSubmit} />
         </div>
 
-        {/* Render CrawlStatusMonitor and pass the currentJobId */}
-        <div className="bg-gray-800/50 backdrop-blur-lg rounded-2xl p-6 border border-gray-700 shadow-xl">
-           <h2 className="text-2xl font-semibold mb-4 text-green-400">Discovered Pages</h2>
-           {/* Log jobStatus before passing to monitor */}
-           {(() => {
-             if (jobStatus) {
-               console.log("page.tsx: Passing jobStatus to CrawlStatusMonitor:", JSON.stringify(jobStatus));
-             }
-             return null; // Return null to render nothing
-           })()}
-           <CrawlStatusMonitor
-             jobId={currentJobId}
-             status={jobStatus}
-             error={jobError}
-             isLoading={isPollingLoading}
-             selectedUrls={selectedUrls}
-             isCrawlingSelected={isCrawlingSelected}
-             onSelectionChange={handleSelectionChange}
-             onCrawlSelected={handleCrawlSelectedClick}
-           />
-        </div>
+        {/* Render the new CrawlUrls component and the Dialog trigger */}
+        {currentJobId && ( // Only render if there's an active job
+          <div className="bg-gray-800/50 backdrop-blur-lg rounded-2xl p-6 border border-gray-700 shadow-xl">
+            <div className="flex justify-between items-center mb-4">
+              <h2 className="text-2xl font-semibold text-cyan-400">Crawl Queue</h2>
+              {/* Dialog Trigger Button */}
+              <Dialog>
+                <DialogTrigger asChild>
+                  <Button variant="outline" size="icon" className="bg-white text-black hover:bg-gray-100 hover:text-black">
+                    <Info className="h-4 w-4" />
+                    <span className="sr-only">View Overall Job Status</span>
+                  </Button>
+                </DialogTrigger>
+                <DialogContent className="sm:max-w-[625px] bg-gray-900 border-gray-700 text-gray-300">
+                  <DialogHeader>
+                    <DialogTitle className="text-green-400">Overall Job Status</DialogTitle>
+                    <DialogDescription>
+                      Detailed status for Job ID: {currentJobId}
+                    </DialogDescription>
+                  </DialogHeader>
+                  {/* Render CrawlStatusMonitor inside the Dialog */}
+                  <CrawlStatusMonitor
+                    jobId={currentJobId}
+                    status={jobStatus}
+                    error={jobError}
+                    isLoading={isPollingLoading}
+                    onStatusUpdate={handleStatusUpdate} // Keep this handler
+                  />
+                </DialogContent>
+              </Dialog>
+            </div>
+            {/* CrawlUrls Component */}
+            <CrawlUrls
+              jobId={currentJobId}
+              urls={jobStatus?.urls || {}}
+              selectedUrls={selectedUrls}
+              onSelectionChange={handleSelectionChange}
+              onCrawlSelected={handleCrawlSelectedClick}
+              isCrawlingSelected={isCrawlingSelected}
+            />
+          </div>
+        )}
+
+        {/* Removed the separate temporary container for CrawlStatusMonitor */}
 
         {/* Keep SubdomainList for now, but it might be replaced by CrawlStatusMonitor's display */}
         {/* Log if legacy SubdomainList condition is met */}