Skip to content

Conversation

@MarshalX
Copy link
Contributor

@MarshalX MarshalX commented Jun 27, 2023

  • separate documents by batches
  • run many scans in parallel
  • fix races with auth token
  • remove hardcoded scan_id from the context
  • remove hardcoded issue_detected from the context
  • add ability to have many scan_id per CLI session
  • fix scan_id in async scans (generates on BE side)
  • fix report_url
  • add support for scan errors (including timeouts) in batching (UI)
    • collect all errors during the scan and don't show until the end
    • show errors in correct --output format!
      • add JSON format support for exceptions. Comment: not required yet
    • raise polling timeout errors
  • code refactoring. don't use inner functions if possible (for threads)
  • update is_scan_failed logic by issue_detected (we have many issue_detected now)
  • add --no-progress-meter flag
  • disable progress bar for JSON output
  • implement multi-sectioned progress bar
  • integrate progress bar to all scans
  • move all info logs to debug ones. only progress bar should stay
  • remove halo package

Printers:

  • rework text printer
    • add support for aggregated scan results
    • change source of scan_id
    • change source of report_url
    • add support for many scan_id per print (UI)
  • rework json printer
    • add support for aggregated scan results
    • change source of scan_id
    • change source of report_url
    • ❌ add support for many scan_id per print (UI). Comment: I can't do it without changing JSON structure. I was not allowed to change it
  • rework table printer
    • add support for aggregated scan results
    • change source of scan_id
    • change source of report_url
    • add support for many scan_id per print (UI)

@MarshalX MarshalX changed the title CM 22718 - Support scan in batches CM-22718 - Support scan in batches Jun 28, 2023
@MarshalX MarshalX changed the title CM-22718 - Support scan in batches CM-22718 - Support scan in batches and progress bar Jun 30, 2023
@MarshalX MarshalX changed the title CM-22718 - Support scan in batches and progress bar CM-22718 - Support scan in batches with progress bar Jun 30, 2023
@MarshalX
Copy link
Contributor Author

MarshalX commented Jun 30, 2023

progress bar demo (fake progress speed):

Screen.Recording.2023-06-30.at.19.04.51.mov

Each batch has its own scan id. This was not standard behavior for CLI. This is how many scan_id were integrated into printers:

  • text output:
    image
  • table output:
    image
  • JSON is not supported without changing the struct
    image

@MarshalX
Copy link
Contributor Author

MarshalX commented Jul 3, 2023

Errors displaying. UI synced with Maria. One row means one batch. UUID is scan_id for debug purposes:

image

@MarshalX MarshalX marked this pull request as ready for review July 3, 2023 09:24
@MarshalX MarshalX merged commit 657c7fc into main Jul 5, 2023
@MarshalX MarshalX deleted the CM-22718-Support-Scan-in-batches branch July 5, 2023 09:54
@MarshalX
Copy link
Contributor Author

i place it here for future changes

// The current:
{
    "scan_id": "someUuid",
    "detections": [
        {
            "type": "slack-token",
            "detection_details": {...},
            ...
        }
     ]
}

// The new one. That will support batches
{
    "scan_results" : [
        {
            "scan_id": "someUuid",
            "report_url": "someUrl",
            // it's flexible  and we can add more field here in the future
            "detections": [
                {
                    "type": "slack-token",
                    "detection_details": {...},
                    ...
                }
            ]
        }
    ]

}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants