Skip to content

feat: Review crawl without analysis run #2719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

SuaYoo
Copy link
Member

@SuaYoo SuaYoo commented Jul 9, 2025

Resolves #2718

Changes

Enables manual QA review for successfully finished crawls.

Manual testing

  1. Log in as crawler
  2. Click "Crawling"
  3. Click complete or stopped workflow
  4. Click "+ Add Review" in "Latest Crawl" tab. Verify navigation to review page
  5. Click pages in page list. Verify screenshot updates as expected
  6. Approve, reject, and leave comments on pages. Verify pages update as expected
  7. Sort by "Approval". Verify pages update as expected
  8. Click "Finish Review"
  9. Submit review. Verify navigation back to workflow with updated QA Rating
  10. Regression test QA for a crawl with an analysis run

Screenshots

Page Image/video
QA Review Screenshot 2025-07-14 at 12 05 11 PM
QA Review Screenshot 2025-07-14 at 12 05 15 PM
QA Review Screenshot 2025-07-14 at 12 10 36 PM

Follow-ups

We should update the user guide and any related resources #2730

@SuaYoo SuaYoo marked this pull request as ready for review July 9, 2025 19:03
@SuaYoo SuaYoo requested review from ikreymer, emma-sg, DaleLore and tw4l July 9, 2025 19:05
@SuaYoo SuaYoo force-pushed the frontend-qa-review-without-analysis branch from 1e7ad09 to b9ba80b Compare July 10, 2025 02:10
@tw4l
Copy link
Member

tw4l commented Jul 10, 2025

In general, this is looking nice and working well, but I think will require a little more work to make the scenario where users actually do want to run the QA analysis a bit easier/better.

Take the scenario of a new user who clicks on the "Add Review" link from the workflow after a crawl completes. Not being already familiar with our QA system, they'd only know that they can add additional information into the Review Dashboard by running analysis if they dig into a tooltip's text. If they do see that text, there's no button in the Review Dashboard to then go ahead and do that, and no easy link to direct them to the page where they'd have to kick that off. They'd only see a "Return to Workflow" link and "Exit Review", which also takes them back to the workflow. And then they'd have to know to navigate to the crawl object's QA tab on their own.

it might be that we could address this by adding a button to run Analysis directly in the Review Dashboard? That might require some additional UI changes though, e.g. to see the status of the run. Alternatively, we could provide a link to the crawl detail QA tab so that they can do it there, but making users go out to a different page, hit a button, and then come back also feels a little clunky.

In addition, I think we may need to update the docs a bit to reflect the changes made in this PR.

@SuaYoo
Copy link
Member Author

SuaYoo commented Jul 10, 2025

it might be that we could address this by adding a button to run Analysis directly in the Review Dashboard? That might require some additional UI changes though, e.g. to see the status of the run. Alternatively, we could provide a link to the crawl detail QA tab so that they can do it there, but making users go out to a different page, hit a button, and then come back also feels a little clunky.

We show this run dropdown for crawls with analysis runs:

Screenshot 2025-07-10 at 10 06 37 AM

We could add a "Run/Rerun Analysis" option here?

@SuaYoo SuaYoo marked this pull request as draft July 10, 2025 17:08
@tw4l
Copy link
Member

tw4l commented Jul 10, 2025

We could add a "Run/Rerun Analysis" option here?

That seems like a good start! And maybe a link to the crawl's QA pane somewhere?

@SuaYoo SuaYoo force-pushed the frontend-qa-review-without-analysis branch from 8530c68 to d9e7934 Compare July 14, 2025 17:52
@emma-sg
Copy link
Member

emma-sg commented Jul 14, 2025

I feel like "+ Add Review" ends up being a little confusing, I'd expect it to immediately let you add a review/rating, rather than taking you to a separate review page for you to then do the reviewing. Maybe "+ Review Crawl" would make more sense?

@emma-sg
Copy link
Member

emma-sg commented Jul 14, 2025

Also I think the note about "comparison view unavailable" is a useful one to leave in maybe?

@SuaYoo SuaYoo marked this pull request as ready for review July 14, 2025 18:22
@SuaYoo
Copy link
Member Author

SuaYoo commented Jul 14, 2025

Maybe "+ Review Crawl" would make more sense?
Also I think the note about "comparison view unavailable" is a useful one to leave in maybe?

Updated!

Comment on lines +561 to +568
"Screenshot, text, and resource quality metrics are only available for analyzed crawls. Run analysis to view and compare all QA metrics.",
)}
>
<div
class="flex items-center gap-1.5 text-xs text-neutral-500"
>
<sl-icon class="text-sm" name="info-circle"></sl-icon>
${msg("Limited view")}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Open to suggestions on how to label and word this (cc @DaleLore )

Screenshot 2025-07-14 at 12 05 15 PM

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might include "comparison" in a couple places? i.e.

Screenshot comparison, text comparison, and resource quality metrics are only available for analyzed crawls. Run analysis to view and compare all QA metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature]: Enable crawl review without analysis run
3 participants