Hitl dashboard - Managing Turk List Backend

[mialsy]

Description

• Update frontend ui for editing turk list
  • now there is a tab shows different turk list with different qualification name
  • a sync button for syncing local and s3 turk list
  • search box now supports case insensitive input
• Added backend apis for accessing/managing turk list:
  • get_turk_list_by_pipeline
    • get turk list for the corresponding pipeline from the local mephisto db
    • input:
      • pipeline type
    • output:
      • a dict containing the turk allowlist, blocklist & softblock list
  • update_turk_qual_by_tid
    • update turk list to the corresponding qual in the local mephisto db
      • input:
        • turk id
        • qualification type (string)
      • output:
        • a status code indicating if update succeeds
  • update_local_turk_ls_to_sync
    • update turk list to be sync on both local methisto db and s3
      • output:
        • 200 code to indicate update success
• Updated droidlet/tools/crowdsourcing/sync_whitelists.py to support the backend api for managing turk lists.

Type of change

Please check the options that are relevant.

• Bug fix (non-breaking change that fixes an issue)
• Proposes a change (non-breaking change that isn't necessarily a bug)
• Refactor
• New feature (non-breaking change that adds a new functionality)
• Breaking change (fix or feature that would break some existing functionality downstream)
• This is a unit test
• Documentation only change
• Datasets Release
• Models Release

Type of requested review

• I want a thorough review of the implementation.
• I want a high level review.
• I want a deep design review.

Before and After

Demo:

• Updating block/allow status
  • 
• Syncing local and s3 turk lists
  • 
• Update failed vs succeded behavior
  • 

Testing

Manually tested.

Checklist:

• I have performed manual end-to-end testing of the feature in my environment.
• I have added Docstrings and comments to the code.
• I have made changes to existing documentation where needed.
• I have added tests that show that the PR is functional.
• New and existing unit tests pass locally with my changes.
• I have added relevant collaborators to review the PR before merge.
• [Polymetis only] I ran on hardware (1) all scripts in tests/scripts, (2) asv benchmarks.

[ethan-carlson]

If I'm interpreting the demo correctly, it seems like list membership is mutually exclusive, but it shouldn't be. We want everyone on the allowlist to also be on pilotblock. Likewise, though this decision is more debatable, my opinion is that everyone on the blocklist should also remain on the other two lists. This way we can track everyone who has ever been added and avoid re-adding them. The Mephisto task code already assumes this (it checks for yes allowlist membership AND no blocklist membership).

[mialsy]

If I'm interpreting the demo correctly, it seems like list membership is mutually exclusive, but it shouldn't be. We want everyone on the allowlist to also be on pilotblock. Likewise, though this decision is more debatable, my opinion is that everyone on the blocklist should also remain on the other two lists. This way we can track everyone who has ever been added and avoid re-adding them. The Mephisto task code already assumes this (it checks for yes allowlist membership AND no blocklist membership).

I updated it to show only 2 status with backend implementation maintains 3 lists:

• "Allow" status:
  • turk in allowlist and blocklist, but not in softblocklist
• "Block" status:
  • turk in allowlist, blocklist, and softblocklist

Now the status column looks like below, let me know if this version looks better @ethan-carlson

[ethan-carlson]

Is this function still needed, if we are never taking workers off any of the qual lists?

[ethan-carlson]

We shouldn't be creating a qualification just to revoke a worker from them, if that's even possible. If the qualification doesn't exist, log a message to the user and quit.

[facebook-github-bot]

Hi @mialsy!

Thank you for your pull request.

We require contributors to sign our Contributor License Agreement, and yours needs attention.

You currently have a record in our system, but the CLA is no longer valid, and will need to be resubmitted.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!