Enable mirroring in anvildev and anvilbox (#7154, #7214)

[nadove-ucsc]

Connected issues: #7154, #7214

Checklist

Author

• PR is a draft
• Target branch is develop
• Name of PR branch matches issues/<GitHub handle of author>/<issue#>-<slug>
• On ZenHub, PR is connected to all issues it (partially) resolves
• PR description links to connected issues
• PR title matches¹ that of a connected issue _{or comment in PR explains why they're different}
• PR title references all connected issues
• For each connected issue, there is at least one commit whose title references that issue

¹ when the issue title describes a problem, the corresponding PR
title is Fix: followed by the issue title

Author (partiality)

• Added p tag to titles of partial commits
• This PR is labeled partial _{or completely resolves all connected issues}
• This PR partially resolves each of the connected issues _{or does not have the partial label}

Author (chains)

• This PR is blocked by previous PR in the chain _{or is not chained to another PR}
• The blocking PR is labeled base _{or this PR is not chained to another PR}
• This PR is labeled chained _{or is not chained to another PR}

Author (reindex, API changes)

• Added r tag to commit title _{or the changes introduced by this PR will not require reindexing of any deployment}
• This PR is labeled reindex:dev _{or the changes introduced by it will not require reindexing of dev}
• This PR is labeled reindex:anvildev _{or the changes introduced by it will not require reindexing of anvildev}
• This PR is labeled reindex:anvilprod _{or the changes introduced by it will not require reindexing of anvilprod}
• This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
• This PR is labeled reindex:partial and its description documents the specific reindexing procedure for dev, anvildev, anvilprod and prod _{or requires a full reindex or carries none of the labels reindex:dev, reindex:anvildev, reindex:anvilprod and reindex:prod}
• This PR and its connected issues are labeled API _{or this PR does not modify a REST API}
• Added a (A) tag to commit title for backwards (in)compatible changes _{or this PR does not modify a REST API}
• Updated REST API version number in app.py _{or this PR does not modify a REST API}

Author (upgrading deployments)

• Ran make docker_images.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (hotfixes)

• Added F tag to main commit title _{or this PR does not include permanent fix for a temporary hotfix}
• Reverted the temporary hotfixes for any connected issues _{or the none of the stable branches (anvilprod and prod) have temporary hotfixes for any of the issues connected to this PR}

Author (before every review)

• Rebased PR branch on develop, squashed fixups from prior reviews
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

Peer reviewer (after approval)

• Actually approved the PR
• PR is not a draft
• Ticket is in Review requested column
• PR is awaiting requested review from system administrator
• PR is assigned to only the system administrator

System administrator (after approval)

• Actually approved the PR
• Labeled connected issues as demo or no demo
• Commented on connected issues about demo expectations _{or all connected issues are labeled no demo}
• Decided if PR can be labeled no sandbox
• A comment to this PR details the completed security design review
• PR title is appropriate as title of merge commit
• N reviews label is accurate
• Moved connected issues to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Checked reindex:… labels and r commit title tag
• Checked that demo expectations are clear _{or all connected issues are labeled no demo}
• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• PR is assigned to only the system administrator _{or this PR is not labeled deploy:gitlab}

System administrator

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Added sandbox label _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab dev _{or PR is labeled no sandbox}
• Pushed PR branch to GitLab anvildev _{or PR is labeled no sandbox}
• Build passes in sandbox deployment _{or PR is labeled no sandbox}
• Build passes in anvilbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in sandbox deployment _{or PR is labeled no sandbox}
• Reviewed build logs for anomalies in anvilbox deployment _{or PR is labeled no sandbox}
• Deleted unreferenced indices in sandbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in dev}
• Deleted unreferenced indices in anvilbox _{or this PR does not remove catalogs or otherwise causes unreferenced indices in anvildev}
• Started reindex in sandbox _{or this PR is not labeled reindex:dev}
• Started reindex in anvilbox _{or this PR is not labeled reindex:anvildev}
• Checked for failures in sandbox _{or this PR is not labeled reindex:dev}
• Checked for failures in anvilbox _{or this PR is not labeled reindex:anvildev}
• PR is assigned to only the system administrator

System administrator

• Mirrored anvilbox
• PR is assigned to only the operator

Operator

• All status checks passed and the PR is mergeable
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but only included p if the PR is also labeled partial}
• Moved connected issues to Merged lower column in ZenHub
• Moved blocked issues to Triage _{or no issues are blocked on the connected issues}
• Pushed merge commit to GitHub

Operator (chain shortening)

• Changed the target branch of the blocked PR to develop _{or this PR is not labeled base}
• Removed the chained label from the blocked PR _{or this PR is not labeled base}
• Removed the blocking relationship from the blocked PR _{or this PR is not labeled base}
• Removed the base label from this PR _{or this PR is not labeled base}

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev

Operator (reindex)

• Deindexed all unreferenced catalogs in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed all unreferenced catalogs in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Deindexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Deindexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Indexed specific sources in dev _{or this PR is neither labeled reindex:partial nor reindex:dev}
• Indexed specific sources in anvildev _{or this PR is neither labeled reindex:partial nor reindex:anvildev}
• Started reindex in dev _{or this PR does not require reindexing dev}
• Started reindex in anvildev _{or this PR does not require reindexing anvildev}
• Checked for, triaged and possibly requeued messages in both fail queues in dev _{or this PR does not require reindexing dev}
• Checked for, triaged and possibly requeued messages in both fail queues in anvildev _{or this PR does not require reindexing anvildev}
• Emptied fail queues in dev _{or this PR does not require reindexing dev}
• Emptied fail queues in anvildev _{or this PR does not require reindexing anvildev}
• Restarted the Data Browser pipeline for the ucsc/hca/dev branch on GitLab in dev _{or this PR does not require reindexing dev}
• Restarted the Data Browser pipeline for the ucsc/lungmap/dev branch on GitLab in dev _{or this PR does not require reindexing dev}
• Restarted deploy_browser job in the GitLab pipeline for this PR in dev _{or this PR does not require reindexing dev}
• Restarted the Data Browser pipeline for the ucsc/anvil/anvildev branch on GitLab in anvildev _{or this PR does not require reindexing anvildev}
• Restarted deploy_browser job in the GitLab pipeline for this PR in anvildev _{or this PR does not require reindexing anvildev}
• PR is assigned to only the system administrator

System administrator

• Mirrored anvildev
• PR is assigned to only the operator

Operator

• Propagated the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels to the next promotion PRs _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, API, reindex:partial, reindex:anvilprod and reindex:prod labels, from the description of this PR to that of the next promotion PRs _{or this PR carries none of these labels}
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

Attention: Patch coverage is 54.54545% with 5 lines in your changes missing coverage. Please review.

Project coverage is 85.02%. Comparing base (ffe1c17) to head (0d5583f).

Files with missing lines	Patch %	Lines
src/azul/plugins/repository/tdr_anvil/__init__.py	54.54%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7223      +/-   ##
===========================================
- Coverage    85.03%   85.02%   -0.02%     
===========================================
  Files          156      156              
  Lines        22460    22471      +11     
===========================================
+ Hits         19099    19105       +6     
- Misses        3361     3366       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[coveralls]

coverage: 85.202% (-0.02%) from 85.217%
when pulling 0d5583f on issues/nadove-ucsc/7154-anvil-uses-nonstandard-md5-enconding
into ffe1c17 on develop.

[dsotirho-ucsc]

Approved.

[hannes-ucsc]

This will get conflicts once the backport I just approved land on develop.

It's not evident from the title of that commit that removes the snapshot what the defect is. Either add a ticket reference or describe the defect in the body of the commit message. Probably the former is warranted so that we can point our collaborators at it.

The base branch was changed.

[hannes-ucsc]

While the link in the message body of the middle commit is helpful, the body should be self-contained and include a description of the alleged defect.

[hannes-ucsc]

The removal of a snapshot requires reindexing so there should be notes in the description and labels on the PR. As per convention, title and body of a commit message should be separated by a blank line.

[hannes-ucsc]

#7223 (review)

[nadove-ucsc]

Agreed at PL that only needed change is added the reindex instructions to the PR description.

[nadove-ucsc]

Agreed on Slack that the PR does not need specific reindex instructions.