Promotion 2026-09-08 prod (#8291) by nadove-ucsc · Pull Request #8295 · DataBiosphere/azul

nadove-ucsc · 2026-09-10T04:38:26Z

Linked issue: #8291

Checklist

Author

Author (reindex)

This PR is labeled reindex:prod _{or the changes introduced by it will not require reindexing of prod}
This PR is labeled reindex:partial and its description documents the specific reindexing procedure for prod _{or requires a full reindex or is not labeledreindex:prod}

Author (mirror)

This PR is labeled mirror:prod _{or the changes introduced by it will not require mirroring of prod}
This PR is labeled mirror:partial and its description documents the specific mirroring procedure for prod _{or requires a full mirroring or is not labeledmirror:prod}

Author (upgrading deployments)

This PR is labeled upgrade _{or does not require upgrading deployments}
This PR is labeled deploy:shared _{or does not modify docker_images.json, and does not require deploying the shared component for any other reason}
This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

PR branch is up to date (if not, merge prod into PR branch to integrate upstream changes)
PR is not a draft
PR is awaiting requested review from system administrator
Status of PR is Review requested
PR is assigned to only the system administrator and the author

System administrator (after approval)

Actually approved the PR
Labeled PR as no sandbox
N reviews label is accurate
Status of PR is Approved
PR is assigned to only the operator and the author

Operator

Pushed PR branch to GitHub

Operator (deploy `.shared` and `.gitlab` components)

Ran _select prod.shared && CI_COMMIT_REF_NAME=prod make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
Ran _select prod.gitlab && python scripts/create_gitlab_snapshot.py --no-restart (see operator manual for details) _{or this PR is not labeled backup:gitlab}
Ran _select prod.gitlab && CI_COMMIT_REF_NAME=prod make -C terraform/gitlab apply(an error from _login_docker_gitlab is benign if the instance was stopped for backup) _{or this PR is not labeled deploy:gitlab}
Checked the items in the next section _{or this PR is labeled deploy:gitlab}
PR is assigned to only the system administrator and the author _{or this PR is not labeled deploy:gitlab}

System administrator (post-deploy of `.gitlab` component)

Background migrations for prod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
PR is assigned to only the operator and the author

Operator (deploy runner image)

Ran _select prod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}

Operator (merge the branch)

All status checks passed and the PR is mergeable
The title of the merge commit starts with the title of this PR
Added PR # reference to merge commit title
Collected commit title tags in merge commit title _{but excluded any p tags}
Pushed merge commit to GitHub
Status of PR is Merged stable

Operator (main build)

¹ Promoted issues and PRs are referenced in the titles of the commits
that the promotion branch introduces to the stable branch. Prior to the
promotion, the status of promoted issues (PRs) is Lower (Merged lower).
Promoted PRs in status Done do not need to be moved.

Operator (reindex)

Operator (mirroring)

Started mirroring in prod _{or this PR is not labelled mirror:prod}
Checked for, triaged and possibly requeued messages in mirror fail queue in prod _{or this PR is not labelled mirror:prod}
Emptied mirror fail queue in prod _{or this PR is not labelled mirror:prod}

Operator

PR is assigned to only the system administrator

System administrator

Removed unused image tags from pycharm image on DockerHub _{or this promotion does not alter references to that image}
Removed unused image tags from bigquery_emulator image on DockerHub _{or this promotion does not alter references to that image}
PR is assigned to no one

Shorthand for review comments

L line is too long
W line wrapping is wrong
Q bad quotes
F other formatting problem

) Add a dedicated ECR repository for Lambda images in the shared TF component, and a Dockerfile that builds Lambda container images from the mirrored Python slim image with awslambdaric installed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

) Add a lambda_image macro to lambdas.mk that builds platform-specific images (arm64 + amd64) via docker buildx, pushes them to ECR, and creates a manifest list. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add `azul_lambda_image_platforms` environment variable to control which platforms are built, defaulting to amd64 first (determining Lambda architecture) with arm64 second. Make targets now iterate dynamically over this variable. Add `lambda_image_platforms` and `lambda_architecture` config properties with Docker-to-Lambda arch mapping for use in Terraform. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Patch Chalice-generated TF config to use `package_type = "Image"` with `image_uri` and `architectures` instead of ZIP-based `filename`, `source_code_hash`, `runtime`, and `handler` - Remove Lambda layer assignment and runtime management config (both are baked into the Docker image now) - Add `replace_triggered_by` lifecycle rule on Lambda aliases to handle the ZIP-to-Image transition - Move image URI generation and architecture mapping from `azul.config` to `azul.docker` (as module-level functions) - Add `Platform.tag_suffix` property and inline `platform_tag_suffix()` in `manage_images.py` Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Remove create_before_destroy from Lambda permissions. It was added when aliases were introduced, ostensibly to ensure old permissions aren't deleted before new ones are created during alias replacement. But aliases are updated in place (only function_version changes), never replaced, so the permissions are never replaced either and CBD never triggers. Worse, during the Zip-to-Image migration, the function replacement cascades into the permission (whose function_name references the function ARN), and CBD on the permission propagates back to the function, causing a 409 "already exists" error. Remove replace_triggered_by from aliases and suppress the Lambda layer resource, neither of which is needed for image-based Lambda functions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

No longer needed now that Lambda functions use Docker images instead of the AWS-managed Python runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When a Lambda function is replaced (e.g., Zip to Image migration), AWS cascade-deletes the function's aliases. Terraform doesn't know about the cascade and tries to update the now-gone alias, resulting in a 404. Add replace_triggered_by on the alias referencing the function's package_type so the alias is also replaced when the function is replaced. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The .pyc files from chalice package have vendor/ relative paths as co_filename, causing truncated tracebacks in the container. Strip them and recompile in the image so tracebacks resolve to /var/task/ sources. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

awslambdaric's StandardLogSink delimits CloudWatch log events by newline. Multi-line log records (e.g., with tracebacks from exc_info) were split into separate CloudWatch events. A custom formatter now replaces newlines with carriage returns within each record, appending a single newline at the end as the event delimiter. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

With Docker image deployment, runtime dependencies are installed directly in the image via pip. The Lambda layer that previously provided shared dependencies to ZIP-based functions is no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Lambda dependencies are now installed via pip inside the Docker image build, so the checked-in wheels and the azul_chalice_bin machinery are no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The GitLab executor image configures per-registry credential helpers via credHelpers rather than the global credsStore. Handle that case when resolving Lambda image digests during Terraform template rendering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

… PR #8244)

#8245 (comment) Derive HTTP timeout from Lambda context's remaining time Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…e on anvilprod (#8258) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

#7466) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

#8245 (comment) Add RateLimitingCacheService Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…size (DataBiosphere/azul-private#414) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…ate#377) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Both places that construct an Access assert that the URL's scheme matches that of the access method, so the combination in this fixture cannot occur in production. The test got away with it because it patches DRSObject.get, bypassing the code that enforces the invariant, and consequently asserted that a client is redirected to a gs:// URL. While here, compose the URL with furl instead of interpolating it into a string. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…rivate#412) The secret is passed as a multipart form field to the token endpoint, and LoggingHttpClient logs the request body, so no pattern matching it meant it appeared in full. Asserting its redactability before the request is what turns a future gap in the pattern into a visible failure instead of a silent disclosure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…DataBiosphere/azul-private#412) The secret for sandbox was stored with a leading space, which Google rejects and which no redaction pattern matches. Catching that where the secret enters the system beats discovering it when a user fails to sign in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

A header value, a query parameter and a string value in logged JSON were only redacted if the value in its entirety was a secret. Redact any secret occurring in such a value instead. This is a prerequisite for redacting the signature of a signed URL, which is only ever a part of the value carrying it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e#390) Add the redaction pattern so that it is detected where redactions are currently applied A signed URL is a bearer capability, but it is never a secret in its entirety, so redact() only redacted one when the URL appeared inside a larger string. Header and JSON values are redacted with fullmatch, which is why the Location header of a redirect to a signed URL was logged in the clear. Only the signature is redacted, leaving the rest of the URL to be of use in the log. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e#390) Don't rely on a mapping's type to redact it when logging Redaction of the logged query parameters and request headers relied on _LogJSONEncoder.default, which json only invokes for values it cannot serialize natively. It therefore worked only because Chalice happens to represent them as MultiDict and CaseInsensitiveMapping. Were either a plain dict, json would serialize it directly, the encoder would never be consulted and the values would be logged verbatim, without any indication that redaction was skipped. Redact both explicitly instead, which leaves the logged output unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e#390) Redact response headers, too A repository file download answers with a 302 whose Location is a signed URL, a bearer capability that anyone reading the log could use until it expires. The response headers were the only logged request or response property that was exempt from redaction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…zul-private#412, DataBiosphere/azul-private#390, PR #8276)

…ng (#8282) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

codecov · 2026-09-10T04:51:41Z

Codecov Report

❌ Patch coverage is 79.02240% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.79%. Comparing base (3d517a6) to head (8906c81).

Files with missing lines	Patch %	Lines
test/integration_test.py	0.00%	42 Missing ⚠️
src/azul/docker.py	24.13%	22 Missing ⚠️
src/azul/indexer/cache_service.py	87.50%	14 Missing ⚠️
src/azul/infra/terraform.py	7.69%	12 Missing ⚠️
src/azul/oauth2.py	20.00%	4 Missing ⚠️
src/azul/logging.py	40.00%	3 Missing ⚠️
src/azul/terra.py	40.00%	3 Missing ⚠️
src/azul/__init__.py	75.00%	1 Missing ⚠️
src/azul/chalice.py	96.42%	1 Missing ⚠️
src/azul/service/user_controller.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             prod    #8295      +/-   ##
==========================================
+ Coverage   84.43%   84.79%   +0.35%     
==========================================
  Files         169      168       -1     
  Lines       24925    25201     +276     
==========================================
+ Hits        21045    21368     +323     
+ Misses       3880     3833      -47

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

coveralls · 2026-09-10T04:51:46Z

coverage: 84.866% (+0.4%) from 84.512% — promotions/2026-09-08-prod into prod

hannes-ucsc and others added 30 commits August 21, 2026 12:00

Merge branch 'anvilprod' into promotions/2026-08-18-anvilprod

892ffac

[r h] Hotfix: Accumulator overflow for AnVIL (#6793)

c59468b

[r h A R] Promotion 2026-08-18 anvilprod (#8238)#8248

38041a3

Backport b649cdd (PR #8233)

a271b72

Add more Claude skills

ea8ab9e

Merge branch 'anvilprod' into promotions/2026-08-25-anvilprod

7a053a8

[R] Remove Lambda runtime version pinning (#7730)

8a51253

No longer needed now that Lambda functions use Docker images instead of the AWS-managed Python runtime. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Grant GitLab CI permission to push Lambda images to ECR (#7730)

5f6a74e

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[R] Remove pre-downloaded runtime wheels (#7730)

7ff0394

Lambda dependencies are now installed via pip inside the Docker image build, so the checked-in wheels and the azul_chalice_bin machinery are no longer needed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[R] Ran make requirements_update

be48a12

[R] Switch Lambda functions from ZIP to Docker image deployment (#7730,…

2adefba

… PR #8244)

[H 1/2] Fix: URL cache actually overloads DUOS (#8245)

663ae38

#8245 (comment) Derive HTTP timeout from Lambda context's remaining time Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[A] Fix: /index/datasets response exceeds maximum allowed payload siz…

8e4e048

…e on anvilprod (#8258) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[A] Fix: /index/projects response exceeds maximum allowed payload size (

fff0a78

#7466) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[A] Bump service lambda version

84ec0a2

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[H 2/2] Fix: URL cache actually overloads DUOS (#8245)

3138e7a

#8245 (comment) Add RateLimitingCacheService Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

[H] Fix: URL cache's redirect behavior is not documented

e58c1c7

[H] Remove pointless return from doctest helper

fe3ee39

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[H] Fix: 400 from /user/authorize due to refresh token of unexpected …

99f878c

…size (DataBiosphere/azul-private#414) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[H] Fix: Security tokens are logged verbatim (DataBiosphere/azul-priv…

dd2aed4

…ate#377) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

nadove-ucsc and others added 13 commits September 3, 2026 20:46

Update uv to 0.12.9 (#8268)

67ae814

[R] make requirements_update

f1483df

[R] Upgrade software dependencies 2026-08-31 (#8268, PR #8278)

6fb286b

Fix: verbatim log of client secrets and signed URLs (DataBiosphere/a…

fc54279

…zul-private#412, DataBiosphere/azul-private#390, PR #8276)

[H] Fix: TDR snapshot metadata is requested redundantly during indexi…

8ad76f1

…ng (#8282) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

[H] Backport: 5e419ec (#8282, PR #8262, PR #8292)

7899ae5

nadove-ucsc self-assigned this Sep 10, 2026

nadove-ucsc added deploy:gitlab [process] PR requires deploying `gitlab` component deploy:shared [process] PR requires deploying `shared` component deploy:runner [process] PR requires deploying `runner` component labels Sep 10, 2026

nadove-ucsc linked an issue Sep 10, 2026 that may be closed by this pull request

Promotion 2026-09-08 #8291

Open

7 tasks

Merge branch 'prod' into promotions/2026-09-08-prod

8906c81

nadove-ucsc marked this pull request as ready for review September 10, 2026 04:39

nadove-ucsc requested a review from hannes-ucsc as a code owner September 10, 2026 04:39

nadove-ucsc mentioned this pull request Sep 10, 2026

Promotion 2026-09-08 #8291

Open

7 tasks

nadove-ucsc assigned hannes-ucsc Sep 10, 2026

nadove-ucsc added the API API change affecting callers label Sep 10, 2026

hannes-ucsc approved these changes Sep 11, 2026

View reviewed changes

hannes-ucsc added 0 reviews [process] Lead didn't request any changes no sandbox [process] PR will not be tested in the sandbox labels Sep 11, 2026

hannes-ucsc removed their assignment Sep 11, 2026

Navigation Menu

Uh oh!

Promotion 2026-09-08 prod (#8291) - #8295

Promotion 2026-09-08 prod (#8291)#8295
nadove-ucsc wants to merge 121 commits into
prodfrom
promotions/2026-09-08-prod

nadove-ucsc commented Sep 10, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 10, 2026

Uh oh!

coveralls commented Sep 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nadove-ucsc commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Author

Author (reindex)

Author (mirror)

Author (upgrading deployments)

Author (before every review)

System administrator (after approval)

Operator

Operator (deploy .shared and .gitlab components)

System administrator (post-deploy of .gitlab component)

Operator (deploy runner image)

Operator (merge the branch)

Operator (main build)

Operator (reindex)

Operator (mirroring)

Operator

System administrator

Shorthand for review comments

Uh oh!

codecov Bot commented Sep 10, 2026

Codecov Report

Uh oh!

coveralls commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nadove-ucsc commented Sep 10, 2026 •

edited

Loading

Operator (deploy `.shared` and `.gitlab` components)

System administrator (post-deploy of `.gitlab` component)

coveralls commented Sep 10, 2026 •

edited

Loading