Skip to content

docs: add a load testing runbook for payload offload - #2594

Open
taddes wants to merge 2 commits into
masterfrom
docs/loadtest-runbook-STOR-715
Open

taddes wants to merge 2 commits into
masterfrom
docs/loadtest-runbook-STOR-715

Conversation

@taddes

@taddes taddes commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Add a runbook for load testing the GCS payload offload path with the Molotov and gcloud cli commands, covering VM setup via gcloud, running against dev, and running a local syncserver with raised limits for expanded payloads.

docs/src/tools/payload-offload-load-testing.md

Closes STOR-715

@taddes taddes self-assigned this Sep 22, 2026
@taddes
taddes requested review from chenba and pjenvey September 22, 2026 16:21
**The emulator is not Spanner.** Method B's write throughput and latency numbers
aren't super helpful and indicative of live spanner performance.

**Cloud Armor Adaptive Protection is armed on nonprod.** The policy has an

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this still accurate? I thought we no longer had Cloud Armor after we moved to WAF (or just no longer its firewall?)

| Method B | Write access to a GCS bucket you control. |

Sync's tenant is not onboarded to PAM, so there is no just-in-time elevation
available. `gcloud pam entitlements search` returns nothing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is true (anymore).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, after you added our JIT entitlements. Will update 👍


| For | You need |
| --- | --- |
| Creating the VM | Compute admin on some project. `sync/developers` has **no** compute roles on the sync tenant projects, so use your own project under the developers folder. |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, thx for pointing out. That grants compute.instanceAdmin.v1 on nonprod. Just had it wrong and didn't factor in that change

> fails with `Quota 'CPUS_PER_VM_FAMILY' exceeded. Limit: 0.0`. N2 and E2 draw
> from the general `CPUS` pool, which does have a default allocation. Check with
> `gcloud compute regions describe us-west1 --project=$PROJ --flatten="quotas[]"
> --format="table(quotas.metric,quotas.limit,quotas.usage)"`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm no gcloud compute pro so I was curious about this. And according to https://docs.cloud.google.com/compute/resource-usage#cpu_quota N2 has a separate pool.

```console
sudo apt update && sudo apt install -y git python3 python3-venv
mkdir -p ~/src && cd ~/src
git clone https://github.com/mozilla-services/syncstorage-rs.git

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: clone with --depth=1 and maybe clone specific tag/commit?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I can add --depth=1.

```console
echo hello | gcloud storage cp - gs://<your-bucket>/write-check.txt --project=$PROJ
gcloud storage rm gs://<your-bucket>/write-check.txt --project=$PROJ
```

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand what's being verified here.

@taddes
taddes force-pushed the docs/loadtest-runbook-STOR-715 branch from 338d7f9 to 917d5f1 Compare October 10, 2026 00:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants