Skip to content

Commit c5b6035

Browse files
marcoienimarcoieni
committed
docs: document gcp backup
1 parent 75138b4 commit c5b6035

File tree

3 files changed

+140
-0
lines changed

3 files changed

+140
-0
lines changed

‎service-catalog/rust-assets-backup/README.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Rust assets backup
2+
3+
## Summary
4+
5+
In GCP (Google Cloud Platform) we keep offsite backups for both Rust releases and crates
6+
to protect us against security threats that could involve losing crates or releases.
7+
These threats were identified in a [threat model] for the project's infrastructure, created by the Rust Foundation's security engineer Walter.
8+
9+
## Motivation
10+
11+
While we have multiple measures in place to prevent accidental deletion of Rust releases or crates in AWS,
12+
e.g. bucket replication to a different region and restricted access, our current setup does not sufficiently protect us against a few threats:
13+
14+
1. _AWS Account compromise_. The [threat model] highlights the risk of an AWS account compromise.
15+
If a malicious actor was able to gain administrator access to the AWS account of one of the [infra-admins],
16+
they could bypass a lot of safe guards and delete data.
17+
2. _AWS Account deletion_. AWS could accidentally delete our account, resulting in the possible deletion of data and backups.
18+
Something similar happened to a customer on [GCP](https://arstechnica.com/gadgets/2024/05/google-cloud-accidentally-nukes-customer-account-causes-two-weeks-of-downtime/) in 2024.
19+
20+
- To mitigate threat 1, the new backup needs to have separate admin access.
21+
- To mitigate threat 2, the new backup needs to be in a separate cloud environment.
22+
23+
## Implementation overview
24+
25+
These new backups are hosted in a dedicated GCP account and have totally separate access controls compared to AWS.
26+
Specifically, none of the current `infra-admins` have admin access to this separate environment to protect against an account compromise.
27+
This GCP account is not used for anything else (just for backups).
28+
29+
The backups are automatically copied daily by GCP.
30+
31+
### Access 👤
32+
33+
We limit admin access to the GCP backups to two members of the Rust Foundation for the following reasons:
34+
35+
- _ensure a strong separation of access_: as explained in the first [motivation](#Motivation), the GCP admins should be different from the AWS admins.
36+
This means we can't give admin access to any of the `infra-admins`.
37+
- _accountability_: The Rust Foundation employees have signed an employment contract and can be legally liable for malicious actions.
38+
39+
People with admin access to the GCP account:
40+
41+
- Joel Marcey (Director of Technology @ Rust Foundation)
42+
- Walter Pearce (Security Engineer @ Rust Foundation)
43+
44+
People with read-only access to the GCP project:
45+
46+
- `infra-admins`
47+
48+
The admin access of the GCP account is bound to the `@rustfoundation.org` Google Workspace account.
49+
This means that if an employee leaves the Rust Foundation, they lose access to the GCP account.
50+
In this case we need to add a new admin to the GCP account.
51+
52+
> [!NOTE]
53+
> The `infra-admins` team can have admin access to the GCP staging project if needed.
54+
55+
### In case of emergency 🧯
56+
57+
- In case our data in AWS is deleted, the `infra-admin` team can restore it by:
58+
- copying the data from GCP to AWS using the GCP read-only access.
59+
- restoring the `crates-io-index` bucket from the `db-dump` stored in the `crates-io` bucket. Use [this](https://github.com/rust-lang/crates.io/blob/e0bb0049daa12f5362def463b04febd6c036d315/src/worker/jobs/git.rs#L19-L129) code.
60+
- If the GCP synchronization mechanism breaks, the Infrastructure team can raise a PR to fix the Terraform configuration and a GCP admin can apply it.
61+
62+
### New threat model 🦹
63+
64+
To delete our data, an attacker would need to compromise both:
65+
66+
- one AWS admin account (an `infra-admin`)
67+
- one GCP admin account (Joel or Walter)
68+
69+
This improves our security posture because compromising two accounts is harder than compromising one.
70+
71+
The accidental account deletion is not a threat anymore because if either AWS or GCP delete our account, we can restore the data from the other provider.
72+
73+
## Implementation details
74+
75+
The account where we store the backup is called `rust-backup`. It contains two GCP projects: `backup-prod` and `backup-staging`.
76+
Here we have one Google [Object Storage](https://cloud.google.com/storage?hl=en) in the `europe-west1` (Belgium) region for the following AWS S3 buckets:
77+
78+
- `crates-io`. CloudFront URL: `cloudfront-static.crates.io`. It contains the crates published by the Rust community.
79+
- `static-rust-lang-org`. CloudFront Url: `cloudfront-static.rust-lang.org`. Among other things, it contains the Rust releases.
80+
81+
For the objects:
82+
83+
- The [storage class](https://cloud.google.com/storage/docs/storage-classes) is set to "archive" for all buckets.
84+
This is the cheapest class for infrequent access.
85+
- [object-versioning](https://cloud.google.com/storage/docs/object-versioning) and [soft-delete](https://cloud.google.com/storage/docs/soft-delete) are enabled,
86+
so that we can recover updates and deletes.
87+
88+
We use [Storage Transfer](https://cloud.google.com/storage-transfer/docs/overview) to automatically transfer the content of the s3 bucket into the Google Object Storage.
89+
This is a service managed by Google. We'll use it to download the S3 buckets from CloudFront to perform a daily incremental transfer. The transfers only move files that are new, updated, or deleted since the last transfer, minimizing the amount of data that needs to be transferred.
90+
91+
## Explanations
92+
93+
- [FAQ](./faq.md)
94+
95+
## How-to Guides
96+
97+
- [Maintenance](./maintenance.md)
98+
99+
[infra-admins]: https://github.com/rust-lang/team/blob/master/teams/infra-admins.toml
100+
[threat model]: https://docs.google.com/document/d/10Qlf8lk7VbpWhA0wHqJj4syYuUVr8rkGVM-k2qkb0QE

‎service-catalog/rust-assets-backup/faq.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Rust Assets Backup: FAQ
2+
3+
## Do we need a multi-region backup for the object storage?
4+
5+
No. [Multi-region](https://cloud.google.com/storage/docs/availability-durability#cross-region-redundancy) only helps if we want to serve this data real-time and we want to have a fallback mechanism if a GCP region fails. We just need this object storage for backup purposes, so we don't need to pay more 👍
6+
7+
## Why did you choose the `europe-west1` GCP region?
8+
9+
It's far from the `us-west-1` region where the AWS S3 buckets are located. This protects us from geographical disasters.
10+
The con is that the latency of the transfer job is higher when compared to a region in the US.
11+
Also, the cost calculator indicates that this regions has a "Low CO2" and it's among the cheapest regions.
12+
13+
## Why GCP?
14+
15+
Both the Rust Foundation and the Rust project have a good working relationship with Google, and it is where the Rust Foundation's Security Initiative hosts its infrastructure.
16+
Due to the good collaboration with Google, we expect that we can cover the costs of the backup with credits provided by Google.

‎service-catalog/rust-assets-backup/maintenance.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Rust Assets Backup: Maintenance
2+
3+
## Monitoring 🕵️
4+
5+
To check that the backups are working:
6+
7+
- Ensure the number of files and the size of the GCP buckets is the same as the respective AWS buckets by looking at the metrics
8+
- Ensure that only the authorized people have access to the account
9+
10+
You can also run the following test:
11+
12+
- Upload a file in an AWS S3 bucket and check that it appears in GCP.
13+
- Edit the file in AWS and check that you can recover the previous version from GCP.
14+
- Delete the in AWS and check that you can recover all previous versions from GCP.
15+
16+
In the future, we might want to create alerts in:
17+
18+
- _Datadog_: to monitor if the transfer job fails.
19+
- _Wiz_: to monitor if the access control changes.
20+
21+
## Backup maintenance 🧹
22+
23+
If a crate version is deleted from the crates-io bucket (e.g. for GDPR reasons), an admin needs to delete it from the GCP bucket as well.
24+
Even though the delete will propagate to GCP, the `soft-delete` feature will preserve the data, so we need to delete it manually.

0 commit comments

Comments
 (0)