You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Add in-memory storage fallback for single-instance deployments
- Redis now only required for HA/multi-replica deployments
- Rewrite README with clearer Quick Start and How It Works sections
- Add comprehensive Kubernetes/Helm production deployment docs
- Document credential model and why proxy needs credentials
# Works with any S3 client/SDK - just change the endpoint URL
90
+
```
91
+
92
+
Your file is now encrypted at rest with AES-256-GCM. The encryption is transparent—your application code doesn't change, only the endpoint URL.
93
+
94
+
> **Note:** The proxy supports any bucket accessible with the configured credentials. You don't configure a specific bucket—just point any S3 request at the proxy and it forwards to the appropriate bucket.
95
+
96
+
---
97
+
98
+
## 🔍 How It Works
99
+
100
+
S3Proxy sits between your application and S3, transparently encrypting all data before it reaches storage.
101
+
102
+
### Request Flow
103
+
104
+
```
105
+
1. Client signs request with credentials (same credentials configured on proxy)
106
+
2. Proxy receives request and verifies SigV4 signature
107
+
3. Proxy encrypts the payload with AES-256-GCM
108
+
4. Proxy re-signs the request (encryption changes the body, invalidating original signature)
109
+
5. Proxy forwards to S3
110
+
6. S3 stores the encrypted data
111
+
```
112
+
113
+
### Why Does the Proxy Need My Credentials?
114
+
115
+
**Short answer:** Because encryption changes the request body, which invalidates the client's signature. The proxy must re-sign requests, and re-signing requires the secret key.
116
+
117
+
With S3's SigV4 authentication, clients sign requests using their secret key but only send the signature—never the key itself. When S3Proxy encrypts your data, it modifies:
118
+
- The request body (now ciphertext instead of plaintext)
119
+
- The `Content-Length` header
120
+
- The `Content-MD5` / `x-amz-content-sha256` headers
121
+
122
+
This breaks the original signature. To forward the request to S3, the proxy must create a new valid signature, which requires having the secret key.
123
+
124
+
**The proxy acts as a trusted intermediary**, not a transparent passthrough. You configure credentials once on the proxy, and all clients use those same credentials to authenticate.
125
+
126
+
```
127
+
┌──────────────┐ SigV4 signed ┌──────────────┐ Re-signed ┌──────────────┐
@@ -105,97 +145,237 @@ S3Proxy uses a **layered key architecture** for maximum security:
105
145
106
146
Your master key never touches S3. DEKs are wrapped and stored as object metadata. Even if someone accesses your bucket, they get nothing but ciphertext.
107
147
148
+
### Multipart Uploads
149
+
150
+
Large files are automatically handled via S3 multipart upload:
151
+
152
+
1. Each part is encrypted independently with its own nonce
153
+
2. Part metadata is tracked in Redis for distributed consistency
154
+
3. On completion, parts are assembled server-side by S3
155
+
4. The final object's metadata contains all part encryption info
156
+
157
+
This enables streaming uploads of arbitrary size without buffering entire files in memory.
158
+
108
159
---
109
160
110
161
## ⚙️ Configuration
111
162
112
-
All settings via environment variables (prefix: `S3PROXY_`):
163
+
All settings are configured via environment variables with the `S3PROXY_` prefix.
|`AWS_ACCESS_KEY_ID`| AWS credentials—used to verify client requests AND sign upstream requests |
171
+
|`AWS_SECRET_ACCESS_KEY`| AWS credentials—clients must use these same credentials |
172
+
173
+
> **Important:** Clients must authenticate using the same `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` configured on the proxy. The proxy verifies incoming signatures and re-signs requests before forwarding to S3. See [How It Works](#-how-it-works) for details.
|`S3PROXY_REDIS_URL`|*(empty)*| Redis URL for HA mode (omit for single-instance in-memory storage) |
197
+
|`S3PROXY_THROTTLING_REQUESTS_MAX`|`10`| Max concurrent requests (0 = unlimited) |
198
+
|`S3PROXY_MAX_UPLOAD_SIZE_MB`|`45`| Max single-request upload size in MB |
199
+
200
+
> **Note:** Redis is only required for multi-instance (HA) deployments where multipart upload state needs to be shared across replicas. For single-instance deployments, the proxy uses in-memory storage.
125
201
126
202
---
127
203
128
-
## 🐳 Deploy to Production
204
+
## ☸️ Production Deployment
129
205
130
-
### Docker Compose (with Redis)
206
+
### Kubernetes with Helm
207
+
208
+
The Helm chart is in `manifests/` and includes Redis HA with Sentinel for distributed state.
The proxy exposes health endpoints for Kubernetes probes:
311
+
-`GET /healthz` — Liveness probe
312
+
-`GET /readyz` — Readiness probe
313
+
314
+
### Security Considerations
315
+
316
+
-**TLS Termination**: The chart defaults to `noTls=true`, expecting TLS termination at the ingress/load balancer
317
+
-**Secrets**: Always use `secrets.existingSecrets` in production—never commit secrets to values files
318
+
-**Network Policy**: Consider restricting pod-to-pod traffic to only allow proxy → Redis
319
+
-**Encryption Key**: Back up your encryption key securely. Losing it means losing access to all encrypted data
320
+
321
+
### Resource Recommendations
322
+
323
+
| Workload | Memory | CPU | Concurrency | Notes |
324
+
|----------|--------|-----|-------------|-------|
325
+
| Standard | 512Mi | 100m | 10 | Default settings |
326
+
| Heavy | 1Gi+ | 500m | 20+ | Large files, high concurrency |
327
+
328
+
Memory scales with concurrent uploads. Use `performance.throttlingRequestsMax` to bound memory usage
165
329
166
330
---
167
331
168
-
## 🛡️ Security Model
332
+
## 🧪 Testing
169
333
170
-
| Threat | Mitigation |
171
-
|--------|------------|
172
-
| S3 bucket breach | All data encrypted with AES-256-GCM |
173
-
| Key extraction from S3 | DEKs wrapped with KEK, KEK never stored |
174
-
| Request tampering | Full AWS SigV4 signature verification |
175
-
| Replay attacks | Nonce uniqueness per object |
334
+
```bash
335
+
make test# Unit tests
336
+
make cluster-test # Full Kubernetes cluster test
337
+
```
176
338
177
339
---
178
340
179
-
## 🤝 Contributing
341
+
## ❓ FAQ
180
342
181
-
PRs welcome! Please include tests for new functionality.
343
+
**Why can't I use my own AWS credentials with the proxy?**
182
344
183
-
```bash
184
-
# Setup dev environment
185
-
uv sync
345
+
The proxy must re-sign requests after encryption (see [How It Works](#-how-it-works)). Re-signing requires the secret key, but S3's SigV4 protocol only sends signatures—never the secret key itself. So the proxy must already have the credentials configured. All clients share the same credentials configured on the proxy.
346
+
347
+
**Can I use different credentials for different clients?**
348
+
349
+
Not currently. The proxy supports one credential pair. If you need per-client credentials, you would deploy multiple proxy instances or implement a credential lookup mechanism.
350
+
351
+
**Can I use this with existing unencrypted data?**
352
+
353
+
Yes. S3Proxy only encrypts data written through it. Existing objects remain readable—S3Proxy detects unencrypted objects and returns them as-is. To migrate, simply copy objects through S3Proxy:
0 commit comments