Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
bfcfd9d
Bump com.azure:azure-storage-blob from 12.29.0 to 12.30.0
dependabot[bot] May 1, 2025
73e7d14
Bump aws-sdk.version from 1.12.782 to 1.12.783
dependabot[bot] May 1, 2025
9aa8ea5
Bump org.eclipse.jetty:jetty-servlet from 11.0.24 to 11.0.25
dependabot[bot] May 1, 2025
3d67b56
Bump com.puppycrawl.tools:checkstyle from 10.22.0 to 10.23.1
dependabot[bot] May 1, 2025
bf2d506
Catch UnsupportedOperationException for getPosixFilePermissions and d…
lukaszherman May 28, 2025
144f56a
Handle UnsupportedOperationException for getPosixFilePermissions and …
lukaszherman May 29, 2025
276465f
Bump com.google.guava:guava from 33.4.0-jre to 33.4.8-jre
dependabot[bot] Jun 1, 2025
e5c20b6
Bump org.apache.maven.plugins:maven-assembly-plugin from 3.5.0 to 3.7.1
dependabot[bot] Jun 1, 2025
7b9e860
Bump org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1
dependabot[bot] Jun 1, 2025
17e490e
Fix Python yield error in s3-tests
gaul Jun 11, 2025
9ddc581
Add .dockerignore
polarctos Jun 11, 2025
1003265
Fix EncryptedBlobStore support for s3 compatible backends without eTa…
polarctos Jun 11, 2025
5f7e597
Run on Eclipse Temurin OpenJDK 21 instead of 17
polarctos Jun 11, 2025
9b0f4cc
Remove linux/arm/v7 container
polarctos Jun 12, 2025
ab68547
Bump aws-sdk.version from 1.12.783 to 1.12.787
dependabot[bot] Jul 1, 2025
a8ac96c
Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml
dependabot[bot] Jul 1, 2025
027cd5f
Bump com.puppycrawl.tools:checkstyle from 10.23.1 to 10.26.1
dependabot[bot] Jul 1, 2025
5ff5896
Force Jetty header cache to be case-sensitive, to avoid signature mis…
Jul 16, 2025
24e869c
Update s3-tests
gaul Aug 4, 2025
45c9402
Bump com.google.jimfs:jimfs from 1.3.0 to 1.3.1
dependabot[bot] Aug 4, 2025
5b5d151
Bump docker/metadata-action from 5.7.0 to 5.8.0
dependabot[bot] Aug 4, 2025
8540e4f
Bump com.github.spotbugs:spotbugs-annotations from 4.9.2 to 4.9.3
dependabot[bot] Aug 4, 2025
53c1d37
Bump ch.qos.logback:logback-classic from 1.5.17 to 1.5.18
dependabot[bot] Aug 4, 2025
aadba1f
Upgrade Surefire to 3.5.2
gaul Aug 4, 2025
cc4cbfe
Munge / key to container name
gaul Aug 21, 2025
944d19d
Add latency blobstore middleware
Phantom-101 Aug 21, 2025
f8ebf43
Prepare for Central Portal releasing
gaul Aug 21, 2025
c13a49d
S3Proxy 2.7.0 release
gaul Aug 21, 2025
2acdb96
Bump to 2.7.1-SNAPSHOT
gaul Aug 21, 2025
664a94d
Update README.md to include latency middleware
Phantom-101 Aug 23, 2025
f8f4359
Include --fail with curl to propagate exit code
gaul Aug 27, 2025
8d96370
Update s3-tests to run conditional get tests
gaul Aug 27, 2025
69297fe
Add NoCacheBlobStore
polarctos Aug 26, 2025
9c587c7
Fix EncryptedBlobStore error handling
polarctos Aug 28, 2025
6e5fbf5
Add test for encrypted conditional get
polarctos Aug 28, 2025
02fbdc1
Parse but do not enforce CRC32
gaul Aug 27, 2025
d069ac1
Ignore all checksum stanzas
gaul Aug 29, 2025
d503c07
Disable retries in AwsSdk2Test
gaul Sep 3, 2025
eb5c59a
Enforce payload checksums
gaul Sep 3, 2025
7abc2e7
Bump njord.version from 0.7.1 to 0.7.5
dependabot[bot] Sep 8, 2025
bd40f73
Bump software.amazon.awssdk:s3 from 2.30.33 to 2.33.4
dependabot[bot] Sep 8, 2025
7087f8a
Bump com.azure:azure-identity from 1.15.3 to 1.17.0
dependabot[bot] Sep 8, 2025
de2b465
Bump com.fasterxml.jackson.dataformat:jackson-dataformat-xml
dependabot[bot] Sep 8, 2025
9ff42d0
Bump org.eclipse.jetty:jetty-servlet from 11.0.25 to 11.0.26
dependabot[bot] Sep 8, 2025
368f7f0
Bump aws-sdk.version from 1.12.787 to 1.12.791
dependabot[bot] Sep 8, 2025
d28d493
Increase number of Dependabot PRs
gaul Sep 8, 2025
ea3a772
Bump actions/download-artifact from 4 to 5
dependabot[bot] Sep 8, 2025
6472b9a
Bump actions/setup-java from 4 to 5
dependabot[bot] Sep 8, 2025
382421c
Bump actions/setup-python from 5 to 6
dependabot[bot] Sep 8, 2025
f7eaae3
Bump org.assertj:assertj-core from 3.27.3 to 3.27.4
dependabot[bot] Sep 8, 2025
7dd9b95
Bump com.github.spotbugs:spotbugs-maven-plugin from 4.9.3.0 to 4.9.4.2
dependabot[bot] Sep 8, 2025
2edaddd
Bump modernizer.version from 3.1.0 to 3.2.0
dependabot[bot] Sep 8, 2025
011f9e1
Bump com.github.spotbugs:spotbugs-annotations from 4.9.3 to 4.9.4
dependabot[bot] Sep 8, 2025
1f5d733
Upgrade to Azurite 3.35
gaul Sep 8, 2025
1d4206b
Bump actions/checkout from 4 to 5
dependabot[bot] Sep 8, 2025
c3351bf
Bump com.azure:azure-storage-blob from 12.30.0 to 12.31.2
dependabot[bot] Sep 8, 2025
694f7bd
Ignore x-amz-checksum-mode
gaul Sep 9, 2025
b125abe
Remove apache-snapshots repository
gaul Sep 9, 2025
e129952
Address a few stray warnings
gaul Sep 9, 2025
226b5aa
Add disabled test for V4 signature payload signing
gaul Sep 10, 2025
c6773dd
Reduce AWS SDK v2 to test scope
gaul Sep 10, 2025
88245b9
Replace fileupload2 dependency with Jetty
gaul Sep 10, 2025
9f66ef5
Correct typos
gaul Sep 10, 2025
818a510
Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.2.1 to 3.6.1
dependabot[bot] Oct 1, 2025
e1f9a94
Bump org.apache.maven.plugins:maven-compiler-plugin
dependabot[bot] Oct 1, 2025
8c69785
Bump org.apache.maven.plugins:maven-jar-plugin from 3.3.0 to 3.4.2
dependabot[bot] Oct 1, 2025
966815e
Bump com.azure:azure-identity from 1.17.0 to 1.18.0
dependabot[bot] Oct 1, 2025
33ed89e
Bump org.apache.maven.plugins:maven-gpg-plugin from 3.0.1 to 3.2.8
dependabot[bot] Oct 1, 2025
671855b
Bump org.apache.maven.plugins:maven-clean-plugin from 3.2.0 to 3.5.0
dependabot[bot] Oct 1, 2025
efb6125
Bump org.apache.maven.plugins:maven-shade-plugin from 3.4.1 to 3.6.1
dependabot[bot] Oct 1, 2025
7e8c5c9
Bump org.apache.maven.plugins:maven-deploy-plugin from 3.1.0 to 3.1.4
dependabot[bot] Oct 1, 2025
cbbf5ab
Bump org.apache.maven.plugins:maven-install-plugin from 3.1.1 to 3.1.4
dependabot[bot] Oct 1, 2025
7c90b85
Bump com.github.spotbugs:spotbugs-annotations from 4.9.4 to 4.9.6
dependabot[bot] Oct 1, 2025
170426e
Bump com.github.spotbugs:spotbugs-maven-plugin from 4.9.4.2 to 4.9.6.0
dependabot[bot] Oct 1, 2025
ddd4ef3
Bump org.apache.maven.plugins:maven-checkstyle-plugin
dependabot[bot] Oct 1, 2025
f0ab938
Bump org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.12.0
dependabot[bot] Oct 1, 2025
4f664a8
Bump org.assertj:assertj-core from 3.27.4 to 3.27.6
dependabot[bot] Oct 1, 2025
de03a3b
Increase number of Dependabot PRs
gaul Oct 1, 2025
5d355bc
Bump ch.qos.logback:logback-classic from 1.5.18 to 1.5.19
dependabot[bot] Oct 1, 2025
81258b2
Bump com.google.guava:guava from 33.4.8-jre to 33.5.0-jre
dependabot[bot] Oct 1, 2025
eace4e4
Bump com.azure:azure-storage-blob from 12.31.2 to 12.31.3
dependabot[bot] Oct 1, 2025
9e91e68
Bump software.amazon.awssdk:s3 from 2.33.4 to 2.34.7
dependabot[bot] Oct 1, 2025
b07c27b
Limit CI to read-only access
gaul Oct 7, 2025
3029cbc
Do not require identity and credential for nio2
gaul Oct 7, 2025
0bf4cb4
Allow Containerize write permission
gaul Oct 7, 2025
5eec0f2
Allow top-level write permission for Docker push
gaul Oct 7, 2025
07d420e
Add If-Match and If-None-Match support for PutObject operations
klaudworks Oct 7, 2025
aecf3f8
replace if-match / if-none-match Java tests with existing Python tests
klaudworks Oct 7, 2025
548036d
fix if-match logic for put operations
klaudworks Oct 7, 2025
b697c82
document non-atomic conditional put operations
klaudworks Oct 7, 2025
4bad761
fix checkstyle violations
klaudworks Oct 7, 2025
97494ee
fix azureblob-sdk handling for non-existant objects
klaudworks Oct 7, 2025
ada7482
Remove incorrect permissions for now
gaul Oct 7, 2025
0a6f683
Update s3-tests to include condition put tests
gaul Oct 7, 2025
50c368e
Add auto-service annotation processor to compiler
gaul Oct 8, 2025
bd7235f
Bump org.apache.maven.plugins:maven-enforcer-plugin from 3.6.1 to 3.6.2
dependabot[bot] Oct 9, 2025
3610945
Bump software.amazon.awssdk:s3 from 2.34.7 to 2.35.3
dependabot[bot] Oct 9, 2025
62ee60f
Bump aws-sdk.version from 1.12.791 to 1.12.792
dependabot[bot] Oct 9, 2025
8456d19
S3Proxy 2.8.0 release
gaul Oct 9, 2025
679912b
Merge tag 's3proxy-2.8.0' into merge-upstream-2.8.0
jupblb Oct 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Exclude everything from context that is not used by COPY steps in the Dockerfile
*
!/target/s3proxy
!/src/main/resources/run-docker-container.sh
3 changes: 2 additions & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ updates:
directory: "/"
schedule:
interval: "monthly"
- package-ecosystem: "maven" # See documentation for possible values
- package-ecosystem: "maven"
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
open-pull-requests-limit: 20
25 changes: 13 additions & 12 deletions .github/workflows/ci-main.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
name: Main CI

on:
push:
branches:
Expand All @@ -24,13 +25,13 @@ jobs:
container_revision: ${{ fromJSON(steps.docker_action_meta.outputs.json).labels['org.opencontainers.image.revision'] }}
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v5
with:
submodules: false
persist-credentials: false
- name: Docker meta
id: docker_action_meta
uses: docker/metadata-action@v5.7.0
uses: docker/metadata-action@v5.8.0
with:
images: |
name=ghcr.io/${{ github.repository }}/container
Expand All @@ -50,16 +51,16 @@ jobs:
runs-on: ubuntu-24.04-arm
needs: [meta]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v5
with:
submodules: "recursive"

- uses: actions/setup-java@v4
- uses: actions/setup-java@v5
with:
distribution: "temurin"
java-version: "11"
cache: "maven"
- uses: actions/setup-python@v5
- uses: actions/setup-python@v6
with:
python-version: "3.11"
cache: "pip"
Expand Down Expand Up @@ -87,18 +88,18 @@ jobs:
mvn test -Ds3proxy.test.conf=s3proxy-filesystem-nio2.conf -Dtest=AwsSdkTest

- name: Install Azurite
run: npx --yes --loglevel info azurite@3.34 --version
run: npx --yes --loglevel info azurite@3.35 --version
- name: Start Azurite
shell: bash
run: npx --yes azurite@3.34 azurite-blob &
run: npx --yes azurite@3.35 azurite-blob &
- name: Maven Test with Azurite
run: |
# TODO: run other test classes
mvn test -Ds3proxy.test.conf=s3proxy-azurite.conf -Dtest=AwsSdkTest

- name: Install Minio
run: |
curl -o minio https://dl.min.io/server/minio/release/linux-$(uname -m | sed -e s/aarch64/arm64/ -e s/x86_64/amd64/)/minio
curl --fail --silent --output minio https://dl.min.io/server/minio/release/linux-$(uname -m | sed -e s/aarch64/arm64/ -e s/x86_64/amd64/)/minio
chmod +x minio
- name: Start Minio
run: |
Expand Down Expand Up @@ -142,12 +143,12 @@ jobs:
runs-on: ubuntu-24.04-arm
needs: [runTests, meta]
steps:
- uses: actions/checkout@v4
- uses: actions/download-artifact@v4
- uses: actions/checkout@v5
- uses: actions/download-artifact@v5
with:
name: s3proxy
path: target
- uses: actions/download-artifact@v4
- uses: actions/download-artifact@v5
with:
name: pom
path: .
Expand Down Expand Up @@ -175,7 +176,7 @@ jobs:
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64,linux/arm/v7
platforms: linux/amd64,linux/arm64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ needs.meta.outputs.container_tags }}
labels: ${{ needs.meta.outputs.container_labels }}
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM docker.io/library/eclipse-temurin:17-jre
FROM docker.io/library/eclipse-temurin:21-jre
LABEL maintainer="Andrew Gaul <andrew@gaul.org>"

WORKDIR /opt/s3proxy
Expand All @@ -12,7 +12,7 @@
src/main/resources/run-docker-container.sh \
/opt/s3proxy/

ENV \

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "JCLOUDS_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_ENCRYPTED_BLOBSTORE_PASSWORD") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_CORS_ALLOW_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_KEYSTORE_PASSWORD") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_CORS_ALLOW_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_KEYSTORE_PASSWORD") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "JCLOUDS_CREDENTIAL") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/

Check warning on line 15 in Dockerfile

View workflow job for this annotation

GitHub Actions / Containerize

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "S3PROXY_ENCRYPTED_BLOBSTORE_PASSWORD") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
LOG_LEVEL="info" \
S3PROXY_AUTHORIZATION="aws-v2-or-v4" \
S3PROXY_ENDPOINT="http://0.0.0.0:80" \
Expand Down
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,13 @@ S3Proxy can modify its behavior based on middlewares:
* [bucket locator](https://github.com/gaul/s3proxy/wiki/Middleware-bucket-locator)
* [eventual consistency modeling](https://github.com/gaul/s3proxy/wiki/Middleware---eventual-consistency)
* [large object mocking](https://github.com/gaul/s3proxy/wiki/Middleware-large-object-mocking)
* [latency](https://github.com/gaul/s3proxy/wiki/Middleware-latency)
* [read-only](https://github.com/gaul/s3proxy/wiki/Middleware-read-only)
* [regex rename blobs](https://github.com/gaul/s3proxy/wiki/Middleware-regex)
* [sharded backend containers](https://github.com/gaul/s3proxy/wiki/Middleware-sharded-backend)
* [storage class override](https://github.com/gaul/s3proxy/wiki/Middleware-storage-class-override)
* [user metadata replacer](https://github.com/gaul/s3proxy/wiki/Middleware-user-metadata-replacer)
* [no cache override](https://github.com/gaul/s3proxy/wiki/Middleware-no-cache)

## SSL Support

Expand All @@ -138,6 +140,7 @@ S3Proxy has broad compatibility with the S3 API, however, it does not support:

S3Proxy emulates the following operations:

* conditional PUT object when using If-Match or If-None-Match
* copy multi-part objects, see [#76](https://github.com/gaul/s3proxy/issues/76)

S3Proxy has basic CORS preflight and actual request/response handling. It can be configured within the properties
Expand Down
Loading