Repository navigation
GCS needs someone with a real bucket to run its test #214
Description
The GCS source ships with construction tests only. Nobody has pointed it at a real bucket.
That gap is not theoretical. S3 passed the same construction tests and still had a bug that
broke every bucket anyone had clicked a "new folder" button in: object_store normalises the
trailing slash off a directory marker, so the filter never fired, the GET asked for a key that
does not exist, and the 404 took the whole sync down. Only a live endpoint found it. GCS shares
that fetch path and adds its own builder configuration on top.
fsouza/fake-gcs-server cannot close this. It implements the JSON API
(/storage/v1/b/...) while object_store lists over the XML API
(/bucket?list-type=2), and returns 404 there. The usual emulator does not exercise the path
we use.
What would help: run the existing test against a real bucket and report what happens.
UTOPIA_GCS_TEST_ENDPOINT=https://storage.googleapis.com \
UTOPIA_GCS_TEST_BUCKET=your-bucket \
UTOPIA_GCS_TEST_KEY="$(cat service-account.json)" \
cargo test -p utopia-server object_storageA bucket with a few files under a docs/ prefix is enough, and read-only credentials are all
it needs. The test only lists and reads.
Worth checking beyond a pass: that a folder created through the Cloud Console is skipped rather
than fetched, and that last_modified comes back — it is the only source of doc_time here.
See crates/utopia-server/src/object_storage.rs.
Activity
FabianIMV commentedon Oct 5, 2026
Ran
it_reads_from_gcsagainst a real bucket: us-central1, flat namespace, uniform access, service account with onlyStorage Object Vieweron the bucket. It passes.Bucket contents under
docs/:a.txt("hello"),b.md,data.csv,notes.txt,readme.md, plus two folders created with the Cloud Console (docs/anddocs/carpeta-consola/).Results:
fetchdrops them by size before any GET, so the S3 failure mode does not reproduce on GCS.last_modifiedis populated on every object.external_keyisgs://<bucket>/docs/a.txtas expected.NoSuchBucketonGET /<bucket>?list-type=2&prefix=docs%2F, so the green run really reached GCS.Raw listing before the size filter
One snag: the test requires
docs/a.txtto exist (.expect("a.txt")), which neither the issue nor the doc comment mentions. My first run panicked with justa.txtuntil I uploaded it.I can open a small PR against
devthat documents the expected fixture in the doc comment, checks the content ofa.txtlike the S3 test does, and asserts the raw listing contains a zero-byte placeholder, so a green run can't come from a bucket without one.WaylandYang commentedon Oct 6, 2026
Thank you, this is exactly the run the issue asked for. The control against a missing bucket is what makes the green one count, and it is good to know the S3 failure does not carry over to GCS.
Yes, please open the PR against
dev. All three changes are welcome:docs/a.txtcontaininghello, plus one folder made in the Cloud Console.a.txt, as the S3 test does.fetchreturns none.One addition: the comment above the test still says GCS is covered by construction tests only. Please replace that with what you ran (date, region, uniform access,
Storage Object Viewer).I will close this issue when the PR lands.
Thanks again for taking the time to set up a real bucket and report it this carefully.
WaylandYang commentedon Oct 6, 2026
Closed by #1091 (4b0d39b on dev): the GCS path has been run against a real bucket, and the test now names its fixture and fails when the bucket has no Console folder to filter. Thanks, @FabianIMV.