Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove prefix from inodes #1303

Merged
merged 5 commits into from
Mar 6, 2025
Merged

Remove prefix from inodes #1303

merged 5 commits into from
Mar 6, 2025

Conversation

passaro
Copy link
Contributor

@passaro passaro commented Mar 5, 2025

When Mountpoint is configured with the --prefix flag, all S3 requests contain the specified prefix as part of the key. Currently, the prefix is duplicated in each Inode entry in the full_key field. This change remove the unnecessary duplication by only storing the partial key and reconstructing the full_key by adding the prefix before performing any S3 request.

Does this change impact existing behavior?

No.

Does this change need a changelog entry? Does it require a version change?

mountpoint-s3 changelog entry. No version change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 10:19 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 12:14 — with GitHub Actions Inactive
@passaro passaro marked this pull request as ready for review March 5, 2025 12:14
@vladem vladem added the performance PRs to run benchmarks on label Mar 5, 2025
@vladem vladem temporarily deployed to PR benchmarks March 5, 2025 22:30 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to PR benchmarks March 5, 2025 22:30 — with GitHub Actions Inactive
@vladem vladem temporarily deployed to PR benchmarks March 5, 2025 22:30 — with GitHub Actions Inactive
vladem
vladem previously approved these changes Mar 5, 2025
View reviewed changes
Copy link
Contributor

@vladem vladem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. So we're storing the prefix in the Superblock, the remaining part in the Inode and constructing the full key shortly before making a request to S3.

Some small suggestions from me, but fine with shipping after benchmarks finish (just in case).

mountpoint-s3/src/superblock/inode.rs Outdated
@@ -34,7 +35,8 @@ struct InodeInner {
parent: InodeNo,
name: String,
// TODO deduplicate keys by string interning or something -- many keys will have common prefixes
full_key: String,
/// Object key not including the prefix.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Object key not including the prefix.
/// Object key not including the prefix used to mount the bucket. May contain '/'.

mountpoint-s3/src/superblock.rs Outdated
@@ -988,7 +995,7 @@ impl SuperblockInner {

let next_ino = self.next_ino.fetch_add(1, Ordering::SeqCst);

let mut full_key = parent.full_key().to_owned();
let mut full_key = parent.key().to_owned();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really a full_key now

@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR benchmarks March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR benchmarks March 5, 2025 23:06 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:07 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR benchmarks March 5, 2025 23:07 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:07 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR benchmarks March 5, 2025 23:07 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 5, 2025 23:07 — with GitHub Actions Inactive
@passaro passaro removed the performance PRs to run benchmarks on label Mar 6, 2025
passaro added 5 commits March 6, 2025 14:38
@passaro
Remove prefix from inodes
0ca4526 
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
@passaro
minor: Remove prefix from S3Filesystem
b8f7f5d 
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
@passaro
minor: Harmonize bucket/key handling in atomic and incremental uploads
f035672 
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
@passaro
Add changelog entry
807d541 
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
@passaro
Address PR feedback
8b5c29c 
Signed-off-by: Alessandro Passaro <alexpax@amazon.co.uk>
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro temporarily deployed to PR integration tests March 6, 2025 14:39 — with GitHub Actions Inactive
@passaro passaro enabled auto-merge March 6, 2025 14:40
@passaro passaro added this pull request to the merge queue Mar 6, 2025
Merged via the queue into awslabs:main with commit d4dc756 Mar 6, 2025
26 checks passed
@passaro passaro deleted the mem/prefix branch March 6, 2025 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vladem vladem vladem approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@passaro @vladem