HADOOP-19793. S3A: use long for file size in S3A content providers, data blocks#8225
Open
ajfabbri wants to merge 3 commits intoapache:trunkfrom
Open
HADOOP-19793. S3A: use long for file size in S3A content providers, data blocks#8225ajfabbri wants to merge 3 commits intoapache:trunkfrom
ajfabbri wants to merge 3 commits intoapache:trunkfrom
Conversation
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
bf2c793 to
e179121
Compare
Contributor
Author
|
Majority of CI failures due to HADOOP-19790 |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
2 times, most recently
from
d0403b1 to
1e1c09c
Compare
ajfabbri
changed the title
|
💔 -1 overall
This message was automatically generated. |
steveloughran
changed the title
steveloughran
approved these changes
Feb 19, 2026
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1
all looks good, and I reviewed that SingleFilePendingCommit file size too.
Regarding test failures
#2 may mean you aren't set up to create a session for the target user/account. I can help there
#3 things have been playing up with the MR cluster tests since the move to JUnit 5...getting anything working was a challenge enough.
Contributor
|
fix the spotbugs by addressing overrideable methods in verify, or make SinglePendingCommit final. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
587e5de to
eed4c14
Compare
Contributor
|
Contributor
Author
I already fixed it but CI has been stuck for over 24 hours! https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8225/8/ |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
eed4c14 to
f94a921
Compare
Contributor
Author
|
Force-push: rebase on latest trunk. |
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
f94a921 to
01e1456
Compare
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
aah, I don't see what spotbugs is complaining about here...I suspect it's the duplicate Class declaration in the same match
| <!-- Despite adding `final` as suggested, spotbugs kept complaining. --> | ||
| <Match> | ||
| <Class name="org.apache.hadoop.fs.s3a.commit.files.PendingSet"/> | ||
| <Class name="org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit"/> |
Contributor
There was a problem hiding this comment.
maybe you have to only do one Class per match?
Contributor
Author
There was a problem hiding this comment.
Thanks for the idea. I tried it. 🤷♂️
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
2 times, most recently
from
7945e63 to
c7640dc
Compare
|
💔 -1 overall
This message was automatically generated. |
Contributor
Author
|
I am at a loss why CI (spotbugs) is failing here. It passes for me locally. That is, |
Adding `final` didn't work so I had to add an exclude. Experimental Warnings: MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT in o.a.h.fs.s3a.commit.files.PendingSet.readObject(ObjectInputStream) and In method o.a.h.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) Called method o.a.h.fs.s3a.commit.files.SinglePendingCommit.validate() s3a: spotbugs: make entire class final Despite changing validate() to final, still get this warning: Unknown bug pattern MC_OVERRIDABLE_METHOD_CALL_IN_READ_OBJECT in org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) At SinglePendingCommit.java:org.apache.hadoop.fs.s3a.commit.files.SinglePendingCommit.readObject(ObjectInputStream) At SinglePendingCommit.java:[line 201] try making the whole class final then. hadoop-aws: add excludes for spotbugs being buggy
Supresses an existing warning that my edit re-triggered. Disabled this check for all of hadoop-aws--it is of questionable value.
ajfabbri
force-pushed
the
fabbri/hadoop-19793
branch
from
c7640dc to
f32ae58
Compare
Contributor
Author
|
Force-push: rebase on latest trunk |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
approved these changes
Feb 27, 2026
Contributor
steveloughran
left a comment
steveloughran
left a comment
There was a problem hiding this comment.
+1
hadoop-tools/hadoop-aws generated 0 new + 0 unchanged - 2 fixed = 0 total (was 2)
you've fixed the spotbugs...those two were "extant" in the existing code. They probably crept in from a spotbugs update.
merge at your leisure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
From HADOOP-19793:
How was this patch tested?
Tested that large scale test with localstack S3.
Ran all integration tests locally against S3 in us-west-2. All passed except:
ITestS3ACannedACLs>AbstractS3ATestBase.setup:111->AbstractFSContractTestBase.setup:197->AbstractFSContractTestBase.mkdirs:355 » ... S3Exception:The bucket does not allow ACLsITestS3ATemporaryCredentials.testSessionTokenPropagation:202getFileStatus on s3a://fabbri-s3a/job-00-fork-0004/test/testSTS/c20306d0-3eca-4264-b210-3d7beb8c80c7: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden (Service: S3, Status Code: 403, Request ID: 6GV4SMMX9EXVDB0J, Extended Request ID: nB4bQ+CHn+Y2BiR4lLSgLtHyWVd05eerIXrsNDhV9Z0o9I/KZ+N8VD3khlSsiJvh0HkuCjDMtB4=):nullThe first failure seems OK given I haven't enabled bucket ACLs.
I'm not sure what the problem is with 2 & 3.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
Nopewhere is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html