Skip to content

HADOOP-19204. VectorIO regression: empty ranges are now rejected #6887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

steveloughran
Copy link
Contributor

@steveloughran steveloughran commented Jun 12, 2024

  • restore out outcome: no-op
  • test this

How was this patch tested?

new test case in AbstractContractVectoredReadTest; tested on

  • local
  • s3 london
  • abfs cardiff

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran steveloughran force-pushed the s3/HADOOP-19204-vector-io-empty-ranges branch from c65b411 to 3ffa8fb Compare June 12, 2024 19:22
- restore old outcome: no-op
- test this
- update spec

Contributed by Steve Loughran

Change-Id: I8a2b8c23f573692bd147de82e98b57d2712b0b16
@steveloughran steveloughran force-pushed the s3/HADOOP-19204-vector-io-empty-ranges branch from 3ffa8fb to ba99be5 Compare June 12, 2024 19:25
Change-Id: Ia4003caa505d30fb8f21bc4081ddb1246831cc49
@hadoop-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 29s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 💚 mvninstall 46m 9s trunk passed
+1 💚 compile 17m 17s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 compile 15m 59s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 checkstyle 1m 18s trunk passed
+1 💚 mvnsite 1m 43s trunk passed
+1 💚 javadoc 1m 16s trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 54s trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 2m 35s trunk passed
+1 💚 shadedclient 35m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 0m 54s the patch passed
+1 💚 compile 16m 50s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javac 16m 50s the patch passed
+1 💚 compile 16m 5s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 javac 16m 5s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 14s the patch passed
+1 💚 mvnsite 1m 40s the patch passed
+1 💚 javadoc 1m 10s the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚 javadoc 0m 55s the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚 spotbugs 2m 43s the patch passed
+1 💚 shadedclient 35m 30s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 30s hadoop-common in the patch passed.
+1 💚 asflicense 1m 2s The patch does not generate ASF License warnings.
221m 19s
Subsystem Report/Notes
Docker ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6887/4/artifact/out/Dockerfile
GITHUB PR #6887
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname Linux 9688dec88bff 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 1f83e7e
Default Java Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6887/4/testReport/
Max. process+thread count 1429 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6887/4/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@apache apache deleted a comment from hadoop-yetus Jun 14, 2024
@apache apache deleted a comment from hadoop-yetus Jun 14, 2024
@apache apache deleted a comment from hadoop-yetus Jun 14, 2024
@mukund-thakur
Copy link
Contributor

Could you please point to the test in Parquet/Spark which is doing this?
Wondering why we have such a test.

@steveloughran
Copy link
Contributor Author

Could you please point to the test in Parquet/Spark which is doing this?
Wondering why we have such a test.

internal sql testing where the query selected none of any of the rows after the push down filtering scanned them. This is a legitimate codepath, just one which didn't get picked up in parquet itself. I'll need to look at those tests there to see if there more we should parameterize on vectorisation

Copy link
Contributor

@mukund-thakur mukund-thakur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@steveloughran steveloughran merged commit 56c8aa5 into apache:trunk Jun 19, 2024
steveloughran added a commit that referenced this pull request Jun 19, 2024
- restore old outcome: no-op
- test this
- update spec

This is a critical fix for vector IO and MUST be cherrypicked to all branches with
that feature

Contributed by Steve Loughran
KeeProMise pushed a commit to KeeProMise/hadoop that referenced this pull request Sep 9, 2024
…che#6887)



- restore old outcome: no-op
- test this
- update spec

This is a critical fix for vector IO and MUST be cherrypicked to all branches with
that feature

Contributed by Steve Loughran
Hexiaoqiao pushed a commit to Hexiaoqiao/hadoop that referenced this pull request Sep 12, 2024
…che#6887)



- restore old outcome: no-op
- test this
- update spec

This is a critical fix for vector IO and MUST be cherrypicked to all branches with
that feature

Contributed by Steve Loughran
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants