Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgraded ElasticSearch to get rid of CVEs. #13747

Merged
merged 6 commits into from
Jan 20, 2022
Merged

Upgraded ElasticSearch to get rid of CVEs. #13747

merged 6 commits into from
Jan 20, 2022

Conversation

dlg99
Copy link
Contributor

@dlg99 dlg99 commented Jan 13, 2022

CVEs are:
CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

Motivation

mvn clean install verify -Powasp-dependency-check -DskipTests found various CVEs

Modifications

Upgraded ElasticSearch to get rid of CVEs.
Had to update maven compiler plugin config, otherwise it was NPE-ing instead of showing actual errors.

Verifying this change

  • Make sure that the change passes the CI checks.

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): YES
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jan 13, 2022
@dlg99
Copy link
Contributor Author

dlg99 commented Jan 14, 2022

/pulsarbot run-failure-checks

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had to update maven compiler plugin config, otherwise it was NPE-ing instead of showing actual errors.

Please revert these changes from this PR, if possible.

There's #13126 about the maven NPE issue. In the past there has been a need to temporarily set forceJavacCompilerUse to find out the compiler error, but it has been possible to remove the option after fixing the compilation issue. Is that also the case here?

At least it seems that setting the maven compiler plugin version isn't necessary since the parent pom sets it.
maven-compiler-plugin version is set via

pulsar/pom.xml

Lines 26 to 30 in e5d828a

<parent>
<groupId>org.apache</groupId>
<artifactId>apache</artifactId>
<version>23</version>
</parent>
and is set to 3.8.1 in https://github.com/apache/maven-apache-parent/blob/99592e229c20f079c1998875e5c24556d217b491/pom.xml#L138-L142

If there's no way around it, then add a comment why forceJavacCompilerUse has been added, such as.

         <!-- workaround https://issues.apache.org/jira/browse/MCOMPILER-346 -->
         <forceJavacCompilerUse>true</forceJavacCompilerUse>

@dlg99
Copy link
Contributor Author

dlg99 commented Jan 14, 2022

@lhotari I removed forceJavacCompilerUse and everything builds/pushed the change.
Please help me understand what's the downside of just keeping it there/adding in another PR?
I presume everyone who updates dependencies now and then gets into that NPE in maven, and have to add the forceJavacCompilerUse to see the errors, and remove it later.
So does it hurt build times or anything else? Why don't we make developers' life easier and keep it until we move to the maven/jvm plugin with the NPE fixed (not soon, probably, as it is not fixed yet)?

pulsar-io/elastic-search/pom.xml Outdated Show resolved Hide resolved
@lhotari
Copy link
Member

lhotari commented Jan 17, 2022

@lhotari I removed forceJavacCompilerUse and everything builds/pushed the change. Please help me understand what's the downside of just keeping it there/adding in another PR? I presume everyone who updates dependencies now and then gets into that NPE in maven, and have to add the forceJavacCompilerUse to see the errors, and remove it later. So does it hurt build times or anything else? Why don't we make developers' life easier and keep it until we move to the maven/jvm plugin with the NPE fixed (not soon, probably, as it is not fixed yet)?

I believe that the downside is reduced performance. In addition, the problem might happen in any module and therefore it doesn't make sense to change the setting for just a few modules if it were to be enabled globally.
There might also be some changes in how maven compiler issues are displayed when forceJavacCompilerUse is used.

The way to make developers' life easier is to raise awareness of this issue. I added a simpler workaround to #13126, it's by passing -Dmaven.compiler.forceJavacCompilerUse=true to the mvn command line. To make it easier to find this information we could add the information to README or the contributor guide.
I just added this solution to stack overflow.

@lhotari
Copy link
Member

lhotari commented Jan 17, 2022

btw. Regarding the NPEs with maven-compiler-plugin, version 3.9.0 was released recently with fixes to some NPE issues. https://lists.apache.org/thread/8bm3powmfy46z25k9jgrbkf1kxf5j6yk . I'll create a PR to upgrade to 3.9.0 . PR is #13789.

dlg99 added 4 commits January 18, 2022 08:49
Had to update maven compiler plugin config, otherwise it was NPE-ing
instead of showing actual errors.

CVEs are:
CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147
…c images (and elastic.co no longer releases OSS images)
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work @dlg99 ! LGTM

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

great

@lhotari lhotari requested review from merlimat and Jason918 January 19, 2022 16:46
@lhotari lhotari merged commit 1af8d3f into apache:master Jan 20, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jan 20, 2022
* Upgraded ElasticSearch to get rid of CVEs.

CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

* Elastic search client version >= 7.11 no longer works with OSS Elastic images (and elastic.co no longer releases OSS images)

* Fixed tests for Elasticsearch

* pom cleanup

(cherry picked from commit 1af8d3f)
lhotari added a commit to lhotari/pulsar that referenced this pull request Jan 20, 2022
dlg99 added a commit to dlg99/pulsar that referenced this pull request Jan 20, 2022
lhotari pushed a commit that referenced this pull request Jan 20, 2022
dlg99 added a commit to dlg99/pulsar that referenced this pull request Jan 20, 2022
* Upgraded ElasticSearch to get rid of CVEs.

CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

* Elastic search client version >= 7.11 no longer works with OSS Elastic images (and elastic.co no longer releases OSS images)

* Fixed tests for Elasticsearch

* pom cleanup
lhotari pushed a commit that referenced this pull request Jan 21, 2022
…nSearch one) (#13867)

* Upgraded ElasticSearch to get rid of CVEs. (#13747)

* Upgraded ElasticSearch to get rid of CVEs.

CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

* Elastic search client version >= 7.11 no longer works with OSS Elastic images (and elastic.co no longer releases OSS images)

* Fixed tests for Elasticsearch

* pom cleanup

* Switched to OpenSearch client for Elastic (Apache 2 licensed)
nicoloboschi added a commit to datastax/pulsar that referenced this pull request Jan 21, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jan 21, 2022
…nSearch one) (apache#13867)

* Upgraded ElasticSearch to get rid of CVEs. (apache#13747)

* Upgraded ElasticSearch to get rid of CVEs.

CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

* Elastic search client version >= 7.11 no longer works with OSS Elastic images (and elastic.co no longer releases OSS images)

* Fixed tests for Elasticsearch

* pom cleanup

* Switched to OpenSearch client for Elastic (Apache 2 licensed)

(cherry picked from commit bef3071)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Feb 28, 2022
…nSearch one) (apache#13867)

* Upgraded ElasticSearch to get rid of CVEs. (apache#13747)

* Upgraded ElasticSearch to get rid of CVEs.

CVE-2020-7020
CVE-2020-7021
CVE-2021-22132
CVE-2021-22134
CVE-2021-22144
CVE-2021-22147

* Elastic search client version >= 7.11 no longer works with OSS Elastic images (and elastic.co no longer releases OSS images)

* Fixed tests for Elasticsearch

* pom cleanup

* Switched to OpenSearch client for Elastic (Apache 2 licensed)

(cherry picked from commit bef3071)
(cherry picked from commit 6deb24c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants