[CI] DebMetadataTests test05CheckLintian failing

[albertzaharovits]

Build scan:
https://gradle-enterprise.elastic.co/s/hx2aguhh5kmfi/tests/:qa:os:destructiveDistroTest.default-deb/org.elasticsearch.packaging.test.DebMetadataTests/test05CheckLintian

Reproduction line:
null

Applicable branches:
ONLY main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.packaging.test.DebMetadataTests&tests.test=test05CheckLintian

Failure excerpt:

org.elasticsearch.packaging.util.Shell$ShellException: Command was not successful: [bash -c lintian --fail-on-warnings /var/lib/jenkins/workspace/elastic+elasticsearch+main+multijob+packaging-tests-unix/os/ubuntu-18.04-packaging/distribution/packages/deb/build/distributions/elasticsearch-8.3.0-SNAPSHOT-amd64.deb]
   result: exitCode = [1] stdout = [W: elasticsearch: shlib-with-executable-stack usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/lib/libtorch_cpu.so
N: 403 tags overridden (189 errors, 171 warnings, 43 info)] stderr = [warning: the authors of lintian do not recommend running it with root privileges!
warning: --fail-on-warnings is deprecated]

  at __randomizedtesting.SeedInfo.seed([E4AA662FB0B3F8ED:2454FAC30AC79D56]:0)
  at org.elasticsearch.packaging.util.Shell.runScript(Shell.java:143)
  at org.elasticsearch.packaging.util.Shell.run(Shell.java:73)
  at org.elasticsearch.packaging.test.DebMetadataTests.test05CheckLintian(DebMetadataTests.java:38)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[albertzaharovits]

This started failing 5 days ago, on Friday:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P28D&search.timeZoneId=Europe/Bucharest&tests.container=org.elasticsearch.packaging.test.DebMetadataTests&tests.sortField=FAILED&tests.test=test05CheckLintian&tests.unstableOnly=true

[albertzaharovits]

FWIW: happens on ubuntu-18.04, debian10, and debian11

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

This is happening as a result of upgrading PyTorch from version 1.9 to version 1.11 in elastic/ml-cpp#2238.

On x86_64 the library libtorch_cpu.so really does have an executable stack now:

$ readelf -a libtorch_cpu.so | grep -C 2 GNU_STACK
  GNU_EH_FRAME   0x0000000006ff0434 0x0000000006ff0434 0x0000000006ff0434
                 0x0000000000183bcc 0x0000000000183bcc  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RWE    0x10
  GNU_RELRO      0x0000000007db4400 0x0000000007db5400 0x0000000007db5400

(It's the E in RWE.)

This is not the case on aarch64:

$ readelf -a libtorch_cpu.so | grep -C 2 GNU_STACK
  GNU_EH_FRAME   0x0000000003ddb188 0x0000000003ddb188 0x0000000003ddb188
                 0x00000000000fa60c 0x00000000000fa60c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x0000000004722370 0x0000000004732370 0x0000000004732370

There have been other cases in the past where PyTorch has needed to make changes to avoid an executable stack: pytorch/pytorch@639d1c7 and pytorch/pytorch@0e9613c

We need to try to find out which source file within PyTorch is causing this with PyTorch 1.11 and whether it's a mistake or deliberate this time.

PyTorch 1.11 is only in the main branch, so we have until 8.3.0 feature freeze to figure this out or else revert back to PyTorch 1.9.

[droberts195]

The same problem exists in the pre-built binary that you can download from https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.11.0%2Bcpu.zip.

$ readelf -a libtorch_cpu.so | grep -C 2 GNU_STACK
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000165f8000 0x00000000165f8000  R E    0x1000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RWE    0x10
  GNU_EH_FRAME   0x00000000163f3e3c 0x00000000163f3e3c 0x00000000163f3e3c

So it's not that we made a silly mistake when building it from source. The official download for 1.11 also has an executable stack.

I also downloaded https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.10.0%2Bcpu.zip and https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.9.0%2Bcpu.zip and found that 1.10 also has an executable stack but 1.9 doesn't. So it was a change between 1.9 and 1.10 that caused this.