For 4.1.4: Changes required to use zip instead of 7z (backport #14301)

[mergify]

The erlang.mk docs state that you can drop-in zip on systems that don't have, or can't install, the p7zip package. Unfortunately, this doesn't quite work with the RabbitMQ build due to a difference in how zip adds a directory passed to it as an argument vs how 7z does.

When 7z is passed a directory as an argument, it uses the basename of the directory when adding it to the archive. This is important during the CLI tools build, because several Elixir applications are added to the final archive. Here is the command and relevant output lines showing this behavior:

make V=2 2>&1 | tee /tmp/make-7zip.txt

Notice that absolute paths to Elixir app dirs are passed to 7z, yet the basename of those paths are added to the archive:

set -x; 7z a -tzip -mx=9 -mtc=off  /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/*
+ 7z a -tzip -mx=9 -mtc=off /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/elixir /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/ex_unit /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/iex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/logger /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/mix

Here is what is archived in .erlang.mk/escript.zip:

zip file contents

$ unzip -t .erlang.mk/escript.zip
Archive:  .erlang.mk/escript.zip
    testing: csv/ebin/Elixir.CSV.Decoding.Decoder.beam   OK
    testing: csv/ebin/Elixir.CSV.Decoding.Parser.beam   OK
    testing: csv/ebin/Elixir.CSV.Defaults.beam   OK
    testing: csv/ebin/Elixir.CSV.Encode.Any.beam   OK                                                                                                                                                                                             testing: csv/ebin/Elixir.CSV.Encode.BitString.beam   OK                                                                                                                                                                                       testing: csv/ebin/Elixir.CSV.Encode.beam   OK                                                                                                                                                                                                 testing: csv/ebin/Elixir.CSV.Encoding.Encoder.beam   OK
    testing: csv/ebin/Elixir.CSV.EscapeSequenceError.beam   OK
    testing: csv/ebin/Elixir.CSV.RowLengthError.beam   OK
    testing: csv/ebin/Elixir.CSV.StrayEscapeCharacterError.beam   OK
    testing: csv/ebin/Elixir.CSV.beam   OK
    testing: csv/ebin/csv.app         OK
    testing: csv/ebin/dep_built       OK                                                                                                                                                                                                          testing: eex/                     OK                                                                                                                                                                                                          testing: eex/ebin/                OK
    testing: eex/ebin/Elixir.EEx.Compiler.beam   OK
    testing: eex/ebin/Elixir.EEx.Engine.beam   OK
    testing: eex/ebin/Elixir.EEx.SmartEngine.beam   OK
    testing: eex/ebin/Elixir.EEx.SyntaxError.beam   OK
    testing: eex/ebin/Elixir.EEx.beam   OK
    testing: eex/ebin/eex.app         OK
    testing: eex/lib/                 OK
    testing: eex/lib/eex/             OK
    testing: eex/lib/eex/compiler.ex   OK
    testing: eex/lib/eex/engine.ex    OK
    testing: eex/lib/eex/smart_engine.ex   OK
    testing: eex/lib/eex.ex           OK
    testing: elixir/                  OK
    testing: elixir/ebin/             OK
    testing: elixir/ebin/Elixir.Access.beam   OK
    testing: elixir/ebin/Elixir.Agent.Server.beam   OK
    testing: elixir/ebin/Elixir.Agent.beam   OK

When zip is passed a directory as an argument, the full path to that directory is re-created within the archive, which breaks the expectation of escript and the BEAM when it runs. Here is the command and relevant output lines showing this behavior. Note that you must use the -r argument or zip won't descend into directories passed to it.

verbose output when zip is non-recursive

set -x; /usr/bin/zip -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/*
+ /usr/bin/zip -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/elixir /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/ex_unit /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/iex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/logger /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/mix
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/elixir/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/ex_unit/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/iex/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/logger/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/mix/ (stored 0%)

make ESCRIPT_ZIP='/usr/bin/zip -r -9' V=2 2>&1 | tee /tmp/make-zip-bad.txt

From the /tmp/make-zip-bad.txt output file:

set -x; /usr/bin/zip -r -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/*
+ /usr/bin/zip -r -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/elixir /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/ex_unit /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/iex /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/logger /home/lrbakken/opt/elixir/1.18.4-otp-27/lib/mix
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/ (stored 0%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/Elixir.EEx.SyntaxError.beam (deflated 21%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/eex.app (deflated 57%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/Elixir.EEx.Engine.beam (deflated 25%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/Elixir.EEx.Compiler.beam (deflated 18%)
  adding: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/Elixir.EEx.beam (deflated 17%)

From the .erlang.mk/escript.zip archive itself:

$ unzip -t .erlang.mk/escript.zip
Archive:  .erlang.mk/escript.zip
    testing: rabbitmq_cli/ebin/Elixir.CSV.Encode.List.beam   OK
    testing: rabbitmq_cli/ebin/Elixir.CSV.Encode.Map.beam   OK
    testing: rabbitmq_cli/ebin/Elixir.CSV.Encode.PID.beam   OK
    testing: rabbitmq_cli/ebin/Elixir.CSV.Encode.Tuple.beam   OK
    testing: rabbitmq_cli/ebin/Elixir.RabbitCommon.Records.beam   OK
...
...
...
    testing: stdout_formatter/ebin/stdout_formatter_table.beam   OK
    testing: stdout_formatter/ebin/stdout_formatter_utils.beam   OK
    testing: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/   OK
    testing: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/   OK
    testing: home/lrbakken/opt/elixir/1.18.4-otp-27/lib/eex/ebin/Elixir.EEx.SyntaxError.beam   OK
...
...
...

With this patch, everything works correctly. Here's the command I used to test this patch and capture output. Since we're directly globbing the ebin dirs for Elixir libraries, as well as other Elixir deps, the -r argument is no longer necessary to use with zip:

make ESCRIPT_ZIP='/usr/bin/zip -9' V=2 2>&1 | tee /tmp/make-zip-with-patch.txt

Relevant lines from the make-zip-with-patch.txt file. Notice that the existing code in erlang.mk to add Elixir deps does add them in an APP/ebin/* format, which works with both 7z and zip. This is why I structured this patch in the same manner:

set -x; cd /home/lrbakken/development/rabbitmq/rabbitmq-server/deps && /usr/bin/zip -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip \
	csv/ebin/* json/ebin/* stdout_formatter/ebin/*
+ cd /home/lrbakken/development/rabbitmq/rabbitmq-server/deps
...
...
...
set -x; cd /home/lrbakken/opt/elixir/1.18.4-otp-27/lib && /usr/bin/zip -9 /home/lrbakken/development/rabbitmq/rabbitmq-server/.erlang.mk/escript.zip eex/ebin/* elixir/ebin/* logger/ebin/* mix/ebin/*
+ cd /home/lrbakken/opt/elixir/1.18.4-otp-27/lib

Finally, here's the full output of unzip -t on .erlang.mk/escript.zip, using this patch:
escript.zip-content.txt

---

This is an automatic backport of pull request #14301 done by [Mergify](https://mergify.com).