[GR-40874] Perf support

[adinn]

This PR implements changes to Linux debug info generation specified in issue #4810. These enable the Linux tool perf to display profile information for GraalVM native images.

This patch 'fixes' perf modulo from a few small remaining details:

perf expects many references from definitions to associated declarations (specification or abstract_origin attribute values) to be implemented as compile unit (CU) local offsets because it assumes that a single CU owns both the referring and referenced records. There are still a few cases in Java debug info where definitions and declarations need to be in different CUs and so must use an info section global offset. This is no problem for gdb but perf cannot cope with it.

perf does display generated machine code for sampled code addresses. However, it appears to be adding an extra 4K offset to the code address before decompiling, meaning that the wrong code is displayed. I am not clear why this is happening.

perf occasionally lists sampled method code addresses as hex numbers (e.g. something like 0x4061ba) instead of using the fully qualified method name. I am not clear why this is happening. The odd thing is that perf delegates to binary addr2line to do name, file and line lookup and this program prints the correct name, file, line and even inline frame info for these addresses.

[adinn]

@stooke It would be good if you could review this patch even though it is mostly incidental to the Windows code. There is one semantic change which I believe does not affect the Windows code but I'd very much like you to validate that assumption.

I have repurposed fields primaryClasses and primaryClassesIndex of class DebugInfoBase, renaming them to instanceClasses and instanceClassesIndex. The associated iterator has also changed named to getInstanceClasses().

The associated collection instances were originally populated with ClassEntry records for all classes notified via the DebugCodeInfo API i.e. classes which had at least one top level compiled method. When inline info arrived the list and associated index was supplemented with any extra classes for which methods were found in inline frames.

That meant that some of these so-called primary classes did not actually have any top level compiled methods. Luckily code that processes entries in these collections appears to always check whether the entry has any compiled methods, by calling ClassEntry::isPrimary(), or not care and simply iterate over an empty list of primaryEntry records.

In consequence I have repurposed this list so that it is populated with all instance classes that get presented via the DebugTypeInfo API (i.e. excluding array types,primitive types and the object header struct type). This then allows the Linux code to generate debug info for all instance classes by calling getInstanceClasses(). It would be very helpful if you could check whether this is ok as far as the Windows code is concerned.

[stooke]

@adinn I'll take a look.

[zakkak]

Thanks @adinn. It looks good to me other than some typos and a few comments I added for consideration.

[SergejIsbrecht]

@adinn, regarding perfs behavior you could probably ask the maintainer Arnaldo (Red Hat). This is his handle on Twitter https://twitter.com/acmel?t=Dt5BWUsXmY27r7REYbSCuw&s=09

[adinn]

@SergejIsbrecht I resolved the problem with display of hex addresses for some methods in the most recent push. The problem is that perf only starts translating sampled code addresses from some given code section (.text, .plt, .init) to fn/method names if the address lies in the range starting at the first exported symbol. It seems to assume that code addresses preceding the first exported symbol occupy dead space.

So, if you don't preserve method symbols (i.e. pass -H:-DeleteLocalSymbols) then the first symbol will likely be some way down the .text segment. With the current GraalVM this is about 4M from the start, where class VMLocator exposes a method as a C entry point. Samples for methods which follow this symbol get looked up and reported using the DWARF name. Samples for methods that precede it are reported as hex addresses.

The fix is to add a dummy code symbol at the start of the .text section when DeleteLocalSymbols has value true.

[adinn]

@SergejIsbrecht Also, assuming you do retain local symbols, the latest version of this patch will provide a pretty reliable perf annotate output for compiled Java methods. You can run a bare call

$ perf annotate

to see stats for the whole program. Alternatively, you can annotate a single method by passing its symbol name

perf annotate PrintStream_format_9338ddec90bb0bd89998e4b56896abba6d7c4ded

Unfortunately, at present you have to provide the symbolic name for the method, not the DWARF name java.io.PrintStream::format. That long string is an SHA1 digest of the actual method signature. If you want to identify the symbolic names you can run nm on your generated binary image.

I'd be interested to hear your feedback on the current state of play with this feature.

[SergejIsbrecht]

@adinn , I am currently compiling for x64. I will attach some FlameGraphs, when everything is running with your branch.

[SergejIsbrecht]

@adinn ,

first of all it look way better in comparison to before, but I still see some "unknown" addresses

-H:Class=hello.HelloWorld -H:IncludeResources=.*/BenchmarkList -H:Log=registerResource:verbose --initialize-at-build-time=org.openjdk.jmh.infra,org.openjdk.jmh.util.Utils,org.openjdk.jmh.runner.InfraControl,org.openjdk.jmh.runner.InfraControlL0,org.openjdk.jmh.runner.InfraControlL1,org.openjdk.jmh.runner.InfraControlL2,org.openjdk.jmh.runner.InfraControlL3,org.openjdk.jmh.runner.InfraControlL4 -H:-SpawnIsolates -H:+PreserveFramePointer -H:GenerateDebugInfo=2

     0,47%     0,00%  ool-6-worker-23  bench               [.] com.oracle.svm.core.posix.thread.PosixPlatformThreads::setNativeName (inlined)                                                ▒
     0,46%     0,00%  ool-6-worker-13  bench               [.] org.jctools.queues.MpscUnboundedArrayQueue::poll (inlined)                                                                    ▒
     0,46%     0,00%  ool-6-worker-13  bench               [.] org.jctools.queues.BaseMpscLinkedArrayQueue::poll (inlined)                                                                   ▒
     0,43%     0,00%  te-jmh-worker-1  anon                [.] 0x00007fe4cf00116f                                                                                                            ▒
     0,42%     0,00%  te-jmh-worker-1  anon                [.] 0x00007fe51a7011f7                                                                                                            ▒
     0,42%     0,00%  te-jmh-worker-1  anon                [.] 0x00007fe4cea011f7                                                                                                            ▒
     0,40%     0,00%  te-jmh-worker-1  anon                [.] 0x00007fe518b02727                                                                                                            ▒
     0,36%     0,05%  te-jmh-worker-1  [vdso]              [.] __vdso_clock_gettime                                                                                                          ▒
     0,36%     0,00%  te-jmh-worker-1  libc-2.31.so        [.] __clock_gettime_2 (inlined)                                                                                                   ▒
     0,28%     0,00%  te-jmh-worker-1  anon                [.] 0x00007fe518b4e477                                                                                                            ▒
     0,24%     0,24%  te-jmh-worker-1  [vdso]              [.] 0x00000000000006c8                                                                                                            ▒
     0,24%     0,00%  te-jmh-worker-1  [vdso]              [.] 0x00007ffde6ffe6c8       

(perf report output)

I compiled a JMH benchmark to a native-image and ran perf record -F 99 --call-graph dwarf -- ./bench

(full view of jmh-worker-1: unknown and anon visible)

(partial view of jmh-worker-1)

(partial view of jmh-worker-1)
(Flamegraphs were created with https://github.com/flamegraph-rs/flamegraph)

./flamegraph --deterministic --palette java -c "record -F 99 --call-graph dwarf" -- ./bench

If you would like I could create some FlameGraphs with async-profiler with an actual JVM for comparisons.

Also there are still some SHA1 hashes around

-    8,83%     8,83%  ool-6-worker-13  bench               [.] IsolateEnterStub__VmLocatorSymbol__vmLocatorSymbol__bec84cad1f8708102cd8814ef3e496531bf6ff5b__b6484ec9f5c47b82c12ad1112cecc290▒
     8,63% java.util.concurrent.ForkJoinPool::runWorker (inlined)                                                                                                                            ▒
        start_thread                                                                                                                                                                         ▒
        com.oracle.svm.core.code.IsolateEnterStub::PosixPlatformThreads_pthreadStartRoutine_38d96cbc1a188a6051c29be1299afe681d67942e (inlined)                                               ▒
        com.oracle.svm.core.posix.thread.PosixPlatformThreads::pthreadStartRoutine (inlined)                                                                                                 ▒
        com.oracle.svm.core.thread.PlatformThreads::threadStartRoutine (inlined)                                                                                                             ▒
        java.util.concurrent.ForkJoinWorkerThread::run (inlined)                                                                                                                             ▒
        java.util.concurrent.ForkJoinPool::runWorker (inlined)                                                                                                                               ▒
        java.util.concurrent.ForkJoinPool::scan (inlined)                                                                                                                                    ▒
        java.util.concurrent.ForkJoinPool$WorkQueue::topLevelExec (inlined) 

[adinn]

@SergejIsbrecht Thanks very much for doing that! It certainly looks very interesting.

I still see some "unknown" addresses ...
0,43% 0,00% te-jmh-worker-1 anon [.] 0x00007fe4cf00116f
0,42% 0,00% te-jmh-worker-1 anon [.] 0x00007fe51a7011f7

I do not know where those sample addresses come from but they most definitely look like they are not from method code compiled into the .text section (nor even code provided by SVM in the .init section). .text usually starts around 0x400500 for a small program, maybe a bit more for a large one. objdump -h <imagefile> will tell you the start VMA and extent of section .text for your image. Those addresses looks like they have come out of a dynamically allocated region. They might be in a dynamic shared library, I suppose, but I would have expected perf to be able to provide an address for whatever shared lib code your program relies on.

If you would like I could create some FlameGraphs with async-profiler with an actual JVM for comparisons.

Yes, that might be interesting to see how easy it is to compare what is going on under the actual benchmark run methods. Of course, your graph will include a slab of profile data for both JVM startup and benchmark warmup (including a load of execution bundled as 'interpreter') all of which will be missing in the Graal case.

Also there are still some SHA1 hashes around

• 8,83% 8,83% ool-6-worker-13 bench [.] IsolateEnterStub__VmLocatorSymbol__vmLocatorSymbol__bec84cad1f8708102cd8814ef3e496531bf6ff5b__b6484ec9f5c47b82c12ad1112cecc290
  8,63% java.util.concurrent.ForkJoinPool::runWorker (inlined)
  start_thread
  com.oracle.svm.core.code.IsolateEnterStub::PosixPlatformThreads_pthreadStartRoutine_38d96cbc1a188a6051c29be1299afe681d67942e (inlined)
  com.oracle.svm.core.posix.thread.PosixPlatformThreads::pthreadStartRoutine (inlined)

That's not necessarily an indication that the name is being derived from a symbol rather than from the debuginfo. In this case the outermost pthread start method whose name includes an SHA1 hash is a synthetic stub method. Methods like this one only ever get presented to the debug info generator with a method name that includes an SHA1 hash (there is no 'real' Java name). In this case I am certain that the printed name has been derived from debug info because the class name is package qualified. If a symbol had been used to name the sample then there would be no package qualifier (the symbol naming algorithm used by GraalVM constructs symbols as BareClass_Method_sha1hash).

[adinn]

@SergejIsbrecht Thinking about it those weird addresses may be the result of improper stack unwinding. That could indicate an issue with the generation of the .debug_frame section. I'll see if I can reproduce similar weirdness and then try to debug it.

[SergejIsbrecht]

@adinn,

ll see if I can reproduce similar weirdness and then try to debug it.

Don't bother. I will give you one later that day.

[SergejIsbrecht]

@adinn ,

I am currently working on an example. Here are some FlameGraphs with Java11 Hotspot C2. FlameGraphs are captured with async-profiler 2.8 on Linux (5.4)

JMH Gradle Plugin:

profilers.set(listOf("async:libPath=/home/user/Applications/async-profiler-2.8-linux-x64/build/libasyncProfiler.so;output=flamegraph;cstack=fp;event=cpu;interval=5000000;threads=false"))

https://gist.github.com/SergejIsbrecht/dda8abca7c3c1004f03f1507bc1e9240

(Just download the HTML files and open them)

https://gist.github.com/SergejIsbrecht/6a87bd9a5cf4ef89b7ec9ff2f8f04d95

This FlameGraph was captured with your branch and flamegraph-rs.

flamegraph --deterministic --palette java -c "record -F 99 --call-graph dwarf" -- ./bench

The Hotspot C2 FlameGraph kind of looks different. There are a lot of consumeCPU stackframes, which are not available in the native-image ones. Maybe they were just inlined? It would be strange when the native-image would inline this calls, but the Hotspot C2 would not.

[olpaw]

Thanks a lot for a adding perf-support!

See individual review comments. In addition:

• Please also test if this works with images that are shared libraries (i.e. built with --shared).

• Please also add documentation on how to use perf with native-images.
  Either in docs/reference-manual/native-image/DebugInfo.md of you create an new docs/reference-manual/native-image/Profiling.md. It should contain info how the image needs to be built so that it can be profiled with perf, what is currently working, current limitations, plans/ideas for future improvements.

[olpaw]

You can add a non-API HostedOption (false by default) and use it here to throw a VMError if it helps you with finding if there are still cases where you reach this code.

[adinn]

I am confident we can now reduce this call to a lookup, with an assert that the result is found, and dispense with any need for proof. So, I'll push an extra change to that effect.

Why so? Well, the continued existence of this method is an artefact of creeping change in the meaning of the fields now renamed to instancesClasses and instanceClassesIndex. Originally these fields tracked instance classes with top level compiled methods. The methods and associated classes were presented via the DebugCodeInfo API and ultimately derived by iterating over the NativeImageCodeCache. The expectation was that all of these classes would already have been presented via the DebugTypeInfo API, ultimately derived by iterating over the Universe type. However, the list was originally only meant to include the proper (i.e. strict) subset of the notified instance classes which needed to be qualified with method location and range info.

When we started reporting frame data via the DebugCodeInfo API we found classes that were i) not already identified via the DebugTypeInfo API or ii) had inlined methods but no top level methods. These also required generation of inline method location and range info. So, we threw them into the list and index, creating a ClassEntry if none existed and then sorted them out case by case in code that used the list of classes.

The first problem was an input hygiene problem. We ran into 'missing' types because frames were actually referencing analysis types instead of hosted types. That has been fixed on the notification side (NativeImageDebugInfoProvider) by remapping classes from analysis to hosted domain before passing them through the DebugCodeInfo API.

The second problem reflect the fact that we really need to iterate over all instance classes at certain points in the code and only over some instance classes at others. A ClassEntry now identifies whether or not it has top level compiled methods and/or inlined methods and all code that now consumes the instanceClasses list tests for the relevant cases.

So, both the problems we originally faced appear to be resolved. The current code populates this list and index with all expected instance classes during traversal of universe types set. So, the checks and updtes that used to be done in ensureClassEntry really ought to be redundant.

n.b. There is another follow-up task needed to consolidate the current position beyond redefining ensureClassEntry. From one point of view fields instanceClasses and instanceClassesIndex are redundant. We now only consume the instaceClasses list via a Stream API. The stream content manifested as a list in instanceClasses can be derived virtually by filtering field types with a type test for ClassEntry and cast before returning. Similarly, lookups in the index map instanceClassesIndex can be replaced with a lookup in the index map typesIndex that also employs an instanceof check. So, we have the option to virtualize the instance class list and index, trading off space against a small amount of stream processing time.

Whichever way we decide to go with this a code change is needed. There are a couple of cases left where iteration over all instance classes was and still is performed by streaming and filtering field types (this made sense when not all instance classes were listed in field instanceClasses). I'm happy to fix this as part of this patch or in a follow-up. Do you have a preference for retaining two lists or virtualizing one of them?

[adinn]

Fixed. I retained the separate list of instance classes but everything is now using streams derived either from the types list or from the classes list.

[adinn]

@SergejIsbrecht

The Hotspot C2 FlameGraph kind of looks different. There are a lot of consumeCPU stackframes, which are not available in the native-image ones. Maybe they were just inlined? It would be strange when the native-image would inline this calls, but the Hotspot C2 would not.

Thanks for providing the flame graphs -- they are very interesting. I am not sure why you are seeing frames omitted in the native graph but I have a suspicion what is going on. Before mving on to that I'll note that both Hotspot and GraalVM should be able to allocate sample addresses to inlined methods rather than simply to the top level compiled method. However, this will rarely be 100% accurate. Because of the way code gets reorganized during optimization machine code that derives from one method often gets accounted for as belonging to another method. That said, I don't think this explains the disparity seen in your example.

So, here's my guess as to what is happenign. When you built your image did you pass option -H:+SourceLevelDebug? When you omit this option then very little frame information is passed through from the compiler to the debugger. That means that, apart from obvious places like call points and exception handler entries, where a top level and inline frame stack has to be well-defined, and maybe one or two other places where frame info is available, the debug generator can only assume that code addresses belong to the top level method. That might account for why all of the samples that should belong to consumeCPU are being allocated to the Lambda...run method.

n.b. the reason this argument is not the default is that including extra frame info in the compiler graph may cause the register allocator to think that values in registers are alive for slightly longer than would otherwise be assumed. That's not necessarily going to be a problem for all compiled methods but it can easily cause a noticeable drop in performance for benchmarks, especially micro-benchmarks. So, be aware that if you do switch this flag on you might see your benchmark run a bit slower.

[SergejIsbrecht]

@adinn ,

it does not seem to matter whether I use -H:+SourceLevelDebug or -H:-SourceLevelDebug when generating the image. At least the generated FlameGraphs looked the same. I did not see any consume stack-frames.

When I run perf with perf record -F 499 --call-graph dwarf -- ./bench I get following perf report:

perf 279819 66305.036635:       1792 cycles: 
        ffffffffafc73616 native_write_msr+0x6 ([kernel.kallsyms])
        ffffffffafc0d335 intel_tfa_pmu_enable_all+0x35 ([kernel.kallsyms])
        ffffffffafc07addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)

While importing following issue is displayed:

The hex address seems quite unusual.

            55e712eb128a com.oracle.svm.core.posix.thread.PosixPlatformThreads::pthreadStartRoutine+0x6408a (inlined)
            55e712e4d05c com.oracle.svm.core.code.IsolateEnterStub::PosixPlatformThreads_pthreadStartRoutine_38d96cbc1a188a6051c29be1299afe681d67942e+0x17c (inlined)
            7efd88131608 start_thread+0xd8 (/usr/lib/x86_64-linux-gnu/libpthread-2.31.so)
                       7 [unknown] ([unknown])

(perf report)

Later today I will run some native rust / c++ application with perf in order to see whether I see such unusual addresses.

[adinn]

It does not seem to matter whether I use -H:+SourceLevelDebug or -H:-SourceLevelDebug when generating the image. At least the generated FlameGraphs looked the same. I did not see any consume stack-frames.

Ok, thanks very much for testing it.

When I run perf with perf record -F 499 --call-graph dwarf -- ./bench I get following perf report:

perf 279819 66305.036635: 1792 cycles:
ffffffffafc73616 native_write_msr+0x6 ([kernel.kallsyms])
ffffffffafc0d335 intel_tfa_pmu_enable_all+0x35 ([kernel.kallsyms])
ffffffffafc07addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xc14d2 vs 0x90ed8)
...

Well it looks like the debuginfo is incorrect. Can you provide me with the benchmark source and detailed instructions on how to generate the binary? I'd really like to have a look at the debuginfo it contains. I'd also like to run addr2line under gdb to see what is causing it to print those errors.

[SergejIsbrecht]

@adinn, sure. I will provide a working example in some hours. Anyways, this jmh bench is quite trivial, therefor any code should suffer from the same issue.

[zakkak]

Thanks for addressing all the comments Andrew. It looks good to me!

[olpaw]

Would be cool if c++filt would support our mangling.

[adinn]

Yes it would. I have had a quick look into what perf does to unmangle names. By default it calls bfd_demangle which has its own slightly weird expectations about how to handle names e.g. arrays of TYPE are modelled as JArray<TYPE>. If (when) that fails it should punt to a routine in the perf code called java_demangle_sym. This knows how to demangle bytecode type and member signatures into Java syntax. For example it will turn Ljava/lang/StringLatin1;equals([B[B)Z into boolean java.lang.StringLatin1.equals(byte[], byte[])". We might be able to use that for some cases but I guess it won't do for methods exposed as C entry points which actually need a linker-friendly name.

Note that anyway I am not currently using the symbol name as the linkage_name atttribute in the method.definition. At present I generate that attribute as <fully_qualified_class_name>::<bare_method_name> in order to ensure that 1) perf report prints something sensible when there are no local symbols 2) perf annotate includes this name at the start of the listing for a given method. We can probably do better than this (but maybe in a follow-up patch).

[olpaw]

but maybe in a follow-up patch

Definitely

[olpaw]

Can you point me at the things that need fixing?

I will run the PR in our CI to see what comes up.

[olpaw]

Can you point me at the things that need fixing?

I will run the PR in our CI to see what comes up.

mx --kill-with-sigquit --strict-compliance debuginfotest --output-path /b/b/e/main/substratevm/svmbuild failed:

(gdb) backtrace
Checker backtrace hello.Hello::main: match 3 failed at line 3 #3  com.oracle.svm.core.JavaMainWrapper::doRun (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:232

Checker backtrace hello.Hello::main {
  re.compile('#0[ \t]+hello\\.Hello::main .* at hello/Hello\\.java:76')
  re.compile('#1[ \t]+0x[0-9a-f]+ in com\\.oracle\\.svm\\.core\\.JavaMainWrapper::runCore0 .* at [a-z/]+JavaMainWrapper\\.java:[0-9]+')
  re.compile('#2[ \t]+0x[0-9a-f]+ in com\\.oracle\\.svm\\.core\\.JavaMainWrapper::runCore .* at [a-z/]+JavaMainWrapper\\.java:[0-9]+')
  re.compile('#3[ \t]+ com\\.oracle\\.svm\\.core\\.JavaMainWrapper::run .* at [a-z/]+JavaMainWrapper\\.java:[0-9]+')
  re.compile('#4[ \t]+0x[0-9a-f]+ in com\\.oracle\\.svm\\.core\\.code\\.IsolateEnterStub::JavaMainWrapper_run_[0-9a-f]+ .*')
}

#0  hello.Hello::main (args=0x7fa367e010f0) at hello/Hello.java:76
#1  0x000000000041447a in com.oracle.svm.core.JavaMainWrapper::runCore0 () at com/oracle/svm/core/JavaMainWrapper.java:175
#2  0x000000000041413d in com.oracle.svm.core.JavaMainWrapper::runCore () at com/oracle/svm/core/JavaMainWrapper.java:135
#3  com.oracle.svm.core.JavaMainWrapper::doRun (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:232
#4  0x0000000000442e1e in com.oracle.svm.core.JavaMainWrapper::run (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:218
#5  com.oracle.svm.core.code.IsolateEnterStub::JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b (__0=<optimized out>, __1=<optimized out>) at com/oracle/svm/core/code/IsolateEnterStub.java:1

WARNING: psutil is not available, java process detection is less accurate
  File "/b/b/e/mxtool/mx_enter.py", line 41, in <module>
    mx._main_wrapper()
  File "/b/b/e/mxtool/mx.py", line 18093, in _main_wrapper
    main()
gate: 07 Sep 2022 13:32:35(+18:53) END:   image debuginfotest [0:01:46.338883] [disk (free/total): 362.5GB/511.3GB]

[adinn]

mx --kill-with-sigquit --strict-compliance debuginfotest --output-path /b/b/e/main/substratevm/svmbuild failed:

(gdb) backtrace
Checker backtrace hello.Hello::main: match 3 failed at line 3 #3  com.oracle.svm.core.JavaMainWrapper::doRun (argc=<optimized out>, argv=<optimized out>) at com/oracle/svm/core/JavaMainWrapper.java:232

Ok, I think that's from the latest rebase thanks to commit 1b6c1bf "Add support for running a native-image main entry point in a new thread"

It introduces a new call into the runCore hierarchy by delegating to doRun or doRunInNewThread

This should have been caught by an mx debuginfotest gate test when the patch was merged. Any idea why not?

A patch is on its way.

[adinn]

Ah, that patch did update the test script. It looks like I broke it again when I did the merge.

[olpaw]

mx --kill-with-sigquit --strict-compliance debuginfotest --output-path /b/b/e/main/substratevm/svmbuild failed:

btw, apart from that everything looks good 🤞

[adinn]

btw, apart from that everything looks good crossed_fingers

Ok, great. I just pushed a patch that fixes the debug info test. All that is left is to squash the commits.

[adinn]

And there's the squashed version

[olpaw]

And there's the squashed version

Restarted internal CI

[olpaw]

@adinn the PR is in the merge queue.