Discussion: ObjWriter in C#

[filipnavara]

Let me start with a bit of a background. Last year I wrote a library for manipulating MachO object files in C# called Melanzana. Few weeks ago, I started doing some changes to the ObjWriter code and I found it to be quite suboptimal in terms of performance. As an exercise I wrote a prototype of an ObjWriter replacement in C# for the MachO files based on my library. I later extended it to emit DWARF debugging information and ELF files as well through the LibObjectFile library. Obviously, the code is not production ready and it is not on par with the current ObjWriter library but I wanted to gauge whether there would be an interest carrying this experiment forward.

What works?

• Producing MachO object files for osx-x64 and osx-arm64 targets
• Producing ELF files for linux-x64 and linux-arm64 targets
• Producing COFF files for win-x64 and win-arm64 targets
• DWARF debugging information for types, methods, variables, and line numbers
• CodeView debugging information for types, methods, variables, and line numbers

What doesn't work?

• No shared symbol (COMDAT) support for ELF
• No ARM32 and X86 support (NativeAOT doesn't properly support them anyway)

The obvious advantage of the approach is that the object writing libraries (whether it is LibObjectFile or Melanzana) are closer to the data model that the ILCompiler emits. That makes it more efficient at producing the raw section data, relocations, and symbol tables. The LLVM-based ObjWriter has high overhead (15%-30% of the whole compilation process in my tests). Some of the overhead can be reduced (eg. switching sections is expensive) but it usually comes at the cost of writing at least some part of the code in C#.

The disadvantage is that this is a lot of code and two external library references. Essentially, it's trading one dependency for another. While MachO and ELF formats are already part of the initial experiment the COFF one is not (all Windows targets).

There's also a middle way. Parts of the experiment can be reused to feed the data more efficiently into the current ObjWriter. For example, the unwinding sections (__eh_frame, __compact_unwind) could be produced completely in the managed code. That would avoid lot of overhead for section switching at minimal impact on portability (can be used only for specific targets).

The experiment also serves as a good testbed to quickly evaluate some space savings. Some little things I noticed:

• Non-primary LSDA frames contain relative pointer to main LSDA. This can be generated without symbols and relocations easily.
• Currently MachO produces __eh_frame data with 8-byte PC relative pointers. The linker can handle 4-byte ones as well so they can be used (as already done for ELF).
• ELF produces PLT relocations even within the same section. This seems to inhibit some optimizations and it's not recommended.

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[filipnavara]

cc @MichalStrehovsky @kant2002

[kant2002]

My unlove to ObjWriter comes from one crash which I diagnose dotnet/runtimelab#1316 (comment) whole thing memory hungry during compilation. NativeAOT already have rough experience so adding one occasional OOM does not helps.

[filipnavara]

The memory usage could be improved significantly. In fact, aside from the DWARF part, it likely uses way less memory already. There is potential for even bigger savings since I currently write the section data into MemoryStreams but I could easily reuse the existing arrays and just wrap them in a custom Stream type that iterates over them...

FWIW I am perfectly fine with ditching this as hack week project but I am also fine salvaging the good parts if it makes things fast(er) or improves the source build story.

[am11]

+1 for the idea and spirit! 🙂

No DWARF5 yet

FWIW, HP libunwind (the one used in CoreCLR PAL) also does not parse DWARF 4 and 5 correctly yet.

it usually comes at the cost of writing at least some part of the code in C#.

Agreed. One less dependency is also a positive for source-build. Moreover, continuous maintenance of dependency has a measurable cost and working across repos using nuget transport is another con of ObjectWriter.

ps - prior to ObjectWriter, there used to be an AsmWriter written in C#: dotnet/corert@ad6e1ba.

[filipnavara]

No DWARF5 yet

FWIW, HP libunwind (the one used in CoreCLR PAL) also does not parse DWARF 4 and 5 correctly yet.

The current ObjWriter DWARF 5 support is basically just writing different version number in the header. It doesn't use any of the new features and it's essentially useless. I could easily replicate that but there's really no point except for checking a checkbox.

The main benefit of DWARF 5 seems to be reducing the number of relocations. That's also non-issue for Mach-O since no relocations are necessary. Apple's linker doesn't copy DWARF to the final executable, instead it produces debug maps that reference the original DWARF data in the .o files. They can by collected by dsymutil into a bundled format but there's still no need for relocations.

[MichalStrehovsky]

Cc @markples @BrianBohe for thoughts on removing the LLVM dependency

[filipnavara]

I pushed a bit more code to the branch. Now it can produce ELF and COFF too. There are still feature gaps but it's much closer to supporting all the platforms that NativeAOT can target today.

[filipnavara]

I added CodeView debugging support. That brings it close to full parity with the old code on the supported platforms. It still needs cleanup, comments, and fixing some register mappings.

[filipnavara]

I pushed a version with the memory usage optimization mentioned in #77178 (comment). A non-scientific benchmark with Stopwatch for generating COFF+CodeView currently looks like this:

> ilc "@artifacts\tests\coreclr\obj\windows.x64.Debug\Managed\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.ilc.rsp"
Emitting object file took 00:00:00.9877349
> $Env:DOTNET_USE_LLVM_OBJWRITER=1
> ilc "@artifacts\tests\coreclr\obj\windows.x64.Debug\Managed\nativeaot\SmokeTests\Exceptions\Exceptions\native\Exceptions.ilc.rsp"
Emitting object file took 00:00:03.5289337

[agocke]

Hey Filip, we're still thinking about the replacement. Compile time improvement would be nice, but I don't think it's very high priority since users can mostly use the JIT version for day-to-day development. Size-on-disk improvements are much more interesting.

Overall, I'm not sure whether we feel a managed version is more maintainable than LLVM, and we may have some other uses for LLVM anyway, so we might end up wanting to keep the code around regardless. We'll just need some more time to talk to some other engineers in the org who have more experience and would be better informed about the cost of maintaining LLVM objwriter.

Regardless, the work you've done is great! Very impressive.

[filipnavara]

I appreciate the feedback.

I don't think the maintainability is significantly worse (or better) than the current ObjWriter.
It would be possible to drop external library dependencies altogether, if that makes any difference. It's just how the experiment evolved. That would simplify source builds a bit.

The speed improvements could be significant but it doesn't necessarily scale with size. I built tests in the whole repository and sometimes it saves 10 seconds, other times it saves 3. It makes a difference but it's not the be all and end all.