Zig-based alternative to linker scripts

[skyfex]

Sorry for the cheesy title, feel free to change it

With the current state and plans of Zig, it's looking like the only case where you need a language other than Zig for writing firmware/software, is if you need a custom linker script. What's worse, the documentation for GCC/LLD linker scripts are not all that great, and they can be hard to understand. The scripting language is also not really powerful enough on its own to do more complex things, so I've seen a case where the C preprocessor has been used to generate linker scripts.

If linker scripts could be replaced with Zig code, Zig would end up being an extremely elegant solution for embedded firmware in particular.

As far as I see it, there's two parts to solving this:

1. Linking can not be handled by compile time evaluation, but compile time constant values/structures should be available to the linker scripts. Some values, such as fixed memory regions and addresses, could be calculated at compile time. These values are generally useful for both the Zig runtime code itself, and to the linker script. There may be some work to define compile time functions, and pre-defined variable names or struct types that the linker can use.

2. Define a way for Zig code to run at link time, and define functions and data
   structures that gives the Zig linker code information about the sizes and flags of sections , and instruct the linker how code and data should be laid out in memory.

This feature may be dependent on the Zig self-hosting linker #1535, but it would be nice if a proof-of-concept could be done with LLD. Does LLD have an API that can be used or does it only work with linker scripts?

[andrewrk]

I like where this is going.

This feature may be dependent on the Zig self-hosting linker #1535, but it would be nice if a proof-of-concept could be done with LLD. Does LLD have an API that can be used or does it only work with linker scripts?

It's always possible that this feature would ultimately generate a linker script (without necessarily exposing it to the programmer) for use with LLD.

[skyfex]

I thought I'd play around with this idea. Is there a way to write a file at compile time? I couldn't find anything on that, and just putting standard file IO code in comptime doesn't seem to work.

Another question is if there's a good way of extracting symbols and their values (for constants) in the build system before linking?

[andrewrk]

It's not possible by design to write a file in comptime code. I don't think that's needed anyway for this feature.

The feature might need access to metadata about symbols in object files, assembled assembly files, and compiled C files.
To do that the code will probably need to read ELF files and PE files.

[markfirmware]

@skyfex @andrewrk I have a project that generates the .ld file:

https://github.com/markfirmware/zig-vector-table/blob/c69f04caf843d9b7977980a5230f3ced415003dc/build.zig#L118-L202

[markfirmware]

@skyfex most recent: https://github.com/markfirmware/zig-vector-table/blob/master/linker.zig

[skyfex]

Excellent. I looked a bit at doing build scripts, but it seemed the documentation there was lacking so it was a bit difficult. This looks very much like what I was thinking.

What I would like is for the information used to generate the linker script be available both at runtime and in the build script. But I suppose that's just a matter of having it in a shared zig file which is imported in both build.zig and the program itself?

I also looked at the source code of LLD to see if Zig could somehow direct linking without a linker script at all. It looks like it should be pretty simple, it's just a matter of setting up "OutputSections":

https://github.com/llvm-mirror/lld/blob/0f13b95a30df8808f4507259b45f4afa0b480d75/ELF/OutputSections.h

The tricky part is that some of the sections attributes are defined as dynamic expressions, e.g:
Expr addrExpr;

As far as I understand is that this is because an output section could be located relative to the address and size of sections that precede it, and their size depends on the number and size of the input sections that contain them. The address is evaluated after all the sections has been set up.

So the problem of defining a format for a Zig linker script boils down to how you decide to generate those expressions.

Expr is defined here btw, and is just a function:

https://github.com/llvm-mirror/lld/blob/d51111a692e7ec957b91b9ba3ec626c54c5c50a5/ELF/LinkerScript.h

using Expr = std::function<ExprValue()>;

Where ExprValue is just a struct that evaluates to an address, possibly relative to a section and possibly with alignment.

Could a function in build.zig (or imported in build.zig) somehow be passed along to LLD and be assigned as an Expr function?

[alexnask]

What I would like is for the information used to generate the linker script be available both at runtime and in the build script. But I suppose that's just a matter of having it in a shared zig file which is imported in both build.zig and the program itself?

I think the cleanest way to do this would be to use exe.addBuildOption(...) in your build file with the generated linker script as well as outputing the .ld file.
The script will then be available in @import("build_options")

[markfirmware]

Whilst pondering your comments, I added the generated files to the repo for your perusal.

https://github.com/markfirmware/zig-vector-table/tree/master/generated/generated_linker_files

[markfirmware]

Regarding build_options, that might be a nice place to put things but they need to be primitive types - see #3127.

Regarding working more directly with lld, that will take more effort than I have available. It does beg the question though of how much of the linker machinery needs to exposed to the bare metal programmer? They need to know flash vs ram and that code, read-only data and initial mutable data go in flash, bss needs to get set to 0 in ram and there can be data in ram with undefined initial values. Then there is the root, the vector table, in flash that drags in everything else. These are abstractions that must be understood and it might be possible to understood them without referring to a linker.

I dont think the .ARM.exidx section is needed - I need to study that further.

[pixelherodev]

Just a piece of data: custom linker scripts can be very useful for e.g. JITs on AMD64.

For instance, if I'm emulating a 32-bit system, and I can ensure that no host data (the application itself) resides at an address used by the system I'm emulating, I can map physical memory to the emulated device 1:1 and never have to translate addresses (which can significantly improve translation times).

[ikskuh]

There's a lot more you can do with linker scripts, even:

• Embedded Resources
• Encoding meta-data for plugin systems (just use a custom section)
• Add application icons
• Encode multiple entry points for non-standard OS (also via custom section)
• Encode compatibility information (use ELF files for firmware updates and encode the compatible hardware platforms in custom sections)