[sysvabi64] Add chapter on Thread Local Storage #311

smithp35 · 2025-02-20T17:37:00Z

The thread local storage chapter contains:

A description of Thread Local Storage based on addenda32
The key design decisions of AArch64 TLS such as tls variant, tls dialect, TCB size.
The ABI required code sequence for TLSDESC that must be emitted exactly, as GNU ld requires it to be.
Sequences for the different code-models.
Relaxations for GD->IE, GD->LE and IE->LE.
Synchronization requirements for Lazy TLSDESC. With advice not to support it due to overhead of synchronization.

MaskRay · 2025-02-21T06:19:34Z

sysvabi64/sysvabi64.rst

+and ``PT_TLS`` as the program header with type PT_TLS. ``PAD`` must be
+the smallest positive integer that satisfies the following congruence:
+
+``TP + TCB + PAD ≡ PT_TLS.p_vaddr (modulo PT_TLS.p_align)``


TP+TCB+PAD on the left could be confusing, as TCB is placed before TP. Perhaps mention the requirement of TP first (= 0 (modulo p_align)), then describe PAD and this formula.

I'll see if I can word it better. I've found it difficult to try and explain the formula intuitively.

MaskRay · 2025-02-21T06:25:40Z

sysvabi64/sysvabi64.rst

+Given that ``TP ≡ 0 (modulo PT_TLS.p_align)``. An expression
+for `PAD` is ``PAD = (PT_TLS.p_vaddr - TCB) mod PT_TLS.p_align``.
+
+A significant number of dynamic linkers use a different calculation


I believe that glibc Variant I handles p_vaddr!=0 (mod p_align) correctly. The bug (https://sourceware.org/bugzilla/show_bug.cgi?id=24606
) is for Variant II (x86 etc).

I have fixed FreeBSD rtld's Variant II in https://reviews.freebsd.org/D31538 . Its Variant I may or may not have the bug.

musl has been good since 1.1.23

Therefore, it's probably not "a significant number" but yeah p_vaddr=0 (mod p_align) is good for maximum compatibility

I found it difficult to be confident on the status of the various dynamic linkers. I can remove the significant part.

The glibc bug looks good for static TLS, it does mention in https://sourceware.org/bugzilla/show_bug.cgi?id=24606#c7 that dynamic TLS still needs p_vaddr to be 0 (modulo p_align).

MaskRay · 2025-02-21T06:30:37Z

sysvabi64/sysvabi64.rst

+    add  xn, tp, :tprel_hi12:var, lsl #12 // R_AARCH64_TLSLE_ADD_TPREL_HI12 var
+    ldr  xn, [xn, #:tprel_lo12_nc:var]    // R_AARCH64_TLSLE_LDST64_TPREL_LO12_NC var
+
+Static link time TLS Relaxations


Perhaps call this Optimization to be consistent with x86/ppc and "Relocation optimization" (ADRP) and leave the term "relocation relaxation" for RISC-V style section shrinking.

For TLS specifically I'd prefer to keep relaxation as that's what its been referred to in all the previous literature such as Drepper's ELF Handling for Thread Local Storage and the TLSDESC paper too. It should help people searching in the references.

I take the point that it ought to have been called optimization. I'll add a sentence to say that we're using relaxation as a term from the existing literature.

MaskRay · 2025-02-21T06:39:15Z

sysvabi64/sysvabi64.rst

+
+Undefined Weak Symbols
+
+An undefined weak symbol has the value 0. As the resolver function


This might be the glibc behavior, but musl doesn't have the special __dl_tlsdesc_undefweak. I think it's better to allow flexibility and require a particular behavior on undefined weak TLS.

This part is just an example of what can be done. I've written at the top of the section

The TLS resolver functions are not standardized by this ABI as they are internal to the dynamic linker

and

These examples are for illustrative purposes only

I'll see if there's anything I can do to state that there is no requirement to implement a specific resolver function.

smithp35

Thanks very much for the review.

I've updated based on this and some comments I received internally.

smithp35 · 2025-02-21T14:56:07Z

sysvabi64/sysvabi64.rst

+Given that ``TP ≡ 0 (modulo PT_TLS.p_align)``. An expression
+for `PAD` is ``PAD = (PT_TLS.p_vaddr - TCB) mod PT_TLS.p_align``.
+
+A significant number of dynamic linkers use a different calculation


I found it difficult to be confident on the status of the various dynamic linkers. I can remove the significant part.

The glibc bug looks good for static TLS, it does mention in https://sourceware.org/bugzilla/show_bug.cgi?id=24606#c7 that dynamic TLS still needs p_vaddr to be 0 (modulo p_align).

smithp35 · 2025-02-21T14:58:49Z

sysvabi64/sysvabi64.rst

+    add  xn, tp, :tprel_hi12:var, lsl #12 // R_AARCH64_TLSLE_ADD_TPREL_HI12 var
+    ldr  xn, [xn, #:tprel_lo12_nc:var]    // R_AARCH64_TLSLE_LDST64_TPREL_LO12_NC var
+
+Static link time TLS Relaxations


For TLS specifically I'd prefer to keep relaxation as that's what its been referred to in all the previous literature such as Drepper's ELF Handling for Thread Local Storage and the TLSDESC paper too. It should help people searching in the references.

I take the point that it ought to have been called optimization. I'll add a sentence to say that we're using relaxation as a term from the existing literature.

smithp35 · 2025-02-21T15:14:47Z

sysvabi64/sysvabi64.rst

+
+Undefined Weak Symbols
+
+An undefined weak symbol has the value 0. As the resolver function


This part is just an example of what can be done. I've written at the top of the section

The TLS resolver functions are not standardized by this ABI as they are internal to the dynamic linker

and

These examples are for illustrative purposes only

I'll see if there's anything I can do to state that there is no requirement to implement a specific resolver function.

smithp35 · 2025-02-21T15:29:01Z

sysvabi64/sysvabi64.rst

+and ``PT_TLS`` as the program header with type PT_TLS. ``PAD`` must be
+the smallest positive integer that satisfies the following congruence:
+
+``TP + TCB + PAD ≡ PT_TLS.p_vaddr (modulo PT_TLS.p_align)``


I'll see if I can word it better. I've found it difficult to try and explain the formula intuitively.

stuij · 2025-06-20T12:46:03Z

sysvabi64/sysvabi64.rst

+Rules governing thread local storage on AArch64
+-----------------------------------------------
+
+*  How to denote TLS in source programs.


For these How.. bullet points, I'd replace the period at the end with a :, to prime readers for the format.

stuij · 2025-06-20T12:49:15Z

sysvabi64/sysvabi64.rst

+
+This document and AAELF64_ are concerned with:
+
+*  How to relocate, statically and dynamically, with respect to symbols


In How to relocate.. I'm expecting the subject at some point. Especially with the subclause and the parentheses it becomes a bit of a confusing sentence that kinda peters out.

I took the format from the 32-bit ABI https://github.com/ARM-software/abi-aa/blob/main/addenda32/addenda32.rst#41introduction-to-thread-local-storage which has a similar list of bullet points followed by It is the last two bullet points that are the subject of this ABI.

What I'm trying to say is that the ABI only covers a small part of what is needed to support TLS. I may have lost that part. I'll try and reword.

stuij · 2025-06-20T12:52:32Z

sysvabi64/sysvabi64.rst

+constructed when the program is first loaded.
+
+For the purpose of addressing TLS, components, referred to as modules,
+of an application are identified using indexes. The module index for


components, referred to as modules, of an application -> components of an application, referred to as modules,

stuij · 2025-06-20T13:16:35Z

sysvabi64/sysvabi64.rst

+TLS variable is accessed the thread's generation count is compared
+with the global generation count which can be used to trigger updates
+of the DTV. The details are platform specific.
+


At this point it might be useful to give a small high-level example along the lines of for example when dynamic library x gets loaded, we add TLS block, add DTV entry.., just to drive off confusion.

I'll see what I can do. This part of TLS is, strictly speaking, outside the scope of the ABI, it is implementation details of a dynamic loader.

I think there's three parts:

dlopen, dlclose increment/decrement the global generation number independently of whether the module uses TLS.

tls_get_addr() compares the thread's generation counter against the global generation number and will resize/reallocate the dynamic thread vector if it is different.

Initially the dynamic thread vector entry is unallocated. On first use tls_get_addr will allocate and initialize the TLS.

stuij · 2025-06-20T13:45:26Z

sysvabi64/sysvabi64.rst

+
+AArch64 TLS SystemV design choices
+
+* AArch64 uses variant 1 TLS as described in ELFTLS_.


Perhaps mention ELFTLS when doing the TLS introduction as a for more in-depth info resource.

ACK. I'll mention that the introduction in the ABI is only sufficient to describe the terms used like Thread Control Block. A general introduction can be found in ELFTLS_

stuij · 2025-06-23T13:55:56Z

sysvabi64/sysvabi64.rst

+additional code to add ``x0`` to be added to ``tp``, this is not part
+of the ABI required TLSDESC code sequence.
+
+Small Code Model


In this section and the sections below I'd put semicolons after all the model headers: Small Code Model:, or use some kind of header format.

I'll use a semicolon. I think I'm already at the lowest level of header.

stuij · 2025-06-23T13:58:44Z

sysvabi64/sysvabi64.rst

+TLS variables from the thread pointer are static link time
+constants. The code sequences are the same for all code models.
+
+The instruction sequences below are not ABI. Using the instructions


are not ABI doesn't sound wholly English?

Perhaps: are not ABI specifications, but using..?

I've used are "not required by the ABI, but using".

stuij · 2025-06-23T14:02:01Z

sysvabi64/sysvabi64.rst

+Static link time TLS Relaxations
+--------------------------------
+
+Relaxation is a term used by the TLS literature such as ELFTLS_ to


Maybe move this general info up a bit. By now we have used the term twice in this patch.

Say the first 2 paragraphs until using the constrained model.

stuij · 2025-06-23T14:07:16Z

sysvabi64/sysvabi64.rst

+implemented.  Due to the restrictions on calling convention, the
+resolver routines must be written in assembly language.
+
+Static TLS Specialization


For these as well I would add a colon.

stuij · 2025-06-23T14:13:31Z

sysvabi64/sysvabi64.rst

+    movz    xn, :tprel_g1:var // R_AARCH64_TLSLE_MOVW_TPREL_G1 var
+    movk    xn, :tprel_g0:var // R_AARCH64_TLSLE_MOVW_TPREL_G0_NC var
+
+TLS Descriptors


I'd move these two more general sections above Code sequences for accessing TLS variables.

I'll take a look at this tomorrow.

The thread local storage chapter contains: * A description of Thread Local Storage based on addenda32 * The key design decisions of AArch64 TLS such as tls variant, tls dialect, TCB size. * The ABI required code sequence for TLSDESC that must be emitted exactly, as GNU ld requires it to be. * Sequences for the different code-models. * Relaxations for GD->IE, GD->LE and IE->LE. * Synchronization requirements for Lazy TLSDESC. With advice not to support it due to overhead of synchronization.

* Edits to split up the bullet points in How to denote TLS in source. * Changed program-own state to process-state as the thread-id may not be stored separately from the programs data. * Removed typically from some of the descriptions as the typically will almost always be the case for a sysvabi platform. * Linked alignment padding to the definition. * Provided a bit more information about generation counters.

* Rearranged formulas and used TCBsize to make it clearer. * Taken out "significant" from a significant number of dynamic linkers. * Give reason for using relaxation rather than optimization. * Clarify that there is no requirement to implement any TLSDESC resolver given in the sysvabi.

Change the input register in add xn, xn, :tprel_hi12:var, lsl ARM-software#12 to the thread pointer tp. We want to calculate the offset from the thread pointer so it needs to be an input of the add.

Document the decision in the GCC mailing list thread TLSDESC clobber ABI stability/futureproofness? https://gcc.gnu.org/legacy-ml/gcc/2018-10/msg00112.html TLSDESC resolver functions assume that any registers added by an extension are caller saved for a TLSDESC call. A brief summary: Dynamic TLS may be lazy allocated upon the first use of a TLSDESC resolver. This may involve calls to heap allocation functions provided by the user, which may use registers from extensions like SVE and SME. As the resolver function can't know what is saved it would have to save all SVE and SME state. This would be way more expensive than a caller save, and an older libc written prior to the introduction of the extension would be unaware of them so the caller has to do the save. * The SVE and SME state is already

smithp35

Thanks very much for the comments. I'll hopefully have a new patch tomorrow.

smithp35 · 2025-06-23T15:23:23Z

sysvabi64/sysvabi64.rst

+Rules governing thread local storage on AArch64
+-----------------------------------------------
+
+*  How to denote TLS in source programs.


smithp35 · 2025-06-23T15:27:40Z

sysvabi64/sysvabi64.rst

+
+This document and AAELF64_ are concerned with:
+
+*  How to relocate, statically and dynamically, with respect to symbols


I took the format from the 32-bit ABI https://github.com/ARM-software/abi-aa/blob/main/addenda32/addenda32.rst#41introduction-to-thread-local-storage which has a similar list of bullet points followed by It is the last two bullet points that are the subject of this ABI.

What I'm trying to say is that the ABI only covers a small part of what is needed to support TLS. I may have lost that part. I'll try and reword.

smithp35 · 2025-06-23T15:28:44Z

sysvabi64/sysvabi64.rst

+constructed when the program is first loaded.
+
+For the purpose of addressing TLS, components, referred to as modules,
+of an application are identified using indexes. The module index for


smithp35 · 2025-06-23T16:00:39Z

sysvabi64/sysvabi64.rst

+TLS variable is accessed the thread's generation count is compared
+with the global generation count which can be used to trigger updates
+of the DTV. The details are platform specific.
+


I'll see what I can do. This part of TLS is, strictly speaking, outside the scope of the ABI, it is implementation details of a dynamic loader.

I think there's three parts:

dlopen, dlclose increment/decrement the global generation number independently of whether the module uses TLS.

tls_get_addr() compares the thread's generation counter against the global generation number and will resize/reallocate the dynamic thread vector if it is different.

Initially the dynamic thread vector entry is unallocated. On first use tls_get_addr will allocate and initialize the TLS.

smithp35 · 2025-06-23T16:07:12Z

sysvabi64/sysvabi64.rst

+
+AArch64 TLS SystemV design choices
+
+* AArch64 uses variant 1 TLS as described in ELFTLS_.


ACK. I'll mention that the introduction in the ABI is only sufficient to describe the terms used like Thread Control Block. A general introduction can be found in ELFTLS_

smithp35 · 2025-06-23T17:13:07Z

sysvabi64/sysvabi64.rst

+additional code to add ``x0`` to be added to ``tp``, this is not part
+of the ABI required TLSDESC code sequence.
+
+Small Code Model


I'll use a semicolon. I think I'm already at the lowest level of header.

smithp35 · 2025-06-23T17:13:41Z

sysvabi64/sysvabi64.rst

+knows that the TLS variable is defined in the same module as the code
+that is accessing the variable. In this case the offset of the TLS
+variable from the start of the module's TLS block is a static link
+time constant. Instead of dynamically calculating the offset of the


smithp35 · 2025-06-23T17:16:54Z

sysvabi64/sysvabi64.rst

+TLS variables from the thread pointer are static link time
+constants. The code sequences are the same for all code models.
+
+The instruction sequences below are not ABI. Using the instructions


I've used are "not required by the ABI, but using".

smithp35 · 2025-06-23T17:19:29Z

sysvabi64/sysvabi64.rst

+    movz    xn, :tprel_g1:var // R_AARCH64_TLSLE_MOVW_TPREL_G1 var
+    movk    xn, :tprel_g0:var // R_AARCH64_TLSLE_MOVW_TPREL_G0_NC var
+
+TLS Descriptors


I'll take a look at this tomorrow.

smithp35 · 2025-06-23T17:20:02Z

sysvabi64/sysvabi64.rst

+implemented.  Due to the restrictions on calling convention, the
+resolver routines must be written in assembly language.
+
+Static TLS Specialization


Include a pseudo code description of __tls_get_addr with deferred TLS for dynamic modules.

Use integers modulo m to avoid excess use of (modulo m). Explain the congruence symbol. Put expression first so derivation is optional.

are used.

smithp35

Thanks for the review comments. I've made the following updates:

Simple NFC text changes.
Better description of deferred TLS and generation count.
Reworded the padding size derivation.
Moved paragraphs around to make the flow a bit easier.

Should be visible as 4 separate commits

smithp35 · 2025-06-27T12:25:00Z

sysvabi64/sysvabi64.rst

+Static link time TLS Relaxations
+--------------------------------
+
+Relaxation is a term used by the TLS literature such as ELFTLS_ to


smithp35 · 2025-06-27T12:35:23Z

sysvabi64/sysvabi64.rst

+    movz    xn, :tprel_g1:var // R_AARCH64_TLSLE_MOVW_TPREL_G1 var
+    movk    xn, :tprel_g0:var // R_AARCH64_TLSLE_MOVW_TPREL_G0_NC var
+
+TLS Descriptors


smithp35 force-pushed the sysvabitls branch from a261722 to 98ab4d7 Compare February 20, 2025 17:47

MaskRay reviewed Feb 21, 2025

View reviewed changes

smithp35 commented Feb 21, 2025

View reviewed changes

This was referenced Mar 18, 2025

[AArch64] Add an option to get the TLS pointer from software llvm/llvm-project#130932

Open

[rtabi64?] Add method for Software-based TLS Pointer Retrieval #316

Open

smithp35 mentioned this pull request Apr 23, 2025

[lld][ICF] Prevent merging two sections when they point to non-globals llvm/llvm-project#136641

Open

stuij reviewed Jun 23, 2025

View reviewed changes

smithp35 added 6 commits June 23, 2025 16:17

fix CI due to broken reference

3289370

[sysvabi64] Correct local exec instruction to use tp as input

02cd934

Change the input register in add xn, xn, :tprel_hi12:var, lsl ARM-software#12 to the thread pointer tp. We want to calculate the offset from the thread pointer so it needs to be an input of the add.

smithp35 commented Jun 23, 2025

View reviewed changes

smithp35 added 4 commits June 27, 2025 09:09

[sysvabi64] Simple fixes from Ties' review [NFC]

804889e

Update description of the generation counter and __tls_get_addr

638243e

Include a pseudo code description of __tls_get_addr with deferred TLS for dynamic modules.

Reword derivation of TLS padding to make it easier to understand

01cd716

Use integers modulo m to avoid excess use of (modulo m). Explain the congruence symbol. Put expression first so derivation is optional.

Move around some paragraphs to introduce some topics before they

e46f8bf

are used.

smithp35 force-pushed the sysvabitls branch from 1ed7570 to e46f8bf Compare June 27, 2025 12:43

smithp35 commented Jun 27, 2025

View reviewed changes


		Undefined Weak Symbols

		An undefined weak symbol has the value 0. As the resolver function


		This document and AAELF64_ are concerned with:

		* How to relocate, statically and dynamically, with respect to symbols


		AArch64 TLS SystemV design choices

		* AArch64 uses variant 1 TLS as described in ELFTLS_.

[sysvabi64] Add chapter on Thread Local Storage #311

Are you sure you want to change the base?

[sysvabi64] Add chapter on Thread Local Storage #311

Uh oh!

Conversation

smithp35 commented Feb 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smithp35 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smithp35 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment