lldb stepping fails in macOS processes with a 32-bit/custom LDT installed

[mrpippy]

macOS 10.15 added support for processes to set a custom LDT, allowing a 64-bit process to set up and far jump to a 32-bit code segment. Wine (specifically CrossOver by CodeWeavers) uses this functionality to run 32-bit Windows EXEs.

However, lldb is not able to correctly step when in the non-64-bit code segment. Stepping resets %cs back to the 64-bit segment, generally causing a crash soon after (because 32-bit instructions are being executed as 64-bit).

I've attached a sample executable derived from an XNU test which tests the custom LDT functionality. It's pretty easy to reproduce the bug. Also note that although Rosetta 2 implements the custom LDT functionality, I wasn't able to reproduce this bug there. Run this on an Intel Mac:

lldb-32bit-ldt.zip

% sw_vers
ProductName:	macOS
ProductVersion:	12.5
BuildVersion:	21G72
% uname -a
Darwin Brendans-Mac-mini.local 21.6.0 Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:25 PDT 2022; root:xnu-8020.140.41~1/RELEASE_X86_64 x86_64

% cd lldb-32bit-ldt
% make
cc -arch x86_64 -g3 -o ldt ldt.c ldt_code32.s -Wall -Wextra -Wl,-pagezero_size,0x1000 -DDEBUG

% lldb ./ldt
(lldb) target create "./ldt"
Current executable set to '/Users/pip/lldb-32bit-ldt/ldt' (x86_64).
(lldb) version
lldb-1316.0.9.46
Apple Swift version 5.6.1 (swiftlang-5.6.0.323.66 clang-1316.0.20.12)
(lldb) break set -n code_32
Breakpoint 1: where = ldt`code_32, address = 0x000000000000b000
(lldb) run
Process 82565 launched: '/Users/pip/lldb-32bit-ldt/ldt' (x86_64)
32-bit code is at 0xb000
lowstack addr = 0x210ff0
size = 8192, szlimit = 32768
Mapping code @0xa000..0xb04b => 0xf0000..0xf104b
i386_get_ldt returned 3
i386_set_ldt returned 3
base 0x0 lim 0x0 type 0x0 dpl 0 present 0 opsz 0 granular 0
base 0x0 lim 0xfffff type 0x12 dpl 3 present 1 opsz 1 granular 1
base 0x0 lim 0x0 type 0x0 dpl 0 present 0 opsz 0 granular 0
base 0x0 lim 0xfffff type 0x1a dpl 3 present 1 opsz 1 granular 1
Updated gsbase for stack at 0x201000..0x211000 to 0x7000033880e0
[thread 0x700003388000] tsd base => 0x7000033880e0
Setting new GS base: 0x1008200
Process 82565 stopped
* thread #2, stop reason = EXC_BREAKPOINT (code=EXC_I386_BPT, subcode=0x0)
    frame #0: 0x00000000000f1001
->  0xf1001: movl   %esp, %ebp
    0xf1003: pushq  %rbx
    0xf1004: callq  0xf1009
    0xf1009: popq   %rbx
Target 0: (ldt) stopped.
(lldb) reg read $cs
      cs = 0x000000000000001f                           < ------ 32-bit code segment
(lldb) stepi
Process 82565 stopped
* thread #2, stop reason = instruction step into
    frame #0: 0x00000000000f1003
->  0xf1003: pushq  %rbx
    0xf1004: callq  0xf1009
    0xf1009: popq   %rbx
    0xf100a: subl   $0x8, %esp
Target 0: (ldt) stopped.
(lldb) reg read $cs
      cs = 0x000000000000002b                            < ------ after one step, %cs is now the 64-bit code segment
(lldb) reg read $rsp
     rsp = 0x0000000000210f98
(lldb) stepi
Process 82565 stopped
* thread #2, stop reason = instruction step into
    frame #0: 0x00000000000f1004
->  0xf1004: callq  0xf1009
    0xf1009: popq   %rbx
    0xf100a: subl   $0x8, %esp
    0xf100d: movl   0x1c(%rbp), %eax
Target 0: (ldt) stopped.
(lldb) reg read $rsp
     rsp = 0x0000000000210f90                            < ------ the processor really is in 64-bit mode now, $rsp decreased by 0x8

I'm not sure, but this bug might be related to macOS adding a 'full' thread state flavor (x86_THREAD_FULL_STATE64). I believe it's only (and I think must be) used for processes that have set a custom LDT. This flavor contains the normal x86_64 thread state, but also ds, es, ss, and GSbase. Maybe lldb needs to use the full thread state flavor when it's available to avoid %gs being reset?

_STRUCT_X86_THREAD_FULL_STATE64
{
        _STRUCT_X86_THREAD_STATE64      __ss64;
        __uint64_t                      __ds;
        __uint64_t                      __es;
        __uint64_t                      __ss;
        __uint64_t                      __gsbase;
};

[llvmbot]

@llvm/issue-subscribers-lldb

[mrpippy]

This is resolved by the above commit, but a note about the attached test program: even with the fix, when stepping through code_32 it stops with a crash on the ljmpl instruction. Even continuing after the initial break at the start of code_32 crashes at the same place.

I found that putting nops at the top of code_32 resolves the issue--my guess is that when setting the breakpoint, lldb replaces the first byte of code_32 with int (0xcc) or similar, and then has to emulate whatever instruction it replaced. lldb thinks this is x86_64 code, not x86, so the emulation is incorrect and that later causes the crash.

I don't know if there's a way to tell lldb what architecture a code region is, similar to the -A argument to disassemble. In theory lldb could read the LDT and know that the current cs is a 32-bit segment, but I don't know how possible that is.