Support for FreeBSD-arm64

[sec]

Hi,
Not to spam #14537 (as it's already quite big) :)
As full native build of either v6 and v7 is possible under FreeBSD-amd64, I went ahead and tried to make the same process work for ARM64.

After getting configure to success, I've tried:

1. Using Linux arm64 host to cross compile to FreeBSD (with custom rootfs created with libs, etc.)
   As it was used before, to cross compile from Linux-amd64 to Freebsd-x64, this fail at very early stage with:

[  1%] Built target System.Globalization.Native-Static
Scanning dependencies of target coreclrpal
[  1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/dec/state.c.o
[  1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
[  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
In file included from /root/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
/root/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
#include <type_traits>
         ^~~~~~~~~~~~~
[  1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/enc/backward_references.c.o
[  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
1 error generated.
[  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_time.c.o
make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2

Command I've used ROOTFS_DIR=/home/ubuntu/arm64 ./src/coreclr/build-runtime.sh -arm64 -debug clang14 -cross -os FreeBSD.
This does look like something missing in configure step, as those headers are present on rootfs_dir and host system:

root@instance-20220624-1221:/# find . -name type_traits
./home/ubuntu/arm64/usr/include/c++/v1/tr1/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/experimental/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/type_traits

./usr/include/c++/11/tr2/type_traits
./usr/include/c++/11/tr1/type_traits
./usr/include/c++/11/experimental/type_traits
./usr/include/c++/11/type_traits

./root/runtime/src/coreclr/inc/clr_std/type_traits

so I guess something during configure stage should be added to look in those directories?

2. Using FreeBSD arm64 host directly to compile
   Trying with both -cross and without, it fails with the same error:

[  4%] Building C object libs-native/System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_evp_pkey_ecdh.c.o
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:450:1: error: unknown type name 'fpsimd_context'
fpsimd_context* GetNativeSigSimdContext(native_context_t *mc)
^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:9: error: unknown type name 'fpsimd_context'
        fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:47: error: unknown type name 'fpsimd_context'
        fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
                                              ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:82: error: no member named '__reserved' in '__mcontext'
        fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
                                                                 ~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:458:30: error: use of undeclared identifier 'FPSIMD_MAGIC'
        if(fp->head.magic == FPSIMD_MAGIC)
                             ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:460:46: error: use of undeclared identifier 'fpsimd_context'
            _ASSERTE(fp->head.size >= sizeof(fpsimd_context));
                                             ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:461:69: error: no member named '__reserved' in '__mcontext'
            _ASSERTE(size + fp->head.size <= sizeof(mc->uc_mcontext.__reserved));
                                                    ~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/dbgmsg.h:362:35: note: expanded from macro '_ASSERTE'
#define _ASSERTE(expr) do { if (!(expr)) { ASSERT("Expression: " #expr "\n"); } } while(0)
                                  ^~~~
In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:28: error: use of undeclared identifier 'fpsimd_context'
    } while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
                           ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:70: error: no member named '__reserved' in '__mcontext'
    } while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
                                                     ~~~~~~~~~~~~~~~ ^
/usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:480:7: error: unknown type name 'fpsimd_context'
const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
      ^
10 errors generated.
[  4%] Linking C static library libSystem.Native.a
--- pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o] Error code 1

make[2]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
*** [all] Error code 6

make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug
1 error

make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug

Command used was the same as with ad. 1. This looks more complicated, as it have something to do with libunwind being used? No idea how to get over that :)

Including patches I've come up with.

If anyone expierenced with cross build could comment on that, to even confirm it's possible in the current state to make that work or would it require far more work?

If someone is interested in helping/testing/making this work I can provide access to either FreeBSD or Linux ARM64 box that I use.

runtime_freebsd_arm64.txt

[Thefrank]

OSX/BSD builds use the target system's libunwind:
from src/coreclr/pal/src/CMakeLists.txt

if(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)
     # On OSX and *BSD, we use the libunwind that's part of the OS
     set(CLR_CMAKE_USE_SYSTEM_LIBUNWIND 1)
endif(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)

which makes it add only include_directories(SYSTEM $ENV{ROOTFS_DIR}/usr/local/include)

[am11]

Yup, I think we just need to update the list of compatible context registers for freebsd-arm64 to make it work.

[sec]

If that would only make sense to me I would maybe help more, still learing on the fly. I found #40149 and doing quick change:

diff --git a/src/coreclr/pal/src/include/pal/context.h b/src/coreclr/pal/src/include/pal/context.h
index fec5139fd5e..2cc4941d653 100644
--- a/src/coreclr/pal/src/include/pal/context.h
+++ b/src/coreclr/pal/src/include/pal/context.h
@@ -407,7 +407,7 @@ inline void *FPREG_Xstate_Ymmh(const ucontext_t *uc)

 #if defined(HOST_ARM64)

-#ifndef TARGET_OSX
+#if !defined(TARGET_OSX) && !defined(TARGET_FREEBSD)

 #define MCREG_X0(mc)      ((mc).regs[0])
 #define MCREG_X1(mc)      ((mc).regs[1])
@@ -482,7 +482,7 @@ const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
     return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
 }

-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX

 #define MCREG_X0(mc)      ((mc)->__ss.__x[0])
 #define MCREG_X1(mc)      ((mc)->__ss.__x[1])

native compile fail with:

In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:177:2: error: unsupported architecture
#error unsupported architecture
 ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:183:5: error: use of undeclared identifier 'ASSIGN_UNWIND_REGS'
    ASSIGN_UNWIND_REGS
    ^
2 errors generated.
--- pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o] Error code 1

Is this the context registers that needs to be updated (looked at the code and there's no ARM64 for any of the targets)?

[janvorli]

@sec, what are the detected features (the HAVE_xxx) for the FreeBSD arm64 build? When you do a clean build, it lists them to the console. From those, I should be able to tell where to update the context registers extraction.

[sec]

@janvorli There you go:

-- Performing Test COMPILER_SUPPORTS_F_STACK_PROTECTOR_STRONG - Success
-- Performing Test COMPILER_SUPPORTS_W_IMPLICIT_FALLTHROUGH - Success
-- Performing Test HAVE_LTO - Failed
-- Performing Test COMPILER_SUPPORTS_F_ALIGNED_NEW - Success
-- Performing Test HAVE_DIRENT_D_TYPE - Success
-- Performing Test C_SUPPORTS_WUNGUARDED_AVAILABILITY - Success
-- Performing Test HAVE_IN_PKTINFO - Failed
-- Performing Test HAVE_IP_MREQN - Success
-- Performing Test HAVE_STATFS_VFS - Failed
-- Performing Test HAVE_STATFS_MOUNT - Success
-- Performing Test HAVE_FLOCK64 - Failed
-- Performing Test HAVE_IFADDRS - Success
-- Performing Test HAVE_UTSNAME_DOMAINNAME - Failed
-- Performing Test HAVE_STAT_BIRTHTIME - Success
-- Performing Test HAVE_STAT_FLAGS - Success
-- Performing Test HAVE_STAT_TIMESPEC - Success
-- Performing Test HAVE_STAT_TIM - Success
-- Performing Test HAVE_STAT_NSEC - Success
-- Performing Test HAVE_DIRENT_NAME_LEN - Success
-- Performing Test HAVE_STATFS_FSTYPENAME - Success
-- Performing Test HAVE_STATVFS_FSTYPENAME - Failed
-- Performing Test HAVE_GNU_STRERROR_R - Failed
-- Performing Test HAVE_READDIR_R - Success
-- Performing Test KEVENT_HAS_VOID_UDATA - Success
-- Performing Test HAVE_FDS_BITS - Success
-- Performing Test HAVE_PRIVATE_FDS_BITS - Success
-- Performing Test HAVE_SENDFILE_4 - Failed
-- Performing Test HAVE_SENDFILE_6 - Failed
-- Performing Test HAVE_SENDFILE_7 - Success
-- Performing Test HAVE_GETNAMEINFO_SIGNED_FLAGS - Success
-- Performing Test HAVE_SHM_OPEN_THAT_WORKS_WELL_ENOUGH_WITH_MMAP - Success
-- Performing Test HAVE_CLOCK_MONOTONIC - Success
-- Performing Test HAVE_CLOCK_REALTIME - Success
-- Performing Test BIND_ADDRLEN_UNSIGNED - Success
-- Performing Test IPV6MR_INTERFACE_UNSIGNED - Success
-- Performing Test INOTIFY_RM_WATCH_WD_UNSIGNED - Failed
-- Performing Test HAVE_MKSTEMPS - Success
-- Performing Test HAVE_MKSTEMP - Success
-- Performing Test HAVE_NETINET_TCP_VAR_H - Success
-- Performing Test HAVE_NETINET_UDP_VAR_H - Success
-- Performing Test HAVE_NETINET_IP_VAR_H - Success
-- Performing Test HAVE_NETINET_ICMP_VAR_H - Success
-- Performing Test HAVE_TCP_H_TCPSTATE_ENUM - Failed
-- Performing Test HAVE_GETDOMAINNAME_SIZET - Failed
-- Performing Test HAVE_IN_EXCL_UNLINK - Success
-- Performing Test HAVE_TCP_H_TCP_KEEPALIVE - Failed
-- Performing Test HAVE_BUILTIN_MUL_OVERFLOW - Success
-- Performing Test HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS - Success
-- Performing Test HAVE_VM_FLAGS_SUPERPAGE_SIZE_ANY - Failed
-- Performing Test HAVE_MAP_HUGETLB - Failed
-- Performing Test HAVE_LTTNG_TRACEPOINT_H - Failed
-- Performing Test HAVE_CPUSET_T - Success
-- Performing Test HAVE_TM_GMTOFF - Success
-- Performing Test HAVE_GREGSET_T - Failed
-- Performing Test HAVE___GREGSET_T - Failed
-- Performing Test HAVE_FPSTATE_GLIBC_RESERVED1 - Failed
-- Performing Test HAVE_SYSINFO_WITH_MEM_UNIT - Failed
-- Performing Test HAVE_FPREGS_WITH_CW - Failed
-- Performing Test REALPATH_SUPPORTS_NONEXISTENT_FILES - Failed
-- Performing Test SSCANF_SUPPORT_ll - Success
-- Performing Test SSCANF_CANNOT_HANDLE_MISSING_EXPONENT - Failed
-- Performing Test HAVE_LARGE_SNPRINTF_SUPPORT - Success
-- Performing Test HAVE_BROKEN_FIFO_SELECT - Failed
-- Performing Test HAVE_BROKEN_FIFO_KEVENT - Failed
-- Performing Test HAVE_SCHED_GET_PRIORITY - Success
-- Performing Test HAVE_WORKING_GETTIMEOFDAY - Success
-- Performing Test HAVE_WORKING_CLOCK_GETTIME - Success
-- Performing Test HAVE_CLOCK_MONOTONIC_COARSE - Success
-- Performing Test HAVE_CLOCK_THREAD_CPUTIME - Success
-- Performing Test MMAP_ANON_IGNORES_PROTECTION - Failed
-- Performing Test ONE_SHARED_MAPPING_PER_FILEREGION_PER_PROCESS - Failed
-- Performing Test PTHREAD_CREATE_MODIFIES_ERRNO - Failed
-- Performing Test SEM_INIT_MODIFIES_ERRNO - Failed
-- Performing Test HAVE_PROCFS_CTL - Failed
-- Performing Test HAVE_PROCFS_MAPS - Failed
-- Performing Test HAVE_PROCFS_STAT - Failed
-- Performing Test HAVE_PROCFS_STATUS - Success
-- Performing Test HAVE_COMPATIBLE_ACOS - Success
-- Performing Test HAVE_COMPATIBLE_ASIN - Success
-- Performing Test HAVE_COMPATIBLE_POW - Success
-- Performing Test HAVE_VALID_NEGATIVE_INF_POW - Success
-- Performing Test HAVE_VALID_POSITIVE_INF_POW - Success
-- Performing Test HAVE_COMPATIBLE_ATAN2 - Success
-- Performing Test HAVE_COMPATIBLE_EXP - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGB0 - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGBNAN - Success
-- Performing Test HAVE_COMPATIBLE_LOG - Success
-- Performing Test HAVE_COMPATIBLE_LOG10 - Success
-- Performing Test UNGETC_NOT_RETURN_EOF - Failed
-- Performing Test HAS_POSIX_SEMAPHORES - Success
-- Performing Test GETPWUID_R_SETS_ERRNO - Failed
-- Performing Test FILE_OPS_CHECK_FERROR_OF_PREVIOUS_CALL - Failed
-- Performing Test UNWIND_CONTEXT_IS_UCONTEXT_T - Success
-- Performing Test HAVE_XSWDEV - Success
-- Performing Test HAVE_XSW_USAGE - Failed
-- Performing Test HAVE_PUBLIC_XSTATE_STRUCT - Failed
-- Performing Test HAVE_PR_SET_PTRACER - Failed
-- Performing Test HAVE_FULLY_FEATURED_PTHREAD_MUTEXES - Success
-- Performing Test HAVE_PTHREAD_THREADID_NP - Failed
-- Performing Test HAVE_PTHREAD_GETTHREADID_NP - Success

edit: also attached full log from build attempt.
build_log.txt

[janvorli]

It seems to me that your last change was correct. But you also need to add #elif (defined(HOST_UNIX) && defined(HOST_ARM64)) before the #else with #error that fires. In that #elif, you'd add:

    ASSIGN_REG(Pc) \
    ASSIGN_REG(Sp) \
    ASSIGN_REG(Fp) \
    ASSIGN_REG(Lr) \
    ASSIGN_REG(X19) \
    ASSIGN_REG(X20) \
    ASSIGN_REG(X21) \
    ASSIGN_REG(X22) \
    ASSIGN_REG(X23) \
    ASSIGN_REG(X24) \
    ASSIGN_REG(X25) \
    ASSIGN_REG(X26) \
    ASSIGN_REG(X27) \
    ASSIGN_REG(X28)

I hope I got it right.

[sec]

@janvorli Tried that, it failed with bunch of errors for every register:

In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:190:2: error: use of undeclared identifier 'MCREG_X21'
        ASSIGN_UNWIND_REGS
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:178:2: note: expanded from macro 'ASSIGN_UNWIND_REGS'
        ASSIGN_REG(X21) \
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:189:25: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(unwContext->uc_mcontext) = winContext->reg;
                        ^
<scratch space>:95:1: note: expanded from here
MCREG_X21
^

Leaving this empty, will go on but then fail again with similar errors. If I'm looking at this correctly, I think it's missing some #def for HOST_ARM64 and TARGET_FREEBSD inside src/coreclr/pal/src/include/pal/context.h with those registers?

And also there's again error: unknown type name 'fpsimd_context' - which FreeBSD doesn't have, but if I add || defined(TARGET_FREEBSD) next to TARGET_OSX those errors seems to go away (don't know if that will work in the end though, as I going blind in here. Also the undefined registers errors are there, so my bet is that fixing those first would be real fix).

I wodner, if there's libunwind used (which state FreeBSD is supported under x64 and arm64) - shouldn't this be used (if I assume this code here is about that).

As I saw @sdmaclea did the OSX part of changes - maybe some hints? :)

Just to make sure, this is native build under ARM64 FreeBSD host.

[janvorli]

Ah, I haven't noticed that your change around line 490 was like this:

-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX

That way, you have eliminated both definitions of the MCREG_xxx for arm64. What does the ucontext_t struct look like on FreeBSD arm64? We'll either need to define the MCREG_xxx for the FreeBSD separately or reuse the one for Linux / OSX if it happens to be the same.

[janvorli]

Ah, found it in FreeBSD sources. So it is different and we will need to have another #elif TARGET_FREEBSD there. It seems that the definitions should look like:

#define MCREG_X21(mc) (mc.mc_gpregs.gp_x[21])
#define MCREG_X22(mc) (mc.mc_gpregs.gp_x[22])
#define MCREG_X23(mc) (mc.mc_gpregs.gp_x[23])
#define MCREG_X24(mc) (mc.mc_gpregs.gp_x[24])
#define MCREG_X25(mc) (mc.mc_gpregs.gp_x[25])
#define MCREG_X26(mc) (mc.mc_gpregs.gp_x[26])
#define MCREG_X27(mc) (mc.mc_gpregs.gp_x[27])
#define MCREG_X28(mc) (mc.mc_gpregs.gp_x[28])
#define MCREG_Fp(mc)      (mc.mc_gpregs.gp_fp)
#define MCREG_Lr(mc)      (mc.mc_gpregs.gp_lr)
#define MCREG_Sp(mc)      (mc.mc_gpregs.gp_sp)
#define MCREG_Pc(mc)      (mc.mc_gpregs.gp_pc)

As for the floating point access we will also need an extra code here. Take the following with a grain of salt, it might not be exactly right or might need extra change at another place too:

inline
struct fpregs* GetNativeSigSimdContext(native_context_t *mc)
{
    return &(mc->mc_fpregs);
}

inline
const struct fpregs* GetConstNativeSigSimdContext(const native_context_t *mc)
{
    return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
}

[janvorli]

Btw, this is based on https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/arm64/include/ucontext.h#L53