Port runtime's native build infra

[am11]

First commit copies some files from runtime/eng/native/.

Second commit adapts the changes to synchronize the native build configurations with runtime (that has new platforms support, new compilers support, common compiler configs etc.)

[am11]

cc @hoyosjs, @jkoritzinsky, @janvorli

This brings the same common native infra to diagnostics, which we are using in runtime for coreclr,mono,libs,corehost.

diagnostics repo was once copied from dotnet/coreclr, so adaptation wasn't a big mystery (PR has fast-forward configs by a few generations).

eng/native remained the xcopy (without modifications) of current runtime's main branch. The config differential is applied in consumer scripts (outside eng/native). I had to also copy eng/empty.csproj and eng/versioning.targets to satisfy eng/native/build-commons.sh.

In the future, we can move eng/native to arcade/eng/common/native, where we already have similar common infra scripts and we are now using those here as well.

[am11]

@jkoritzinsky, in diagnostics repo, we need to define char sccsid[] as non-static, so it is visible outside the compilation unit (file) and satisfies this extern:

diagnostics/src/SOS/Strike/strike.cpp

Line 10899 in 66e2534

extern char sccsid[];

since I haven't changed anything in eng/native, I worked around it at the caller in cmake script. Perhaps we can add its support in version generator whether or not to omit static, or just omit static everywhere since we have __attribute__((used)) in version.c, so it remains in binary after stripping (e.g. strings libsos.so | grep '@(#)' shows the version even with non-static).

[am11]

@mikem8361, @hoyosjs, for some reason only 2019 centos7 image is able to run LLDB tests, changing

diagnostics/diagnostics.yml

Line 129 in 9dc72da

dockerImage: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-3e800f1-20190501005343

to 2021 one:

dockerImage: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-20210714125435-9b5bbc2

was failing lldb tests with that error.

[hoyosjs]

Yup, this is a large part of the reason this section is so stale - no one has actually had the time to go an fix the container. I've seen this error locally when using lldb attach in some scenarios is SYS_PTRACE is not available, but I doubt that's the problem here (same kernel used between the two images, and doubt they'll start them with different seccomp settings).

[am11]

In case of a container, the kernel used is from host machine, so it remains the same if we switch the container on same host as you have mentioned.

I think it could be the difference in how these containers are built. There had been many changes in dotnet docker build tool (which also builds prereq images).

[Thefrank]

Thank you so much for doing this! A while ago I submitted a few minor changes to get FreeBSD native building and put "make infrastructure more like other dotnet repos" at the very bottom of my list, it would have been more work than I had time for and judging by the size of this commits it looks like even more than I thought.

So again, my sanity, and likely other people thank you! :)

[am11]

@Thefrank, configurations tend to grow tentacles over time and generally speaking, they seldom get updated / look after for once things start to work. My proposition is to keep things flat, explicit and boring when it comes to configs nobody is thrilled about working on configs and doing anything smart/clever is a moot point. :)

Ideally we should move files from eng/native to arcade's eng/common/native to DRY out this stuff as much as possible. So for the next entry in .NET platform matrix (different OS, Arch, bitness etc.) we just define stuff in one place and bot will propagate the changes to consumer repos in the next auto-sync PR (note: changes in common configs are rare in first place and rarely urgent).

[mikem8361]

Can you fix/rebase the conflicts? Can you also push and test this with our internal official repo/build:

https://dev.azure.com/dnceng/internal/_git/dotnet-diagnostics
https://dev.azure.com/dnceng/internal/_build?definitionId=528

/cc: @hoyosjs

[am11]

Merged main to the branch.

Can you also push and test this with our internal official repo/build:

I don't have permissions to view the internal builds (let alone pressing the Run Pipeline button). :)

[jkoritzinsky]

I'll clone this branch and run it against the internal feed.

[jkoritzinsky]

@mikem8361 Official build link here: https://dev.azure.com/dnceng/internal/_build/results?buildId=1435124&view=results

[jkoritzinsky]

@am11 it looks like your updates to the eng/build.sh script lost the handling for the -dotnetruntimeversion flag (and possibly others). This is causing the official build to fail. Can you take a look and restore the command parsing/handling for the accidentally removed flags?

[am11]

@jkoritzinsky, thanks. Can you give me a build command which I can use to repro locally?

[jkoritzinsky]

I think some of the other parameters of the script are "internal-only" like feed tokens and such, so I don't think I'll be able to get you a full command to use that will actually succeed, but I'll try to give you one that will get you far enough to know the argument handling is fixed:

 /__w/1/s/eng/cibuild.sh -configuration Debug -architecture x64  -test /p:OfficialBuildId=20211021.1 -dotnetruntimeversion 'default' -dotnetruntimedownloadversion 'default' -runtimesourcefeed 'RuntimeFeedUrl' -runtimesourcefeedkey 'RuntimeFeedToken'

[mikem8361]

eng/build.sh had a lot of other special options that may be needed. I don't have enough time right now to review its (and all the rest) right now.

[mikem8361]

But I guess in the end, if the official build is successful then you covered everything.

[am11]

a lot of other special options

The "additional arguments" are handled in eng/native/build-commons.sh, these are the few top level-arguments which we don't want to treat as the additional argumetns.

[jkoritzinsky]

Updated official build: https://dev.azure.com/dnceng/internal/_build/results?buildId=1435252&view=results

[jkoritzinsky]

There's a failure in the packaging and signing step. Looks like the SOS.Package project is still looking for sos.pdb in a Windows_NT.x64.Release folder. @am11, did you change this repo to use the windows target folder for Windows like dotnet/runtime? If so, the SOS.Package.csproj project needs to be updated to look in the right place.

[am11]

From the build logs, it seems to be installing it in subdirectory PDB (matches coreclr current builds):

- main
+ pr

- -- Installing: D:/a/_work/1/s/artifacts/bin/Windows_NT.x86.Debug/./sos.pdb
+ -- Installing: D:/a/_work/1/s/artifacts/bin/Windows_NT.x64.Debug/./PDB/sos.pdb

I have updated the location in SOS.{Symbol.}Package.csproj files accordingly.

[am11]

@jkoritzinsky, can't say if it was enough for validation but one way to find out. 🙂

[jkoritzinsky]

I’ll run another official build in the morning and report back here with the results.

[jkoritzinsky]

Triggered an official build for the last commit: https://dev.azure.com/dnceng/internal/_build/results?buildId=1436462&view=results

[jkoritzinsky]

The official build passed.

[mikem8361]

Does everyone think it is ready to merge?

[hoyosjs]

I'll be taking a look at this after lunch. So far it looks ok.

[am11]

@hoyosjs, as mentioned earlier, it is an unmodified copy of runtime/eng/native/ to make sure we only consume it as-is. In the future, we can move common files to arcade (in the existing directory: eng/common/native) and clean them up from the consumer repositories.

[hoyosjs]

That's mostly fine with me. Just the fix in gen-buildsys.cmd and the utility scripts matter. The other ones are more of, when we do move stuff from runtime to arcade, those are things to layer and make specific.

[am11]

@hoyosjs, I have rebased with main. There are still two separate commits. It would be nice if they are merged without squashing so it's easier to distinguish xcopy changes from adaptation ones (helps during bisecting and blaming processes).

eng/empty.csproj and eng/versioning.targets are in adaptation commit. Perhaps we can move them to eng/native/ or eng/common as well in the future, so code in that dir only depends on arcade.

[mikem8361]

The "sscsid[]" variable in artifacts/obj/_version.c is static on a rebuild. On a clean build, it is fine and is not static. This causes a MacOS build failure and Linux sos will fail in the eeversion command. ``` // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. static char sccsid[] __attribute__((used)) = "@(#)Version N/A @Commit: 2b71b4a"; ```

[mikem8361]

The --skipmanaged and --skipnative build.sh options are gone so test.sh doesn't work anymore.

[mikem8361]

The test.sh (and probably test.cmd) need them to work like they did before.

[mikem8361]

It looks like the problem with _version.c is the one checked into eng/native/version

[mikem8361]

The problem with test.sh it uses "--test", "--skipnative" and "--skipmanaged" with the double dashes which seems to have been removed from eng/build.sh. I think the runtime still supports both single-dash and double-dash options.

[am11]

@mikem8361, abstraction-wise, ./build.sh corresponds to runtime/src/coreclr/build-runtime.sh and not runtime/build.sh. Double hyphen only works with runtime/build.sh because it adds an additional argument processing layer.

[am11]

Opened #2720 to address both concerns.