From 79ea9d13ac1c5a475c2f328924d9b40e0d2893ca Mon Sep 17 00:00:00 2001 From: Dan Moseley Date: Fri, 10 Jan 2020 16:24:38 -0800 Subject: [PATCH] Update workflow readme (#1512) * Update workflow readme * CR feedback and more detail * Add daily workflow for libraries dev * rebuild->build * More updates * typo: * more * More edits * testing * space: --- docs/README.md | 2 + docs/workflow/README.md | 36 ++++- docs/workflow/building/coreclr/README.md | 28 ++-- docs/workflow/building/libraries/README.md | 167 +++++++++++++++------ 4 files changed, 164 insertions(+), 69 deletions(-) diff --git a/docs/README.md b/docs/README.md index 29a8e5596cb933..10ba8feb986638 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,6 +17,8 @@ Getting Started Workflow (Building, testing, etc.) =============== +If you want to contribute a code change to this repo, start here. + - [Workflow Instructions](workflow/README.md) Design Docs diff --git a/docs/workflow/README.md b/docs/workflow/README.md index 288201f2e75401..046bb6b5451ab3 100644 --- a/docs/workflow/README.md +++ b/docs/workflow/README.md @@ -2,6 +2,8 @@ The repo can be built for the following platforms, using the provided setup and the following instructions. Before attempting to clone or build, please check these requirements. +## Build Requirements + | Chip | Windows | Linux | macOS | FreeBSD | | :---- | :------: | :------: | :------: | :------: | | x64 | ✔ | ✔ | ✔ | ✔ | @@ -10,22 +12,48 @@ The repo can be built for the following platforms, using the provided setup and | ARM64 | ✔ | ✔ | | | | | [Requirements](requirements/windows-requirements.md) | [Requirements](requirements/linux-requirements.md) | [Requirements](requirements/macos-requirements.md) | +Before proceeding further, please click on the link above that matches your machine and ensure you have installed all the pre-requisites for build to work. + ## Concepts -The runtime repo can be built from a regular, non-admin command prompt. The repository currently consists of three different partitions: the runtime (coreclr), libraries and the installer. For every partition there's a helper script available in the root (e.g. libraries.cmd/sh). The root build script (build.cmd/sh) should be used to build the entire repository. +The runtime repo can be built from a regular, non-admin command prompt. The repository currently consists of three different major parts: the runtime (a.k.a. coreclr), the libraries and the installer. To build everything you use the root build script (build.cmd/sh), and you add the `-subsetCategory` flag to build just one part. For information about the different options available, supply the argument `-help|-h` when invoking the build script: ``` -libraries -h +build -h ``` On Unix, arguments can be passed in with a single `-` or double hyphen `--`. -## Workflows +## Configurations + +You may need to build the tree in a combination of configurations. This section explains why. + +A quick reminder of some concepts -- see the [glossary](../project/glossary.md) for more on these: + +* **Debug configuration** -- Non-optimized code. Asserts are enabled. +* **Checked configuration** -- Optimized code. Asserts are enabled. Only relevant to CoreCLR. +* **Release configuration** -- Optimized code. Asserts are disabled. Runs at full speed, and suitable for performance profiling. Somewhat poorer debugging experience. -For instructions on how to build, debug, test, etc. please visit the instructions in the workflow sub-folders. +When we talk about mixing configurations, we're discussing three sub-components: + +* **CoreCLR** (often referred to as the runtime, most code under src/coreclr) -- this is the execution engine for managed code. It is written in C/C++. When built in a debug configuration, it is easier to debug into it, but it executes managed code more slowly - so slowly it will take a long time to run the managed code unit tests +* **CoreLib** (also known as System.Private.CoreLib - code under src/coreclr/System.Private.CoreLib) -- this is the lowest level managed library. It has a special relationship with the runtime -- it must be in the matching configuration, e.g., if the runtime you are using was built in a debug configuration, this must be in a debug configuration +* **All other libraries** (most code under src/libraries) -- the bulk of the libraries are oblivious to the configuration that CoreCLR/CoreLib were built in. Like most code they are most debuggable when built in a debug configuration, and, happily, they still run sufficiently fast in that configuration that it's acceptable for development work. + +### What does this mean for me? + +At this point you probably know what you are planning to work on first: the runtime or libraries. + +* if you're working in CoreCLR proper, you may want to build everything in the debug configuration, depending on how comfortable you are debugging optimized native code +* if you're working in most libraries, you will want to use debug libraries with release CoreCLR and CoreLib, because the tests will run faster. +* if you're working in CoreLib - you probably want to try to get the job done with release CoreCLR and CoreLib, and fall back to debug if you need to. The [Building libraries](building/libraries/README.md) document explains how you'll do this. + +Now you know about configurations and how we use them, you will want to read how to build what you plan to work on. Pick one of these: - [Building coreclr](building/coreclr/README.md) - [Building libraries](building/libraries/README.md) +After that, here's information about how to run tests: + - [Testing coreclr](testing/coreclr/testing.md) - [Testing libraries](testing/libraries/testing.md) \ No newline at end of file diff --git a/docs/workflow/building/coreclr/README.md b/docs/workflow/building/coreclr/README.md index f8d52a74caeff0..56d48173077cbd 100644 --- a/docs/workflow/building/coreclr/README.md +++ b/docs/workflow/building/coreclr/README.md @@ -1,34 +1,32 @@ # Building -Once all the necessary tools are in place, building is trivial. Simply run coreclr.cmd/sh script that lives in the repository root. +To build just CoreCLR, use the `--subsetCategory` flag to the `build.sh` (or `build.cmd`) at the repo root: -```bat - .\coreclr.cmd - - [Lots of build spew] +``` +./build.sh --subsetCategory coreclr +``` +or on Windows, +``` +build.cmd --subsetCategory coreclr +``` - Product binaries are available at C:\git\runtime\artifacts\bin\coreclr\Windows_NT.x64.debug - Test binaries are available at C:\git\runtime\artifacts\tests\coreclr\Windows_NT.x64.debug +By default, build generates a 'debug' build type, that includes asserts and is easier for some people to debug. If you want to make performance measurements, or just want tests to execute more quickly, you can also build the 'release' version which does not have these checks by adding the flag `-configuration release` (or `-c release`), for example +``` +./build.sh --subsetCategory coreclr -configuration release ``` -As shown above, the product will be placed in +This will produce outputs as follows: - Product binaries will be dropped in `artifacts\bin\coreclr\..` folder. - A NuGet package, Microsoft.Dotnet.CoreCLR, will be created under `artifacts\bin\coreclr\..\.nuget` folder. - Test binaries will be dropped under `artifacts\tests\coreclr\..` folder. -By default, build generates a 'Debug' build type, that has extra checking (assert) compiled into it. You can -also build the 'release' version which does not have these checks. The build places logs in `artifacts\log` and these are useful when the build fails. The build places all of its output in the `artifacts\obj\coreclr` directory, so if you remove that directory you can force a full rebuild. -The build has a number of options that you can learn about using build -?. Some of the more important options are - - * -skiptests - don't build the tests. This can shorten build times quite a bit, but means you can't run tests. - * -release - build the 'Release' build type that does not have extra development-time checking compiled in. - You want this if you are going to do performance testing on your build. +The build has a number of options that you can learn about using `build -?`. In particular `-skiptests` skips building the tests, which makes the build quicker. See [Running Tests](../../testing/coreclr/testing.md) for instructions on running the tests. diff --git a/docs/workflow/building/libraries/README.md b/docs/workflow/building/libraries/README.md index d21189aaaaa5f5..641271a32b0940 100644 --- a/docs/workflow/building/libraries/README.md +++ b/docs/workflow/building/libraries/README.md @@ -1,9 +1,85 @@ # Build -The libraries build has two logical components, the native build which produces the "shims" (which provide a stable interface between the OS and managed code) and -the managed build which produces the MSIL code and NuGet packages that make up CoreFX. +## Quick Start -Calling the script `libraries` attempts to build both the native and managed code. +Here is one example of a daily workflow for a developer working mainly on the libraries, in this case using Windows: + +``` +:: From root in the morning: +git clean -xdf +git pull upstream master & git push origin master +cd src\coreclr +build -release -skiptests +cd ..\..\ +build -subsetCategory libraries /p:CoreCLRConfiguration=Release + +:: The above you may only perform once in a day, or when +:: you pull down significant new changes. + +:: Switch to working on a given library (RegularExpressions in this case) +cd src\libraries\System.Text.RegularExpressions + +:: If you use Visual Studio, you might open System.Text.RegularExpressions.sln here. + +:: Change to test directory +cd tests + +:: Then inner loop build / test +:: (If using Visual Studio, you might run tests inside it instead) +pushd ..\src & dotnet msbuild & popd & dotnet msbuild /t:buildandtest +``` + +The instructions for Linux are essentially the same: + +``` +# From root in the morning: +git clean -xdf +git pull upstream master & git push origin master +cd src/coreclr +build -release -skiptests +cd ../../ +./build.sh -subsetCategory libraries /p:CoreCLRConfiguration=Release + +# The above you may only perform once in a day, or when +# you pull down significant new changes. + +# Switch to working on a given library (RegularExpressions in this case) +cd src/libraries/System.Text.RegularExpressions + +# Change to test directory +cd tests + +# Then inner loop build / test: +pushd ../src & dotnet msbuild & popd & dotnet msbuild /t:buildandtest +``` + +The steps above may be all you need to know to make a change. Want more details about what this means? Read on. + +## Building everything + +This document explains how to work on libraries. In order to work on library projects or run library tests it is necessary to have built CoreCLR (aka, the "runtime") to give the libraries something to run on. You should normally build CoreCLR in release configuration and libraries in debug configuration. If you haven't already done so, please read [this document](../../README.md#Configurations) to understand configurations. + +These example commands will build a release CoreCLR (and CoreLib) and debug libraries: + +For Linux: +``` +src/coreclr/build.sh -release -skiptests +./build.sh -subsetCategory libraries /p:CoreCLRConfiguration=Release +``` + +For Windows: +``` +src\coreclr\build.cmd -release -skiptests +build.cmd -subsetCategory libraries /p:CoreCLRConfiguration=Release +``` + +Detailed information about building and testing CoreCLR and the libraries is in the documents linked below. + +### More details if you need them + +The above commands will give you libraries in "debug" configuration (the default) using a runtime in "release" configuration which hopefully you built earlier. + +The libraries build has two logical components, the native build which produces the "shims" (which provide a stable interface between the OS and managed code) and the managed build which produces the MSIL code and NuGet packages that make up CoreFX. The commands above will build both. The build configurations are generally defaulted based on where you are building (i.e. which OS or which architecture) but we have a few shortcuts for the individual properties that can be passed to the build scripts: @@ -14,99 +90,80 @@ The build configurations are generally defaulted based on where you are building For more details on the build configurations see [project-guidelines](../../../coding-guidelines/project-guidelines.md#build-pivots). -**Note**: Before working on individual projects or test projects you **must** run `build` from the root once before beginning that work. It is also a good idea to run `build` whenever you pull a large set of unknown changes into your branch. If you invoke the build script without any actions, the default action chain `-restore -build` is executed. This means that restore and build are not implicit when invoking other actions! +If you invoke the build script without any actions, the default action chain `-restore -build` is executed. You can chain multiple actions together (e.g., `-restore -build -buildtests`) and they will execute in the appropriate order. Note that if you specify actions like `-build` explicitly, you likely need to explicitly add `-restore` as well. -**Note:** You can chain multiple actions together but the order of execution is fixed and does not relate to the position of the argument in the command. - -The most common workflow for developers is to call `libraries` from the root once and then go and work on the individual library that you are trying to make changes for. - -By default build only builds the product libraries and none of the tests. If you want to build the tests you can call `libraries -buildtests`. If you want to run the tests you can call `build -test`. To build and run the tests combine both arguments: `libraries -buildtests -test`. To build both the product libraries and the test libraries pass `libraries -build -buildtests` to the command line. - -If you invoke the build script without any argument the default arguments will be executed `-restore -build`. Note that -restore and -build are only implicit if no actions are passed in. +By default build only builds the product libraries and none of the tests. If you want to build the tests you can add the flag `-buildtests`. If you want to run the tests you can add the flag `-test`. To build and run the tests combine both arguments: `-buildtests -test`. To specify just the libraries, use `-subcategory libraries`. **Examples** - Building in release mode for platform x64 (restore and build are implicit here as no actions are passed in) ``` -libraries -c Release -arch x64 +./build.sh -subsetCategory libraries -c Release -arch x64 ``` - Building the src assemblies and build and run tests (running all tests takes a considerable amount of time!) ``` -libraries -restore -build -buildtests -test +./build.sh -subsetCategory libraries -restore -build -buildtests -test ``` - Building for different target frameworks (restore and build are implicit again as no action is passed in) ``` -libraries -framework netcoreapp -libraries -framework netfx +./build.sh -subsetCategory libraries -framework netcoreapp +./build.sh -subsetCategory libraries -framework netfx ``` - Build only managed components and skip the native build ``` -libraries /p:BuildNative=false +./build.sh -subsetCategory libraries /p:BuildNative=false ``` - Clean the entire solution ``` -libraries -clean +./build.sh -subsetCategory libraries -clean ``` -### Build Native -The native build produces shims over libc, openssl, gssapi, and libz. -The build system uses CMake to generate Makefiles using clang. -The build also uses git for generating some version information. -The native component should be buildable on any system. +For Windows, replace `./build.sh` with `build.cmd`. + +### How to building native components only + +The libraries build contains some native code. This includes shims over libc, openssl, gssapi, and zlib. The build system uses CMake to generate Makefiles using clang. The build also uses git for generating some version information. **Examples** - Building in debug mode for platform x64 ``` -./src/libraries/Native/build-native debug x64 +./src/libraries/Native/build-native.sh debug x64 ``` - The following example shows how you would do an arm cross-compile build. ``` -./src/libraries/Native/build-native debug arm cross verbose +./src/libraries/Native/build-native.sh debug arm cross verbose ``` -For more information about extra parameters take a look at the scripts `build-native` under src/Native. - -### Building individual libraries +For Windows, replace `build-native.sh` with `build-native.cmd`. -**Note**: Before working on individual projects or test projects you **must** run `libraries` from the root once before beginning that work. It is also a good idea to run `libraries` whenever you pull a large set of unknown changes into your branch. +## Building individual libraries -Similar to building the entire repo with build.cmd/sh in the root you can build projects based on our directory structure by passing in the directory. We also support -shortcuts for libraries so you can omit the root src folder from the path. When given a directory we will build all projects that we find recursively under that directory. +Similar to building the entire repo with `build.cmd` or `build.sh` in the root you can build projects based on our directory structure by passing in the directory. We also support shortcuts for libraries so you can omit the root src folder from the path. When given a directory we will build all projects that we find recursively under that directory. Some examples may help here. **Examples** -- Build all projects for a given library (ex: System.Collections) including running the tests -``` -libraries System.Collections -``` -or -``` -libraries src\libraries\System.Collections -``` -or +- Build all projects for a given library (e.g.: System.Collections) including running the tests + ``` -cd src\libraries\System.Collections -..\..\libraries . + ./build.sh -subsetCategory libraries src/libraries/System.Collections ``` - Build just the tests for a library project. ``` -libraries src\libraries\System.Collections\tests + ./build.sh -subsetCategory libraries src/libraries/System.Collections/tests ``` - All the options listed above like framework and configuration are also supported (note they must be after the directory) ``` -libraries System.Collections -f netfx -c Release + ./build.sh -subsetCategory libraries System.Collections -f netfx -c Release ``` -### Building individual projects - -You can either use `dotnet msbuild` or `msbuild`, depending on which is in your path. As `dotnet msbuild` works on all supported environments (i.e. Unix) we will use it throughout this guide. +As an alternative, once you are iterating on specific libraries, you can either use `dotnet msbuild` (not `dotnet build` -- that will run a restore as well) or `msbuild`, depending on which is in your path. As `dotnet msbuild` works on both Unix and Windows we will use it throughout this guide. Under the src directory is a set of directories, each of which represents a particular assembly in CoreFX. See Library Project Guidelines section under [project-guidelines](../../../coding-guidelines/project-guidelines.md) for more details about the structure. @@ -133,7 +190,7 @@ dotnet msbuild System.Net.NetworkInformation.csproj /p:OSGroup=Linux dotnet msbuild System.Net.NetworkInformation.csproj /p:ConfigurationGroup=Release ``` -To build for all supported configurations you can use the `BuildAll` and `RebuildAll` tasks: +To build for all supported configurations you can use the `BuildAll` and `RebuildAll` targets: ``` dotnet msbuild System.Net.NetworkInformation.csproj /t:RebuildAll @@ -142,15 +199,25 @@ dotnet msbuild System.Net.NetworkInformation.csproj /t:RebuildAll ### Building all for other OSes By default, building from the root will only build the libraries for the OS you are running on. One can -build for another OS by specifying `libraries -os [value]`. +build for another OS by specifying `./build.sh -subsetCategory libraries -os [value]`. Note that you cannot generally build native components for another OS but you can for managed components so if you need to do that you can do it at the individual project level or build all via passing `/p:BuildNative=false`. ### Building in Release or Debug By default, building from the root or within a project will build the libraries in Debug mode. -One can build in Debug or Release mode from the root by doing `libraries -c Release` or `libraries -c Debug` or when building a project by specifying `/p:ConfigurationGroup=[Debug|Release]` after the `dotnet msbuild` command. +One can build in Debug or Release mode from the root by doing `./build.sh -subsetCategory libraries -c Release` or `./build.sh -subsetCategory libraries -c Debug` or when building a project by specifying `/p:ConfigurationGroup=[Debug|Release]` after the `dotnet msbuild` command. ### Building other Architectures -One can build 32- or 64-bit binaries or for any architecture by specifying in the root `libraries -arch [value]` or in a project `/p:ArchGroup=[value]` after the `dotnet msbuild` command. +One can build 32- or 64-bit binaries or for any architecture by specifying in the root `./build.sh -subsetCategory libraries -arch [value]` or in a project `/p:ArchGroup=[value]` after the `dotnet msbuild` command. + +## Working in Visual Studio + +If you are working on Windows, and use Visual Studio, you can open individual libraries projects into it. From within Visual Studio you can then build, debug, and run tests. + +## Running tests + +For more details about running tests inside Visual Studio, [go here](../../testing/libraries/testing-vs.md) + +For more about running tests, read the [running tests](../../testing/libraries/testing.md) document. \ No newline at end of file