Skip to content

Commit 4211dc4

Browse files
authored
CI: Add support in yml for detecting darc dependency changes in eng/Version.Details.xml (#73435)
## Issue Currently, whenever a darc flow PR is opened, we cannot trigger different jobs based on which dependency got updated. This is needed because in some cases, like for ILLinker, Emscripten updates we want to trigger all the wasm jobs. But that might not be needed for some of the other dependencies. - This causes failures to slip through to `main`, which gets discovered after the fact as other PR failures, or rolling build failures, creating extra work for developers. ## Solution - This PR identifies the changed dependencies, and emits one azure variable per dependency, which can then be used in conditions in the yml . - The changed dependency can be checked as `dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_CommandLine']` - Included this in the `evaluate_paths` job itself, instead of creating another job - Also, use this for wasm jobs
1 parent 5f80c24 commit 4211dc4

10 files changed

+195
-0
lines changed
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# This step template evaluates changes in dependencies defined in `eng/Version.Details.xml`.
2+
# For more information on how this works works look at evaluate-changed-darc-deps.sh docs
3+
# at the beginning of that file.
4+
5+
parameters:
6+
subsetName: ''
7+
# Array containing the arguments that are to be passed down to evaluate-changed-paths.sh
8+
# Note that --azurevariable is always set to the dependency name, no need to pass it down.
9+
arguments: []
10+
11+
steps:
12+
- script: eng/pipelines/evaluate-changed-darc-deps.sh
13+
${{ join(' ', parameters.arguments) }}
14+
displayName: Evaluate eng/Version.Details.xml for dependency changes
15+
name: DarcDependenciesChanged

eng/pipelines/common/evaluate-paths-job.yml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,3 +50,10 @@ jobs:
5050
- --includepaths '${{ join('+', path.include) }}'
5151
- ${{ if ne(path.exclude[0], '') }}:
5252
- --excludepaths '${{ join('+', path.exclude) }}'
53+
54+
- template: evaluate-changed-darc-deps.yml
55+
parameters:
56+
arguments:
57+
# The commit that we're building is always a merge commit that is merging into the target branch.
58+
# So the first parent of the commit is on the target branch and the second parent is on the source branch.
59+
- --difftarget HEAD^1

eng/pipelines/common/templates/wasm-build-only.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ jobs:
3939
condition: >-
4040
or(
4141
eq(variables['alwaysRunVar'], true),
42+
eq(variables['wasmDarcDependenciesChanged'], true),
4243
eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
4344
eq(dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'], true),
4445
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true))

eng/pipelines/common/templates/wasm-build-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ jobs:
3434
condition: >-
3535
or(
3636
eq(variables['alwaysRunVar'], true),
37+
eq(variables['wasmDarcDependenciesChanged'], true),
3738
eq(dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'], true),
3839
eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmbuildtests.containsChange'], true))
3940
# extra steps, run tests

eng/pipelines/common/templates/wasm-debugger-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ jobs:
3333
condition: >-
3434
or(
3535
eq(variables['alwaysRunVar'], true),
36+
eq(variables['wasmDarcDependenciesChanged'], true),
3637
eq(dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'], true),
3738
eq(dependencies.evaluate_paths.outputs['SetPathVars_wasmdebuggertests.containsChange'], true))
3839
# extra steps, run tests

eng/pipelines/common/templates/wasm-library-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ jobs:
4444
condition: >-
4545
or(
4646
eq(variables['alwaysRunVar'], true),
47+
eq(variables['wasmDarcDependenciesChanged'], true),
4748
eq(dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'], true),
4849
eq(dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'], true),
4950
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true))

eng/pipelines/common/templates/wasm-runtime-tests.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ jobs:
3434
condition: >-
3535
or(
3636
eq(variables['alwaysRunVar'], true),
37+
eq(variables['wasmDarcDependenciesChanged'], true),
3738
eq(dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'], true),
3839
eq(dependencies.evaluate_paths.outputs['SetPathVars_allwasm.containsChange'], true),
3940
eq(dependencies.evaluate_paths.outputs['SetPathVars_runtimetests.containsChange'], true))

eng/pipelines/common/xplat-setup.yml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,14 @@ jobs:
108108
${{ if eq(parameters.jobParameters.runtimeFlavor, 'coreclr') }}:
109109
value: CoreCLR
110110

111+
- name: wasmDarcDependenciesChanged
112+
${{ if eq(parameters.archType, 'wasm') }}:
113+
value: $[ or(
114+
eq(dependencies.evaluate_paths.outputs['DarcDependenciesChanged.Microsoft_NET_Workload_Emscripten_Manifest-7_0_100'], true),
115+
eq(dependencies.evaluate_paths.outputs['DarcDependenciesChanged.Microsoft_DotNet_Build_Tasks_Workloads'], true),
116+
eq(dependencies.evaluate_paths.outputs['DarcDependenciesChanged.System_Runtime_TimeZoneData'], true),
117+
eq(dependencies.evaluate_paths.outputs['DarcDependenciesChanged.Microsoft_NET_ILLink_Tasks'], true)) ]
118+
111119
- ${{ each variable in parameters.variables }}:
112120
- ${{ variable }}
113121

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
#!/usr/bin/env bash
2+
: '
3+
Compares contents of `env/Version.Details.xml` between HEAD and difftarget, and emits variables named for
4+
dependencies that satisfy either of:
5+
1. version, or sha changed
6+
2. it is missing from one of the xmls
7+
8+
The dependency names have `.` replaced with `_`.
9+
10+
In order to consume these variables in a yaml pipeline, reference them via: $[ dependencies.<JobName>.outputs["<StepName>.<DependencyName>"] ]
11+
12+
Example:
13+
-difftarget ''HEAD^1''
14+
'
15+
16+
# Disable globbing in this bash script since we iterate over path patterns
17+
set -f
18+
19+
# Stop script if unbound variable found (use ${var:-} if intentional)
20+
set -u
21+
22+
# Stop script if command returns non-zero exit code.
23+
# Prevents hidden errors caused by missing error code propagation.
24+
set -e
25+
26+
usage()
27+
{
28+
echo "Script that emits an azure devops variable with all the dependencies that changed in 'eng/Version.Details.xml' contained in the current HEAD against the difftarget"
29+
echo " --difftarget <value> SHA or branch to diff against. (i.e: HEAD^1, origin/main, 0f4hd36, etc.)"
30+
echo " --azurevariableprefix Name of azure devops variable to create if change meets filter criteria"
31+
echo ""
32+
33+
echo "Arguments can also be passed in with a single hyphen."
34+
}
35+
36+
source="${BASH_SOURCE[0]}"
37+
38+
# resolve $source until the file is no longer a symlink
39+
while [[ -h "$source" ]]; do
40+
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
41+
source="$(readlink "$source")"
42+
# if $source was a relative symlink, we need to resolve it relative to the path where the
43+
# symlink file was located
44+
[[ $source != /* ]] && source="$scriptroot/$source"
45+
done
46+
47+
scriptroot="$( cd -P "$( dirname "$source" )" && pwd )"
48+
eng_root=`cd -P "$scriptroot/.." && pwd`
49+
50+
azure_variable_prefix=''
51+
diff_target=''
52+
53+
while [[ $# > 0 ]]; do
54+
opt="$(echo "${1/#--/-}" | tr "[:upper:]" "[:lower:]")"
55+
case "$opt" in
56+
-help|-h)
57+
usage
58+
exit 0
59+
;;
60+
-difftarget)
61+
diff_target=$2
62+
shift
63+
;;
64+
-azurevariableprefix)
65+
azure_variable_prefix=$2
66+
shift
67+
;;
68+
esac
69+
70+
shift
71+
done
72+
73+
if [[ -z "$diff_target" ]]; then
74+
echo "Argument -difftarget is required"
75+
usage
76+
exit 1
77+
fi
78+
79+
oldXmlPath=`mktemp`
80+
81+
ci=true # Needed in order to use pipeline-logging-functions.sh
82+
. "$eng_root/common/pipeline-logging-functions.sh"
83+
84+
git show $diff_target:eng/Version.Details.xml > $oldXmlPath
85+
# FIXME: errors?
86+
changed_deps=$(python3 "$eng_root/pipelines/get-changed-darc-deps.py" $oldXmlPath eng/Version.Details.xml)
87+
rm -f $oldXmlPath
88+
89+
if [[ -n "$azure_variable_prefix" ]]; then
90+
azure_variable_prefix="${azure_variable_prefix}_"
91+
fi
92+
93+
for dep in $changed_deps; do
94+
dep=`echo $dep | tr \. _`
95+
var_name=${azure_variable_prefix}${dep}
96+
echo "Setting pipeline variable $var_name=true"
97+
Write-PipelineSetVariable -name $var_name -value true
98+
done
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
#
2+
# Emits a comma separated list of dependencies from `eng/Version.Details.xml`
3+
# that changed as compared to another versions file
4+
#
5+
# - we don't really care which is old, and which is new
6+
# - A dependency name is emitted as changed if:
7+
# 1. version, or sha changed
8+
# 2. it is missing from one of the xmls
9+
10+
import xml.etree.ElementTree as ET
11+
import sys
12+
from os.path import exists
13+
14+
def getDependencies(xmlfile):
15+
tree = ET.parse(xmlfile)
16+
root = tree.getroot()
17+
deps = {}
18+
for depElement in root.findall('.//Dependency'):
19+
dep = {}
20+
dep['Version'] = depElement.attrib['Version']
21+
dep['Sha'] = depElement.find('Sha').text
22+
23+
deps[depElement.attrib['Name']] = dep
24+
25+
return deps
26+
27+
def compare(dict1, dict2):
28+
if dict1 is None or dict2 is None:
29+
print('Nones')
30+
return False
31+
32+
if (not isinstance(dict1, dict)) or (not isinstance(dict2, dict)):
33+
print('Not dict')
34+
return False
35+
36+
changed_names = []
37+
all_keys = set(dict1.keys()) | set(dict2.keys())
38+
for key in all_keys:
39+
if key not in dict1 or key not in dict2:
40+
print(key)
41+
# changed_names.append(key)
42+
elif dict1[key] != dict2[key]:
43+
print(key)
44+
# changed_names.append(key)
45+
46+
print(','.join(changed_names))
47+
48+
if len(sys.argv) != 3:
49+
print(f'Usage: {sys.argv[0]} <old Version.Details.xml> <new Version.Details.xml>')
50+
exit(1)
51+
52+
if not exists(sys.argv[1]):
53+
print(f'Cannot find {sys.argv[1]}')
54+
exit(1)
55+
if not exists(sys.argv[2]):
56+
print(f'Cannot find {sys.argv[2]}')
57+
exit(1)
58+
59+
newDeps = getDependencies(sys.argv[1])
60+
oldDeps = getDependencies(sys.argv[2])
61+
62+
compare(oldDeps, newDeps)

0 commit comments

Comments
 (0)