Skip to content

Add OneLocBuild template to arcade #6977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Mar 17, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
0a69efb
Localization tests 1
jonfortescue Jan 28, 2021
48b918c
try again
jonfortescue Jan 28, 2021
c092f29
Test 3
jonfortescue Jan 28, 2021
e359d25
Add package to nuget config
jonfortescue Feb 1, 2021
629583b
add display name
jonfortescue Feb 4, 2021
716f6e2
PAT Auth
jonfortescue Feb 5, 2021
d55cfa5
Merge branch 'master' into LocalizationTests
jonfortescue Feb 5, 2021
4247a53
Ooops
jonfortescue Feb 5, 2021
6fddabf
eng-common-ify
jonfortescue Feb 5, 2021
d371b54
Move things around
jonfortescue Feb 5, 2021
b7d38a1
test onelocbuild disabled
jonfortescue Feb 5, 2021
8d1c843
switch to template
jonfortescue Feb 5, 2021
d7ce6e5
test locproject.json
jonfortescue Feb 5, 2021
1b77d81
publish artifacts
jonfortescue Feb 5, 2021
2166815
Dynamically generate LocProject.json
jonfortescue Feb 8, 2021
67b82fc
Generate in pipeline
jonfortescue Feb 8, 2021
425e6a8
Adjust slightly
jonfortescue Feb 8, 2021
0e964c7
Fix up with relative paths
jonfortescue Feb 8, 2021
34dbbce
Add exclusions
jonfortescue Feb 8, 2021
a11152c
Add proper telemetry
jonfortescue Feb 8, 2021
f6e8783
Add github stuff
jonfortescue Feb 9, 2021
d523d65
Turn OneLocBuild into its own job
jonfortescue Feb 9, 2021
289b6ad
Add xlf support
jonfortescue Feb 18, 2021
022b3cc
Finalize script so it's working and in parity with the dotnet-sdk test
jonfortescue Feb 18, 2021
27f690e
Handle null on allXlfFiles
jonfortescue Feb 18, 2021
c579126
Update scripts and templates
jonfortescue Mar 8, 2021
82ce3d5
Merge branch 'main' into LocalizationTests
jonfortescue Mar 9, 2021
da6f3fb
bring up to date with current version
jonfortescue Mar 10, 2021
c341691
Add more onelocbuild variables
jonfortescue Mar 10, 2021
a67067e
fix comparison
jonfortescue Mar 10, 2021
45625ab
Add better comparison
jonfortescue Mar 10, 2021
e24b533
Change alias, add comments
jonfortescue Mar 10, 2021
0ee488f
Local paths for source files
jonfortescue Mar 11, 2021
893204a
Remove-Item on excluded xlfs
jonfortescue Mar 11, 2021
3dd525b
Add repo type parameter to OneLocBuild template
jonfortescue Mar 11, 2021
c08645c
PR feedback
jonfortescue Mar 12, 2021
17a77fe
Add template support
jonfortescue Mar 16, 2021
2801ff3
Clean up neutral xlf cleanup
jonfortescue Mar 16, 2021
1965668
Add comments for creating the localize directory
jonfortescue Mar 16, 2021
ecb081e
fully revert jobs.yml
jonfortescue Mar 16, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 101 additions & 0 deletions eng/common/generate-locproject.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
Param(
[Parameter(Mandatory=$true)][string] $SourcesDirectory, # Directory where source files live; if using a Localize directory it should live in here
[string] $LanguageSet = 'VS_Main_Languages', # Language set to be used in the LocProject.json
[switch] $UseCheckedInLocProjectJson, # When set, generates a LocProject.json and compares it to one that already exists in the repo; otherwise just generates one
[switch] $CreateNeutralXlfs # Creates neutral xlf files. Only set to false when running locally
)

# Generates LocProject.json files for the OneLocBuild task. OneLocBuildTask is described here:
# https://ceapex.visualstudio.com/CEINTL/_wiki/wikis/CEINTL.wiki/107/Localization-with-OneLocBuild-Task

Set-StrictMode -Version 2.0
$ErrorActionPreference = "Stop"
. $PSScriptRoot\tools.ps1

Import-Module -Name (Join-Path $PSScriptRoot 'native\CommonLibrary.psm1')

$exclusionsFilePath = "$SourcesDirectory\Localize\LocExclusions.json"
$exclusions = @{ Exclusions = @() }
if (Test-Path -Path $exclusionsFilePath)
{
$exclusions = Get-Content "$exclusionsFilePath" | ConvertFrom-Json
}

Push-Location "$SourcesDirectory" # push location for Resolve-Path -Relative to work

# Template files
$jsonFiles = @()
$jsonFiles += Get-ChildItem -Recurse -Path "$SourcesDirectory" | Where-Object { $_.FullName -Match "\.template\.config\\localize\\en\..+\.json" } # .NET templating pattern
$jsonFiles += Get-ChildItem -Recurse -Path "$SourcesDirectory" | Where-Object { $_.FullName -Match "en\\strings\.json" } # current winforms pattern

$xlfFiles = @()

$allXlfFiles = Get-ChildItem -Recurse -Path "$SourcesDirectory\*\*.xlf"
$langXlfFiles = @()
if ($allXlfFiles) {
$null = $allXlfFiles[0].FullName -Match "\.([\w-]+)\.xlf" # matches '[langcode].xlf'
$firstLangCode = $Matches.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if there is more than one "langcode" in the repo (maybe for test assets or soemthing). It seems like the code should at least produce some sort of warning in that situation, since it's going to pick one arbitrarily at that point (and potentially not even deterministically, since we aren't sorting anything)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out that it doesn't matter -- it just needs a file with a langcode since all xlfs also contain English text equivalents, so we just take the first one and make it the "generic" xlf. This solution is actually ripped directly from their docs: https://ceapex.visualstudio.com/CEINTL/_wiki/wikis/CEINTL.wiki/1450/OneLocBuild-Non-Enu-source-file-support-(workaround)

Copy link
Contributor

@ChadNedzlek ChadNedzlek Mar 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this implies that every set of "xlf" files in the entire repository have the same set of languages. What if I have a.en.xlf, and b.fr.xlf.

This means that b.fr.xlf will be lost, because we are only going to look for *.en.xlf.

Copy link
Contributor

@ChadNedzlek ChadNedzlek Mar 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that first sentence going to be something this code assumes, but doesn't validate? That seems... scary... And validating it shouldn't be too hard... just search for ALL xlf files, and then group them by the first part. (That'd be a lot easier if this was C# and not powershell)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically "dotnet script" is a thing. :-) Just sayin...

Copy link
Contributor Author

@jonfortescue jonfortescue Mar 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fairly certain that it's a valid assumption to make because the LanguageSet variable applies to the entire repository -- all xlfs will be translated into the same set of languages.

I agree that the second solution is more robust but it's pretty difficult in powershell... that's exactly what I would have done had this been C# haha.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ouch, quite the assumption for the tool to make, especially with repositories as large as ours. I hope that doesn't bite us. It... seems to hold for the repos I happen to have on my box? :-)

$langXlfFiles = Get-ChildItem -Recurse -Path "$SourcesDirectory\*\*.$firstLangCode.xlf"
}
$langXlfFiles | ForEach-Object {
$null = $_.Name -Match "([^.]+)\.[\w-]+\.xlf" # matches '[filename].[langcode].xlf'

$destinationFile = "$($_.Directory.FullName)\$($Matches.1).xlf"
$xlfFiles += Copy-Item "$($_.FullName)" -Destination $destinationFile -PassThru
}

$locFiles = $jsonFiles + $xlfFiles

$locJson = @{
Projects = @(
@{
LanguageSet = $LanguageSet
LocItems = @(
$locFiles | ForEach-Object {
$outputPath = "Localize\$(($_.DirectoryName | Resolve-Path -Relative) + "\")"
$continue = $true
foreach ($exclusion in $exclusions.Exclusions) {
if ($outputPath.Contains($exclusion))
{
$continue = $false
}
}
$sourceFile = ($_.FullName | Resolve-Path -Relative)
if (!$CreateNeutralXlfs -and $_.Extension -eq '.xlf') {
Remove-Item -Path $sourceFile
}
if ($continue)
{
return @{
SourceFile = $sourceFile
CopyOption = "LangIDOnName"
OutputPath = $outputPath
}
}
}
)
}
)
}

$json = ConvertTo-Json $locJson -Depth 5
Write-Host "(NETCORE_ENGINEERING_TELEMETRY=Build) LocProject.json generated:`n`n$json`n`n"
Pop-Location

if (!$UseCheckedInLocProjectJson) {
New-Item "$SourcesDirectory\Localize\LocProject.json" -Force # Need this to make sure the Localize directory is created
Set-Content "$SourcesDirectory\Localize\LocProject.json" $json
}
else {
New-Item "$SourcesDirectory\Localize\LocProject-generated.json" -Force # Need this to make sure the Localize directory is created
Set-Content "$SourcesDirectory\Localize\LocProject-generated.json" $json

if ((Get-FileHash "$SourcesDirectory\Localize\LocProject-generated.json").Hash -ne (Get-FileHash "$SourcesDirectory\Localize\LocProject.json").Hash) {
Write-PipelineTaskError -Type "warning" -Message "Existing LocProject.json differs from generated LocProject.json. Download LocProject-generated.json and compare them."

exit 1
}
else {
Write-Host "Generated LocProject.json and current LocProject.json are identical."
}
}
75 changes: 75 additions & 0 deletions eng/common/templates/job/onelocbuild.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
parameters:
# Optional: dependencies of the job
dependsOn: ''

# Optional: A defined YAML pool - https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=vsts&tabs=schema#pool
pool:
vmImage: vs2017-win2016

CeapexPat: $(dn-bot-ceapex-package-r) # PAT for the loc AzDO instance https://dev.azure.com/ceapex
GithubPat: $(BotAccount-dotnet-bot-repo-PAT)

SourcesDirectory: $(Build.SourcesDirectory)
CreatePr: true
UseCheckedInLocProjectJson: false
LanguageSet: VS_Main_Languages
LclSource: lclFilesInRepo
LclPackageId: ''
RepoType: gitHub

jobs:
- job: OneLocBuild

dependsOn: ${{ parameters.dependsOn }}

displayName: OneLocBuild

pool: ${{ parameters.pool }}

variables:
- group: OneLocBuildVariables # Contains the CeapexPat and GithubPat
- name: _GenerateLocProjectArguments
value: -SourcesDirectory ${{ parameters.SourcesDirectory }}
-LanguageSet "${{ parameters.LanguageSet }}"
-CreateNeutralXlfs
- ${{ if eq(parameters.UseCheckedInLocProjectJson, 'true') }}:
- name: _GenerateLocProjectArguments
value: ${{ variables._GenerateLocProjectArguments }} -UseCheckedInLocProjectJson


steps:
- task: Powershell@2
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/generate-locproject.ps1
arguments: $(_GenerateLocProjectArguments)
displayName: Generate LocProject.json

- task: OneLocBuild@2
displayName: OneLocBuild
inputs:
locProj: Localize/LocProject.json
outDir: $(Build.ArtifactStagingDirectory)
lclSource: ${{ parameters.LclSource }}
lclPackageId: ${{ parameters.LclPackageId }}
isCreatePrSelected: ${{ parameters.CreatePr }}
repoType: ${{ parameters.RepoType }}
gitHubPatVariable: "${{ parameters.GithubPat }}"
packageSourceAuth: patAuth
patVariable: ${{ parameters.CeapexPat }}
condition: always()

- task: PublishBuildArtifacts@1
displayName: Publish Localization Files
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)/loc'
PublishLocation: Container
ArtifactName: Loc
condition: always()

- task: PublishBuildArtifacts@1
displayName: Publish LocProject.json
inputs:
PathtoPublish: '$(Build.SourcesDirectory)/Localize/'
PublishLocation: Container
ArtifactName: Loc
condition: always()