Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Conda Publishing #17889

Merged
merged 61 commits into from
Apr 13, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
ae2730d
skeleton getting set up for the addition of conda artifacts to our bu…
scbedd Mar 26, 2021
8dcf31b
pass CondaArtifacts to jobs/archetype-sdk-client.yml
scbedd Mar 26, 2021
4e635bb
fix a couple issues with bad yaml that daniel identified
scbedd Mar 26, 2021
16cc743
more scaffolding! almost ready to get the builds working!
scbedd Mar 27, 2021
aaf6296
fully flesh out the structure that creates the combined SDIST.
scbedd Mar 29, 2021
689ad60
adding skeleton. time to combine the sdists
scbedd Mar 29, 2021
6830924
updating the location of the yaml for get-tagged-code
scbedd Mar 29, 2021
ae60d48
adding the steps to combine output and generate a combined sdist from…
scbedd Mar 30, 2021
e76bd01
apply black
scbedd Mar 30, 2021
907e483
clean up the failing builds
scbedd Mar 30, 2021
e9a1fb0
sdists should assemble now!
scbedd Mar 30, 2021
a7d5753
resolve spacing issues
scbedd Mar 30, 2021
ec593f4
update to clean up the targetPath of the clone
scbedd Mar 30, 2021
ec71549
correct a pshell boolean comparison
scbedd Mar 30, 2021
9582a88
resolve issue with setting the Package.Clone location
scbedd Mar 30, 2021
bcd5a45
correct target path
scbedd Mar 30, 2021
bf499cf
add debug output to build_conda_artifacts.py. pass the actual artifac…
scbedd Mar 31, 2021
bcdc24f
temporarily disable docs build
scbedd Mar 31, 2021
c2712d9
cleanup progress
scbedd Mar 31, 2021
1c66f77
add final pieces! almost there!
scbedd Mar 31, 2021
f271ce4
don't call activate
scbedd Mar 31, 2021
4d783b8
call it appropriately
scbedd Mar 31, 2021
59c3aed
default conda to pshell
scbedd Mar 31, 2021
f0cc2d0
swap to use bash instead of pwsh tasks.
scbedd Mar 31, 2021
79dd0ea
remove bad commands
scbedd Mar 31, 2021
d66d4ab
proper print
scbedd Mar 31, 2021
5116bd7
swap the conda environment to be named on a per artifact basis
scbedd Mar 31, 2021
352a275
additional documentation
scbedd Apr 2, 2021
2726c6c
line items
scbedd Apr 2, 2021
f345f43
add the core artifact such that it builds before we do.
scbedd Apr 5, 2021
4f5b220
storage needed a requirement update
scbedd Apr 5, 2021
55a5571
handle singular artifact (build the target only). pull from for combi…
scbedd Apr 6, 2021
eb0bd26
add azsdk channel to conda build invocation
scbedd Apr 7, 2021
019edef
resolve conflicts
scbedd Apr 7, 2021
65f2b10
remove wonky pre-dep
scbedd Apr 7, 2021
895fd32
disable build docs until we're satisfied with the results. add missin…
scbedd Apr 7, 2021
603be52
fixed small bug
scbedd Apr 7, 2021
6089976
resolve yml vs yaml naming issues
scbedd Apr 8, 2021
87fec6c
update requests-oauthlib to actually have the correct name!
scbedd Apr 8, 2021
cec19c2
handle generation issues with the sdist
scbedd Apr 8, 2021
fca430d
resolve issue with single dependency conda builds
scbedd Apr 8, 2021
cdae4a0
black. add comment describing the build folder requirement.
scbedd Apr 8, 2021
f4ab22c
the manifest.in that gets output
scbedd Apr 8, 2021
7f509bd
Update sdk/storage/ci.yml
scbedd Apr 9, 2021
df38c20
Apply suggestions from code review
scbedd Apr 10, 2021
35ea605
update to remove doc disable. update the template to remove py35 and …
scbedd Apr 10, 2021
3c3030e
Merge branch 'feature/add-conda-publishing' of https://github.com/Azu…
scbedd Apr 10, 2021
608ab46
correct a slight spelling issue
scbedd Apr 10, 2021
dec9710
update to consume common template
scbedd Apr 12, 2021
75c87db
cannot have muliple checkout: none steps in the same job
scbedd Apr 12, 2021
a4b0206
disable checkout: none
scbedd Apr 12, 2021
1995e9d
correct yml syntax
scbedd Apr 12, 2021
05579ab
dumb typo
scbedd Apr 12, 2021
69539a7
debugging
scbedd Apr 12, 2021
dbf667d
instead of using temp directory, try using the sources directory
scbedd Apr 12, 2021
f4ea5ad
stop attempting to clean up. clone to fresh directory each time
scbedd Apr 12, 2021
12fae40
merge master
scbedd Apr 12, 2021
a054d66
undo changes to sparse-checkout
scbedd Apr 13, 2021
9cd849e
restore to master
scbedd Apr 13, 2021
7b120ed
make my own safename
scbedd Apr 13, 2021
bc53822
update name of the conda release so a dev can discern the difference
scbedd Apr 13, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions doc/dev/conda-builds.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Azure SDK for Python Conda Distributions

## Local Environment Setup

Follow the instructions [here](https://docs.conda.io/projects/conda-build/en/latest/install-conda-build.html) to install `conda` and `conda-build`.

## CI Build Process

There will be a `CondaArtifact` defined in the `ci.yml` of each service directory. (`sdk/<service>`)

A Conda Artifact defines:
- The location of the `meta.yml`
- Which packages will be pulled into the combined artifact
- The name of the combined artifact
- Any other necessary details.

## How to Build an Azure SDK Conda Package Locally


### Create Your Build Directory
Given how Conda packages are comprised of multiple source distributions _combined_, the buildable source does not exist directly within the azure-sdk-for-python repo. Currently, there is _some_ manual work that needs to be done.

To begin, check your `ci.yml` for a `CondaArtifact`. Each these artifacts will become a single conda package. Let's use `storage/ci.yml` as an example.

```
- name: azure-storage
meta_source: meta.yml
common_root: azure/storage
checkout:
- package: azure-storage-blob
checkout_path: sdk/storage
version: 12.8.0
- package: azure-storage-queue
checkout_path: sdk/storage
version: 12.1.5
- package: azure-storage-file-share
checkout_path: sdk/storage
version: 12.4.1
- package: azure-storage-file-datalake
checkout_path: sdk/storage
version: 12.3.0
```

- `name: azure-storage`: will be the name of the "combined" sdist package that we generate.
- `meta_source: meta.yml`: this is the path (relative to the service directory) to the target conda package meta.yml.
- `common_root: azure/storage`: when generating the combined package, where will we begin combining? This is tightly bound to folder structure within the generated sdist.
- `checkout`: the `checkout` setting is a list of target packages that will go into the combined artifact. These targets will be individually sparse cloned, and copied into the conda build directory. Currently, this is a **manual step** in your local build. Reference `eng/pipelines/templates/get-tagged-code.yml` for exact details on how CI does it.

Before we continue, you should be aware of two primary locations that are necessary, but not referenced directly in the `ci.yml`.

The `build` folder and the `output` folder. The `build` folder (`$(Conda.Build)` variable in CI) is where we will...

- store the cloned package code
- generate the combined sdist

To locally repro without magic given a specific `checkout` artifact:

```
<cd sdk-for-python>
git checkout `<package>_<version>`
grab the entire <package> directory under the <checkout_path>. place into your `build` folder.
```

Given the `storage` example. This is what your `build` folder should look like prior to invoking `build_conda_artifacts.py`.

```
<build directory>/
azure-storage-blob/ <-- the package directly ripped from specified tag
azure-storage-file-datalake/
azure-storage-file-share/
azure-storage-queue/
```

### Create the Combined SDist

Once you have a directory assembled, invoke the script to build. The below command is formatted for visibility, recombine the lines however necessary for your chosen shell environment.


```
python `build_conda_artifacts.py`
-d "<output folder>"
-b "<build folder>"
-m "<resolvable path to sdk/storage/meta.yml>"
-r "azure/storage"
-n "azure-storage"
-s "storage"
```

### Generate the Conda Package

Locally, from the anaconda prompt, set the environment variable `STORAGE_SOURCE_DISTRIBUTION` to the location of the generated sdist. After that:

```bash
export STORAGE_SOURCE_DISTRIBUTION=<path/to/generated/sdist>
cd <meta.yml directory>
conda-build . --output-folder <conda.output>
```
9 changes: 9 additions & 0 deletions eng/pipelines/templates/jobs/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ parameters:
- name: Artifacts
type: object
default: []
- name: CondaArtifacts
type: object
default: []
- name: TestPipeline
type: boolean
default: false
Expand Down Expand Up @@ -61,6 +64,12 @@ jobs:
TestPipeline: ${{ parameters.TestPipeline }}
Artifacts: ${{ parameters.Artifacts }}

- template: ../steps/build-conda-artifacts.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
TestPipeline: ${{ parameters.TestPipeline }}
CondaArtifacts: ${{ parameters.CondaArtifacts }}

- job: 'Analyze'
condition: and(succeededOrFailed(), ne(variables['Skip.Analyze'], 'true'))
variables:
Expand Down
34 changes: 34 additions & 0 deletions eng/pipelines/templates/stages/archetype-conda-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
parameters:
CondaArtifacts: []
TestPipeline: false
DependsOn: Build
ArtifactName: 'not-specified'
ServiceDirectory: 'not-specified'


stages:
- ${{if and(eq(variables['Build.Reason'], 'Manual'), eq(variables['System.TeamProject'], 'internal'))}}:
- ${{ each artifact in parameters.CondaArtifacts }}:
- stage: Release_${{ replace(artifact.name, '-', '_') }}
displayName: 'Conda Release: ${{artifact.name}}'
dependsOn: ${{parameters.DependsOn}}
condition: and(succeeded(), ne(variables['SetDevVersion'], 'true'), ne(variables['Skip.Release'], 'true'), ne(variables['Build.Repository.Name'], 'Azure/azure-sdk-for-python-pr'))
jobs:
- deployment: CondaRelease
displayName: "Publish Conda Artifacts"
condition: ne(variables['Skip.TagRepository'], 'true')
environment: pypi

pool:
name: azsdk-pool-mms-ubuntu-1804-general
vmImage: MMSUbuntu18.04

strategy:
runOnce:
deploy:
steps:
- checkout: self
- pwsh: |
Get-ChildItem -Recurse $(Pipeline.Workspace)/${{parameters.ArtifactName}}/${{artifact.name}}
workingDirectory: $(Pipeline.Workspace)
displayName: Output Visible Conda Artifacts
17 changes: 17 additions & 0 deletions eng/pipelines/templates/stages/archetype-sdk-client.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ parameters:
- name: Artifacts
type: object
default: []
- name: CondaArtifacts
type: object
default: []
- name: TestPipeline
type: boolean
default: false
Expand Down Expand Up @@ -62,6 +65,7 @@ stages:
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
Artifacts: ${{ parameters.Artifacts }}
CondaArtifacts: ${{ parameters.CondaArtifacts }}
${{ if eq(parameters.ServiceDirectory, 'template') }}:
TestPipeline: true
BeforePublishSteps: ${{ parameters.BeforePublishSteps }}
Expand All @@ -85,6 +89,7 @@ stages:
- ${{ each replacement in parameters.MatrixReplace }}:
- ${{ replacement }}


# The Prerelease and Release stages are conditioned on whether we are building a pull request and the branch.
- ${{if and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['System.TeamProject'], 'internal'))}}:
- template: archetype-python-release.yml
Expand All @@ -99,3 +104,15 @@ stages:
TargetDocRepoOwner: ${{ parameters.TargetDocRepoOwner }}
TargetDocRepoName: ${{ parameters.TargetDocRepoName }}
DevFeedName: ${{ parameters.DevFeedName }}

# The Prerelease and Release stages are conditioned on whether we are building a pull request and the branch.
- ${{if and(ne(variables['Build.Reason'], 'PullRequest'), eq(variables['System.TeamProject'], 'internal'))}}:
- template: archetype-conda-release.yml
parameters:
DependsOn: Build
ServiceDirectory: ${{ parameters.ServiceDirectory }}
CondaArtifacts: ${{ parameters.CondaArtifacts }}
ArtifactName: conda
${{ if eq(parameters.ServiceDirectory, 'template') }}:
TestPipeline: true

77 changes: 77 additions & 0 deletions eng/pipelines/templates/steps/build-conda-artifacts.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
parameters:
- name: TestPipeline
type: boolean
default: false
- name: ServiceDirectory
type: string
default: ''
- name: CondaArtifacts
type: object
default: []

steps:
- template: /eng/common/pipelines/templates/steps/set-test-pipeline-version.yml
parameters:
PackageName: "azure-template"
ServiceDirectory: "template"
TestPipeline: ${{ parameters.TestPipeline }}

- task: UsePythonVersion@0
displayName: 'Use Python $(PythonVersion)'
inputs:
versionSpec: $(PythonVersion)

- script: |
pip install -r eng/ci_tools.txt
displayName: 'Prep Environment'

- pwsh: |
mkdir $(Agent.BuildDirectory)/conda/
mkdir $(Agent.BuildDirectory)/conda/output
mkdir $(Agent.BuildDirectory)/conda/build

Write-Host "##vso[task.setvariable variable=conda.output]$(Agent.BuildDirectory)/conda/output"
Write-Host "##vso[task.setvariable variable=conda.build]$(Agent.BuildDirectory)/conda/build"
displayName: 'Create Conda Working Directories'

- ${{ each artifact in parameters.CondaArtifacts }}:
# there may be multiple CondaArtifacts. Be certain $(conda.build) is clean just in case!
- pwsh:
Write-Host "Clean up Conda Build Directory $(conda.build)"
Remove-Item $(conda.build)/* -Recurse -Force
displayName: 'Clean Up Before Building ${{ artifact.name }}'

- ${{ each checkout in artifact.checkout }}:
- template: /eng/pipelines/templates/steps/get-tagged-code.yml
parameters:
DestinationDirectory: $(conda.build)/${{checkout.package}}
Package: ${{checkout.package}}
CheckoutPath: ${{checkout.checkout_path}}
Version: ${{checkout.version}}

- task: PythonScript@0
displayName: 'Build Source Distribution for ${{ artifact.name }}'
inputs:
scriptPath: 'scripts/devops_tasks/build_conda_artifacts.py'
arguments: '-d "$(conda.output)" -b "$(conda.build)" -m "$(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}/${{ artifact.meta_source }}" -r "${{ artifact.common_root }}" -n "${{ artifact.name }}" -s "${{ parameters.ServiceDirectory }}" -o "${{ upper(parameters.ServiceDirectory) }}_SOURCE_DISTRIBUTION"'

- bash: |
echo "##vso[task.prependpath]$CONDA/bin"
displayName: 'Prepend PATH with Conda and INIT'

- bash: |
conda create --yes --quiet --name ${{ artifact.name }}
source activate ${{ artifact.name }}
conda install --yes --quiet --name ${{ artifact.name }} conda-build
displayName: 'Prepare Conda Environment for building ${{ artifact.name }}'

- bash: |
source activate ${{ artifact.name }}
conda-build . --output-folder "$(Agent.BuildDirectory)/conda/output" -c https://azuresdkconda.blob.core.windows.net/channel1/
displayName: 'Activate Conda Environment and Build ${{ artifact.name }}'
workingDirectory: $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}

- template: /eng/common/pipelines/templates/steps/publish-artifact.yml
parameters:
ArtifactPath: '$(Agent.BuildDirectory)/conda/output'
ArtifactName: 'conda'
42 changes: 42 additions & 0 deletions eng/pipelines/templates/steps/get-tagged-code.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
parameters:
- name: DestinationDirectory
type: string
default: ''
- name: Package
type: string
default: ''
- name: CheckoutPath
type: string
default: ''
- name: Version
type: string
default: ''

steps:
- pwsh: |
$targetPath = "$(Agent.TempDirectory)/${{ parameters.Package }}"
if (!(Test-Path $targetPath)) {
mkdir $targetPath
}

Write-Host "##vso[task.setvariable variable=Package.Clone]$targetPath"
displayName: 'Prep for Sparse Checkout'

- template: /eng/common/pipelines/templates/steps/sparse-checkout.yml
parameters:
Paths:
- "${{ parameters.CheckoutPath }}/${{ parameters.Package }}"
Repositories:
- Name: "Azure/azure-sdk-for-python"
Commitish: "${{ parameters.Package }}_${{ parameters.Version }}"
WorkingDirectory: "$(Package.Clone)"
SkipDefaultCheckout: true

- pwsh: |
$pathForCopy = Join-Path -Path "$(Package.Clone)" -ChildPath "${{ parameters.CheckoutPath }}/${{ parameters.Package }}"

Write-Host $pathForCopy
Write-Host ${{ parameters.DestinationDirectory }}

Copy-Item -Path $pathForCopy -Destination ${{ parameters.DestinationDirectory }} -Recurse
displayName: 'Copy Source to Target Directory'
Loading