salesforce delta deployment CICD using azure devops yaml

[bharuk]

Is your proposal related to a problem?

---

Hi we are planning to salesforce delta using azure devops CICD i want help from yaml script to build salesforce delta deployment

trigger: none

resources:
    webhooks:
    - webhook: testspa_new          ### Webhook alias
      connection: testspa_new
      filters:
        - path: pull_request.merged  
          value: true
        - path: pull_request.base.ref  
          value: "develop"


    repositories:
    - repository: Mytest  
      type: githubenterprise
      name: karu-bhar/Mytest
      ref: develop
      endpoint: myrepo1

      extends:
      template: keyfiles.yml

variables:
    - name: devops_dir
      value: "/home/vsts/work/1/scripts-deploy"  
    - name: global_dir
      value: "/home/vsts/work/1/s/Mytest"
    #- group: varhashicorp
    #- group: vargroup-sfdx
     
stages:
 - stage: UnlockedPackage_Creation
   jobs:
      - job: Build 
        timeoutInMinutes: 360
        pool:
         vmImage: ubuntu-latest
         
        steps:
        - checkout: self
          path: scripts-deploy
        - checkout: Mytest
        - task: PowerShell@2
          displayName: 'List files in Git Directory'
          inputs:
            targetType: 'inline'
            script: 'Get-ChildItem -Path "$(System.DefaultWorkingDirectory)" -recurse'    
            workingDirectory: '$(System.DefaultWorkingDirectory)'
        - task: Bash@3
          inputs:
            targetType: 'inline'
            script: |
              cd $(System.DefaultWorkingDirectory)/Mytest
            #cat /home/vsts/work/1/s/Mytest/describe.log
        - task: Bash@3
          displayName: 'Set git config email & name'
          inputs:
            targetType: 'inline'
            script: |
             git config --global user.email "kar.bhar@xx.com"
             git config --global user.name "kar-bhar"
        - task: UsePythonVersion@0
          inputs:
           versionSpec: '3.7'
        - script: |
           python -m pip install --upgrade pip
        - bash:
           #python --version 
           pip install wheel --upgrade
           pip install dohq-artifactory
           pip install hvac
           pip install -r /home/vsts/work/1/scripts-deploy/folder/requirements.txt --no-cache-dir
          # pip install -r /home/vsts/work/1/s/scripts-deploy/folder/requirements.txt  
           
        #- script: pip install -r requirements.txt
        #  displayName: 'Install requirements'
        - bash:
           npm install -g npm
          displayName: Install Latest version of npm  

             #INSTALL SFDX_CLI
        - bash: 
            npm install sfdx-cli --global
          displayName: Install SFDX
           echo y | node_modules/sfdx-cli/bin/run plugins:install sfdx-git-delta
        
        #INSTALL VLOCITY BUILD TOOL
        - bash:
            npm install --global vlocity
          displayName: Install Vlocity
        - bash: |
                 cd $(devops_dir)
                 sfdx force:auth:jwt:grant --clientid "3MVG" --jwtkeyfile "./buildfiles/server.key" --username "bharu@vois.sandbox" --instanceurl "https://login.salesforce.com"
        
        - task: Bash@3
          inputs:
           targetType: 'inline'
           script: |
            $logFile= sfdx force:mdapi:describemetadata -u "@vois.sandbox" -f $global_dir/describe.log --apiversion "56.0"`

Describe a solution you propose

---

Describe alternatives you've considered

---

Additional context

---

[scolladon]

Hi,

I'm not very skilled at azure devops pipeline, but I'll ask experts around me and try to come with more precise advice.

The guidelines when using SGD is to git fetch locally in the CI the references you are going to compare so SGD can access those.

So the checkout step should set fetchDepth: 0.

And then you can use SGD:

- bash:
			mkdir output
            sfdx sgd:source:delta --to "HEAD" --from "<target-branch>" --output output -d

Then you can use the generated package.xml in output/package/package.xml to deploy or the source folder generated in the output

Hope it helps

[laithal]

Hi,

The best way to build YAML pipelines is by trying, failing, and trying again :)

1. I'd stick to either bash (preferred) or PowerShell. Combining them increases complexities.
2. Do not store your secrets in the yaml file! Please consider using Library of assets
   A library is a collection of build and release assets for an Azure DevOps project. Assets defined in a library can be used in multiple build and release pipelines of the project. The Library tab can be accessed directly in Azure Pipelines. You can use it to e.g. store your Salesforce Connected Apps secrets, certifications, creds, etc. e.g. in your Yaml file you then simply reference to the library by e ${{ parameters.clientID }}
3. Add fetchDepth: 0 as line 43

Here is a sample YAML - please test and validate in a testing environment before using in Prod.

`
parameters:
- name: environment
type: string
- name: instanceurl
type: string
default: 'https://login.salesforce.com/'
- name: library
type: string

jobs:
  - deployment: DeployPackage
    environment: ${{ parameters.environment }}
    container: salesforce/salesforcedx:latest-full
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            fetchDepth: 0
          - task: DownloadPipelineArtifact@2
            inputs:
              artifact: drop
              path: '$(Pipeline.Workspace)'
          - task: DownloadSecureFile@1
            name: jwtkeyfile
            displayName: 'Download JWT key file'
            inputs:
              secureFile: 'JWT-${{ parameters.library }}'
          - script: |
              echo y | sfdx plugins:install sfdx-git-delta
              sfdx plugins
            displayName: 'Installing SGD'
          - script: |
              sfdx force:auth:jwt:grant --clientid $(consumerkey) \
              --jwtkeyfile $(jwtkeyfile.secureFilePath) --username $(username) \
              --setalias ${{ parameters.library }} -r ${{ parameters.instanceurl }}
              sfdx force:auth:list
            displayName: 'Authenticate'
          - script: |
              sfdx force:source:deploy -p force-app -w 60 --targetusername $(username) --noprompt       
            displayName: 'Deploy package'
          # Following Script will remove Salesforce components specified in the destructive/destructiveChangesPost.xml file in target Orgs.
          - script: |
              sfdx force:mdapi:deploy -d destructive -w -1 --targetusername $(username) --ignorewarnings
            displayName: 'Destructive Changes'`

[bharuk]

I am getting error destructive changes should i create a directory in salesforce code repository?
Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead.
Warning: The "--targetusername" flag has been deprecated. Use "--target-org | -o" instead.
Error (1): Deploy directory destructive does not exist or is not a directory
/home/vsts/work/_temp/cd3913f9-c8a4-405b-afdf-a1cf72dda31a.sh: line 2: unexpected EOF while looking for matching ``'
/home/vsts/work/_temp/cd3913f9-c8a4-405b-afdf-a1cf72dda31a.sh: line 3: syntax error: unexpected end of file
##[debug]Exit code 2 received from tool '/usr/bin/bash'

[scolladon]

The sample uses a destructive folder.
In this case it should be the <output>/destructiveChanges, where output is the folder passed to --output / -o parameter of the sgd command.

jobs:
  - deployment: DeployPackage
    environment: ${{ parameters.environment }}
    container: salesforce/salesforcedx:latest-full
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
            fetchDepth: 0
          - task: DownloadPipelineArtifact@2
            inputs:
              artifact: drop
              path: '$(Pipeline.Workspace)'
          - task: DownloadSecureFile@1
            name: jwtkeyfile
            displayName: 'Download JWT key file'
            inputs:
              secureFile: 'JWT-${{ parameters.library }}'
          - script: |
              echo y | sfdx plugins:install sfdx-git-delta
              sfdx plugins
            displayName: 'Installing SGD'
          - script: |
              sfdx force:auth:jwt:grant --clientid $(consumerkey) \
              --jwtkeyfile $(jwtkeyfile.secureFilePath) --username $(username) \
              --setalias ${{ parameters.library }} -r ${{ parameters.instanceurl }}
              sfdx force:auth:list
            displayName: 'Authenticate'
          -script: |
              mkdir output
              sfdx sgd:source:delta --to "HEAD" --from "<target-branch>" --output output -d
            displayName: 'Generate incremental package'
          - script: |
              sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(username) --noprompt       
            displayName: 'Deploy package'
          # Following Script will remove Salesforce components specified in the destructive/destructiveChangesPost.xml file in target Orgs.
          - script: |
              sfdx force:mdapi:deploy -d output/destructiveChanges -w -1 --targetusername $(username) --ignorewarnings
            displayName: 'Destructive Changes'`

You need to replace <target branch> with the name of the branch the pull request is targetting, with its remote name.
Per example if your branch target the main branch and the remote is named origin, the target branch is origin/main

[mehdicherf]

@bharuk If that's ok with you I'm going to close this issue as the error does not seem related to the plugin itself, and @laithal and @scolladon have provided sample working Azure pipeline.
Of course feel free to reopen an issue if you face an error related to the plugin.

[bharuk]

Hi I am using above code I am getting unknown error

`trigger: none

resources:
webhooks:
- webhook: gspdeltawebhook ### Webhook alias
connection: gspdeltawebhook
filters:
- path: pull_request.merged
value: true
- path: pull_request.base.ref
value: "develop" ##branch

repositories:
- repository: b2b-gsp-slfdelta  
  type: githubenterprise
  name: VFGroup-TaaS-Tenants/b2b-gsp-slfdelta
  ref: develop
  endpoint: Vodafone_GSP_Delta

variables:
- name: devops_dir
value: "/home/vsts/work/1/b2b-gsp-deltadevops"
- name: global_dir
value: "/home/vsts/work/1/s/b2b-gsp-slfdelta"
- group: vargroup-sfdx
stages:

• stage: Creation
  jobs:

  • job: Build
    timeoutInMinutes: 360
    pool:
    vmImage: ubuntu-latest

    steps:

    • checkout: self
      path: b2b-gsp-deltadevops
    • checkout: b2b-gsp-slfdelta
    • task: DownloadSecureFile@1
      name: DevserverkeyFile
      displayName: download rsa key
      inputs:
      secureFile: 'Devserver.key'
    • script: |
      echo installing the $(DevserverkeyFile.secureFilePath) to the trusted directory
    • bash:
      npm install -g npm
      displayName: Install Latest version of npm
    • bash:
      npm install sfdx-cli --global
      displayName: Install SFDX
    • script: |
      echo y | sfdx plugins:install sfdx-git-delta
      sfdx plugins
      displayName: 'Installing SGD'
      #INSTALL VLOCITY BUILD TOOL
    • bash:
      npm install --global vlocity
      displayName: Install Vlocity
    • bash: |
      cd $(devops_dir)
      sfdx force:auth:jwt:grant --clientid $(DevhubclientId) --jwtkeyfile $(DevserverkeyFile.secureFilePath) --username $(DevhubUserName) --instanceurl $(DevhubInstanceURL)
      displayName: 'Authenticate'
    • script: |
      cd $(global_dir)
      mkdir output
      sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output output -d
      displayName: 'Generate incremental package'
    • script: |
      cd $(global_dir)
      sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(DevhubUserName)
      displayName: 'Deploy package'

branch :origin/develop
feature: feature-version1

Error:

Generating script.
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/0b5e8411-b898-4f3e-a7da-c2d1a8ee14ba.sh
(node:2104) Warning: Deprecated config name: apiVersion. Please use org-api-version instead.
(Use node --trace-warnings ... to show where the warning was created)
{
"error": "--from is not a valid sha pointer: 'origin/develop'",
"output": "output",
"success": false,
"warnings": []
}
##[error]Bash exited with code '1'.
Finishing: Generate incremental package

'

[scolladon]

Hi @bharuk !

It seems the checkout steo does not specifyfetchDepth: 0attribute.
Then the ref 'origin/develop' is not available in the execution context of the pipeline.

Could you try using the fetchDepth attribute in the checkout steps?

[bharuk]

It is working fine steps: - checkout: self path: b2b-gsp-deltadevops - checkout: b2b-gsp-slfdelta fetchDepth: 0
another error deploy package

- script: | cd $(global_dir) mkdir output sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output output -d displayName: 'Generate incremental package' - script: | cd $(global_dir) sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(DevhubUserName) displayName: 'Deploy package'

we created already output directory why it is not found

Error:

/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/f8493e5e-19d5-43c9-b685-373f0d0efbfd.sh Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead. Warning: The "--targetusername" flag has been deprecated. Use "--target-org | -u" instead. Error (1): output/force-app: File or folder not found ##[debug]Exit code 1 received from tool '/usr/bin/bash' ##[debug]STDIO streams have closed for tool '/usr/bin/bash' ##[error]Bash exited with code '1'.

[scolladon]

@bharuk it could be because there is nothing to deploy incrementally (no changes).
In this case the force-app folder is not created.
You could check if the output/package/package.xml contains element by using grep:
grep -q '<types>' output/package/package.xml cf this comment

[bharuk]

`
What is the name of the 'type'. I already have the Salesforce code available inside that force-app directory. mandatory to create output directoy? what is the use of output directory. In which line to pass the grep grep -q 'types' output/package/package.xml'

please explain i am totally confusing.

[scolladon]

What I understand from the issue is that the deploy command try to deploy the content of output/force-app folder without success because it seems the output/force-app folder does not exist.
From what I read, the output folder exist as it is created just before.
So I was thinking maybe the force-app folder does not exist under the output folder.
It happens when sgd does not detect any change (even if -d parameter flag is set). When there are no sources to deploy, it does not copy any.
In order to detect if there are changes to deploy I was suggesting reading the package.xml. If it contains <types> tag then it means it contains elements. If you used the -d parameter then those elements should have been copied and you will be pretty sure it is deployable using force:source:deploy.

So, after having executed sgd to setup the incremental source content, and before deploying it using force:source:deploy, you could check if there is anything to deploy by checking if the package.xml contains any <types> tag (using grep).

[bharuk]

Any alternative method for delta deployment. still I am facing the issues. any python scripts for this delta deployment. why because we are planning multiple stages in azure devops yaml dev sit at uat preprod and prod. still i am getting intial build i am unable to deploy the package. when i am trying grep is no luck.

trigger: none

resources:
    webhooks:
    - webhook: gspdeltawebhook          ### Webhook alias
      connection: gspdeltawebhook
      filters:
        - path: pull_request.merged  
          value: true
        - path: pull_request.base.ref  
          value: "develop"              ##branch


    repositories:
    - repository: b2b-gsp-slfdelta  
      type: githubenterprise
      name: VFGroup-TaaS-Tenants/b2b-gsp-slfdelta
      ref: develop
      endpoint: Vodafone_GSP_Delta
     
  
variables:
    - name: devops_dir
      value: "/home/vsts/work/1/b2b-gsp-deltadevops"  
    - name: global_dir
      value: "/home/vsts/work/1/s/b2b-gsp-slfdelta"
    - group: vargroup-sfdx
stages:
  - stage: Creation
    jobs:
      - job: Build 
        timeoutInMinutes: 360
        pool:
         vmImage: ubuntu-latest
        
        steps:
          - checkout: self
            path: b2b-gsp-deltadevops
          - checkout: b2b-gsp-slfdelta
            fetchDepth: 0
          - task: DownloadSecureFile@1
            name: DevserverkeyFile
            displayName: download rsa key
            inputs:
             secureFile: 'Devserver.key'        
          - script: |
             echo installing the $(DevserverkeyFile.secureFilePath) to the trusted directory
          - bash:
              npm install -g npm
            displayName: Install Latest version of npm
          - bash: 
              npm install sfdx-cli --global
            displayName: Install SFDX
          - script: |
              echo y | sfdx plugins:install sfdx-git-delta
              sfdx plugins
            displayName: 'Installing SGD'
        #INSTALL VLOCITY BUILD TOOL
          - bash:
              npm install --global vlocity
            displayName: Install Vlocity
          - bash: |
                 cd $(devops_dir)
                 sfdx force:auth:jwt:grant --clientid $(DevhubclientId) --jwtkeyfile $(DevserverkeyFile.secureFilePath) --username $(DevhubUserName) --instanceurl $(DevhubInstanceURL)
            displayName: 'Authenticate'
          - script: |
              cd $(global_dir)
               #mkdir output
              sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output
            displayName: 'Generate incremental package'
          - script: |
              cd $(global_dir)
                #grep -q  output/package/package.xml && sfdx force:source:deploy -x output/package/package.xml
                sfdx force:source:deploy -x package/package.xml
              #sfdx force:source:deploy -p output/force-app -w 60 --targetusername $(DevhubUserName)
            displayName: 'Deploy package'

[scolladon]

Hi!

I see multiple potential root cause here:

1. For "displayName: 'Generate incremental package'" step: --output parameter does not have value. It should be:

mkdir output
sfdx sgd:source:delta --to "HEAD" --from "origin/develop" --output output

2. For "displayName: 'Deploy package'" step: deployment uses a package/package.xml. It should use output/package/package.xml

sfdx force:source:deploy -x output/package/package.xml

I hope it helps

[bharuk]

2023-05-11T12:57:17.8740582Z ##[debug]/usr/bin/bash arg: /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh
2023-05-11T12:57:17.8743881Z ##[debug]exec tool: /usr/bin/bash
2023-05-11T12:57:17.8744261Z ##[debug]arguments:
2023-05-11T12:57:17.8745658Z ##[debug] --noprofile
2023-05-11T12:57:17.8746510Z ##[debug] --norc
2023-05-11T12:57:17.8746860Z ##[debug] /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh
2023-05-11T12:57:17.8749822Z [command]/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/ebb9c158-9009-4f48-a258-00afe10ca9a9.sh
2023-05-11T12:57:19.4465795Z Warning: We plan to deprecate this command in the future. Try using the "project deploy start" command instead.
2023-05-11T12:57:19.5008330Z Error (1): No authorization information found for harvesttest.
2023-05-11T12:57:19.5256693Z ##[debug]Exit code 1 received from tool '/usr/bin/bash'
2023-05-11T12:57:19.5260955Z ##[debug]STDIO streams have closed for tool '/usr/bin/bash'
2023-05-11T12:57:19.5297977Z ##[error]Bash exited with code '1'.
2023-05-11T12:57:19.5303975Z ##[debug]Processed: ##vso[task.issue type=error;]Bash exited with code '1'.
2023-05-11T12:57:19.5304992Z ##[debug]task result: Failed
2023-05-11T12:57:19.5306760Z ##[debug]Processed: ##vso[task.complete result=Failed;done=true;]
2023-05-11T12:57:19.5328666Z ##[section]Finishing: CmdLine

above error i am getting no authorization information found

[scolladon]

Hi @bharuk

I would say the authentication step failed here, no ?
Have you made changes in the pipeline file you shared ?

What I can see in there is it seems to authenticate to a devhub but it does not create a scratch org
The authentication does not seems to set a default alias and the deploy does not specify alias to target the deployment.

I hope it helps