Azure DevOps branch protection status

[sereth]

Using Atlantis 0.10.2 with DevOps

In DevOps when branch protection is enabled for the target branch, the Atlantis status updates are not recognized by DevOps to fulfill the required status checks.

It looks like the status checks are currently being posted to the latest commit of the source branch of the pull request.

https://dev.azure.com/mplehmann/POC-Terraform/_apis/git/repositories/POC-Terraform/commits/fe9fb32bb9654b52b2e624d070fe0380c979ecc5/statuses?api-version=5.1

{
  "count": 2,
  "value": [
    {
      "id": 3117312,
      "state": "succeeded",
      "description": "1/1 projects planned successfully.",
      "context": {
        "name": "atlantis/plan",
        "genre": "Atlantis Bot"
      },
      "creationDate": "2019-12-11T05:55:47.03Z",
      "createdBy": {
        "displayName": "Atlantis Bot",
        "url": "https://spsprodcus3.vssps.visualstudio.com/A49a26c68-cf72-446b-b7e3-1d9548e6303b/_apis/Identities/94d969d0-336f-601e-a289-78e66a5f70ed",
        "_links": {
          "avatar": {
            "href": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.OTRkOTY5ZDAtMzM2Zi03MDFlLWEyODktNzhlNjZhNWY3MGVk"
          }
        },
        "id": "94d969d0-336f-601e-a289-78e66a5f70ed",
        "uniqueName": "atlantis-bot@mplehmann.com",
        "imageUrl": "https://dev.azure.com/mplehmann/_api/_common/identityImage?id=94d969d0-336f-601e-a289-78e66a5f70ed",
        "descriptor": "aad.OTRkOTY5ZDAtMzM2Zi03MDFlLWEyODktNzhlNjZhNWY3MGVk"
      }
    },
    {
      "id": 3117304,
      "state": "pending",
      "description": "Plan in progress...",
      "context": {
        "name": "atlantis/plan",
        "genre": "Atlantis Bot"
      },
      "creationDate": "2019-12-11T05:51:44.44Z",
      "createdBy": {
        "displayName": "Atlantis Bot",
        "url": "https://spsprodcus3.vssps.visualstudio.com/A49a26c68-cf72-446b-b7e3-1d9548e6303b/_apis/Identities/94d969d0-336f-601e-a289-78e66a5f70ed",
        "_links": {
          "avatar": {
            "href": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.OTRkOTY5ZDAtMzM2Zi03MDFlLWEyODktNzhlNjZhNWY3MGVk"
          }
        },
        "id": "94d969d0-336f-601e-a289-78e66a5f70ed",
        "uniqueName": "atlantis-bot@mplehmann.com",
        "imageUrl": "https://dev.azure.com/mplehmann/_api/_common/identityImage?id=94d969d0-336f-601e-a289-78e66a5f70ed",
        "descriptor": "aad.OTRkOTY5ZDAtMzM2Zi03MDFlLWEyODktNzhlNjZhNWY3MGVk"
      }
    }
  ]
}

In the Branches view, this does show the status information on the branch. However, the pull request doesn't recognize this as fulfilling the branch protection requirements. It looks like the status checks instead need to be posted to the pull request API URL. If I manually post the same status information to the pull request, it does pass the branch protection requirements.

https://dev.azure.com/mplehmann/POC-Terraform/_apis/git/repositories/POC-Terraform/pullRequests/9/statuses?api-version=5.1-preview.1

{
    "value": [
        {
            "id": 1,
            "state": "succeeded",
            "description": "1/1 projects planned successfully.",
            "context": {
                "name": "atlantis/plan",
                "genre": "Atlantis Bot"
            },
            "creationDate": "2019-12-11T06:30:27.5003658Z",
            "updatedDate": "2019-12-11T06:30:27.5003658Z",
            "createdBy": {
                "displayName": "Michael Lehmann",
                "url": "https://spsprodcus3.vssps.visualstudio.com/A49a26c68-cf72-446b-b7e3-1d9548e6303b/_apis/Identities/e6a43a8b-5325-4306-9944-7fd2b4ae168e",
                "_links": {
                    "avatar": {
                        "href": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh"
                    }
                },
                "id": "e6a43a8b-5325-4306-9944-7fd2b4ae168e",
                "uniqueName": "mplehmann@outlook.com",
                "imageUrl": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh",
                "descriptor": "aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh"
            }
        },
        {
            "id": 2,
            "state": "pending",
            "description": "apply pending.",
            "context": {
                "name": "atlantis/apply",
                "genre": "Atlantis Bot"
            },
            "creationDate": "2019-12-11T06:33:43.172303Z",
            "updatedDate": "2019-12-11T06:33:43.172303Z",
            "createdBy": {
                "displayName": "Michael Lehmann",
                "url": "https://spsprodcus3.vssps.visualstudio.com/A49a26c68-cf72-446b-b7e3-1d9548e6303b/_apis/Identities/e6a43a8b-5325-4306-9944-7fd2b4ae168e",
                "_links": {
                    "avatar": {
                        "href": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh"
                    }
                },
                "id": "e6a43a8b-5325-4306-9944-7fd2b4ae168e",
                "uniqueName": "mplehmann@outlook.com",
                "imageUrl": "https://dev.azure.com/mplehmann/_apis/GraphProfile/MemberAvatars/aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh",
                "descriptor": "aad.MzczY2U2ODUtOWEyNS03NzY3LWE3YjQtMWRjODVkNTI1Yzdh"
            }
        }
    ],
    "count": 2
}

[lkysow]

@mcdafydd does this sound right?

[mcdafydd]

I'll try and dig into this a bit as soon as I can. I should have time tonight or tomorrow afternoon. Thanks for the report!

[mcdafydd]

Hi All,

I went through the documentation about the Azure Devops Pull Request Status API and have started adding a new API call to go-azuredevops to fix the bug. I hadn't gotten to the point of needing this functionality but now I'm excited to start using it for greater collaboration.

It looks straightforward to fix the issue, but I'd like to get some feedback about whether we should always post the pull request status to each "iteration", or only when necessary. This is from the relevant section in the docs:

When the source branch in a PR changes, a new "iteration" is created to track the latest changes. Services that evaluate code changes will want to post new status on each iteration of a PR. Posting status to a specific iteration of a PR guarantees that status applies only to the code that was evaluated and none of the future updates.

At the very least, it looks like the Reset status whenever there are new changes feature will limit this functionality to posting on iterations. I'll start by working on a PR this weekend to always add the iterationId to the status update if that feature is mentioned in the webhook, and always ignoring it otherwise. I don't have enough insight about how this feature impacts the majority of Atlantis users, so I'm happy to adopt the community's feedback in the implementation. Let me know what you think!

[lkysow]

I think posting it to both (iteration and PR) makes sense. We'll have to carry that iteration ID through.

[sereth]

After playing with the status API a bit, I think always posting to the iteration is probably the best approach. However, it looks like this would essentially enforce the behavior of the Reset status whenever there are new changes setting regardless of the branch protection setting for the repository.

I found that if a status has been posted to both the current iteration and the pull request overall, the iteration status takes precedence. If a new commit then bumps the iteration, the pull request status then takes over because there isn't a matching status for the new iteration.

If we always post to the iteration, it would prevent a potential out-of-order status issue if a new commit is posted while Atlantis is processing a plan. For example:

1. Create pull request which triggers Atlantis plan (iteration 1)
2. While plan is in process, push an update to the pull request which would trigger a new plan (iteration 2)
3. The second plan fails because state is locked and posts failed status
4. The first plan finishes successfully and posts succeeded status

If we post only to the pull request, the atlantis/plan status on the pull request would show as succeeded because that was the last status received. If we instead post to the iteration, it would show as failed because that is the status tied to the current iteration.

[mcdafydd]

Awesome, thanks for all that detail.

I haven't found iterationId added to any of the webhook payloads, so right now I'm thinking about getting the current iteration ID by doing a List iterations and then finding the object with the matching source commit ref to the PR webhook. I may just be overlooking something obvious I hope.

Also, the "supportsIterations" flag can be set to true or false when a pull request is created:

If true, subsequent pushes to the pull request will be individually reviewable. Set this to false for large pull requests for performance reasons if this functionality is not needed.

All the PRs I'm opening in the browser seem to be set true by default. I'll check for that as well.

[sereth]

It doesn't look like you're missing anything. I found this was an issue that was reported to the DevOps feedback site and closed as Not a Bug. Using the pull request iterations API is the workaround that was suggested there as well.

[mcdafydd]

I'm just finishing up the tests and will get a PR submitted later today.

Cheers,
David

[mcdafydd]

Added #886