Branch protection rules are not working anymore.

[dgokcin]

Problem Description

I just created a new repository with the same config file that I use for all of my repositories and noticed that although the configs are the same, branch protection rules are not working. Any help is appreciated. To reproduce, you can try creating a new repo from the default settings in the README file. I am attaching my current config below

# These settings are synced to GitHub by https://probot.github.io/apps/settings/

repository:
  # See https://docs.github.com/en/rest/reference/repos#update-a-repository for all available settings.

  # The name of the repository. Changing this will rename the repository
  name: store-api

  # A short description of the repository that will show up on GitHub
  description: Store related apis

  # A URL with more information about the repository
  homepage: https://example.github.io/

  # A comma-separated list of topics to set on the repository
  topics: store, marketplace-integration, order

  # Either `true` to make the repository private, or `false` to make it public.
  private: true

  # Either `true` to enable issues for this repository, `false` to disable them.
  has_issues: false

  # Either `true` to enable projects for this repository, or `false` to disable them.
  # If projects are disabled for the organization, passing `true` will cause an API error.
  has_projects: false

  # Either `true` to enable the wiki for this repository, `false` to disable it.
  has_wiki: false

  # Either `true` to enable downloads for this repository, `false` to disable them.
  has_downloads: false

  # Updates the default branch for this repository.
  default_branch: develop

  # Either `true` to allow squash-merging pull requests, or `false` to prevent
  # squash-merging.
  allow_squash_merge: false

  # Either `true` to allow merging pull requests with a merge commit, or `false`
  # to prevent merging pull requests with merge commits.
  allow_merge_commit: true

  # Either `true` to allow rebase-merging pull requests, or `false` to prevent
  # rebase-merging.
  allow_rebase_merge: false

  # Either `true` to enable automatic deletion of branches on merge, or `false` to disable
  delete_branch_on_merge: false

  # Either `true` to enable automated security fixes, or `false` to disable
  # automated security fixes.
  enable_automated_security_fixes: false

  # Either `true` to enable vulnerability alerts, or `false` to disable
  # vulnerability alerts.
  enable_vulnerability_alerts: false

# Labels: define labels for Issues and Pull Requests
labels:
  - name: fix
    color: '#E1A296'
    description: A label used to mark fixes for automatic release notes generation.

  - name: feature
    # If including a `#`, make sure to wrap it with quotes!
    color: '#336699'
    description: A label used to mark features for automatic release notes generation.

# Collaborators: give specific users access to this repository.
# See https://docs.github.com/en/rest/reference/repos#add-a-repository-collaborator for available options
collaborators:
  # - username: bkeepers
  #   permission: push
  # - username: hubot
  #   permission: pull

  # Note: `permission` is only valid on organization-owned repositories.
  # The permission to grant the collaborator. Can be one of:
  # * `pull` - can pull, but not push to or administer this repository.
  # * `push` - can pull and push, but not administer this repository.
  # * `admin` - can pull, push and administer this repository.
  # * `maintain` - Recommended for project managers who need to manage the repository without access to sensitive or destructive actions.
  # * `triage` - Recommended for contributors who need to proactively manage issues and pull requests without write access.

# See https://docs.github.com/en/rest/reference/teams#add-or-update-team-repository-permissions for available options
teams:
  - name: CodeReview
    # The permission to grant the team. Can be one of:
    # * `pull` - can pull, but not push to or administer this repository.
    # * `push` - can pull and push, but not administer this repository.
    # * `admin` - can pull, push and administer this repository.
    # * `maintain` - Recommended for project managers who need to manage the repository without access to sensitive or destructive actions.
    # * `triage` - Recommended for contributors who need to proactively manage issues and pull requests without write access.
    permission: admin
  - name: DevTeam
    permission: push
  - name: PlatformTeam
    permission: admin

branches:
  - name: develop
    # https://docs.github.com/en/rest/reference/repos#update-branch-protection
    # Branch Protection settings. Set to null to disable
    protection:
      # Required. Require at least one approving review on a pull request, before merging. Set to null to disable.
      required_pull_request_reviews:
        # The number of approvals required. (1-6)
        required_approving_review_count: 1
        # Dismiss approved reviews automatically when a new commit is pushed.
        dismiss_stale_reviews: true
        # Blocks merge until code owners have reviewed.
        require_code_owner_reviews: true
        # Specify which users and teams can dismiss pull request reviews. Pass an empty dismissal_restrictions object to disable. User and team dismissal_restrictions are only available for organization-owned repositories. Omit this parameter for personal repositories.
        dismissal_restrictions:
          users: []
          teams: []
      # Required. Require status checks to pass before merging. Set to null to disable
      required_status_checks:
        # Required. Require branches to be up to date before merging.
        strict: true
        # Required. The list of status checks to require in order to merge into this branch
        contexts: ['Build & Test']
      # Required. Enforce all configured restrictions for administrators. Set to true to enforce required status checks for repository administrators. Set to null to disable.
      enforce_admins: null
      # Prevent merge commits from being pushed to matching branches
      required_linear_history: null
      # Required. Restrict who can push to this branch. Team and user restrictions are only available for organization-owned repositories. Set to null to disable.
      restrictions: null
  - name: main
    # https://docs.github.com/en/rest/reference/repos#update-branch-protection
    # Branch Protection settings. Set to null to disable
    protection:
      # Required. Require at least one approving review on a pull request, before merging. Set to null to disable.
      required_pull_request_reviews:
        # The number of approvals required. (1-6)
        required_approving_review_count: 1
        # Dismiss approved reviews automatically when a new commit is pushed.
        dismiss_stale_reviews: true
        # Blocks merge until code owners have reviewed.
        require_code_owner_reviews: true
        # Specify which users and teams can dismiss pull request reviews. Pass an empty dismissal_restrictions object to disable. User and team dismissal_restrictions are only available for organization-owned repositories. Omit this parameter for personal repositories.
        dismissal_restrictions:
          users: []
          teams: []
      # Required. Require status checks to pass before merging. Set to null to disable
      required_status_checks:
        # Required. Require branches to be up to date before merging.
        strict: true
        # Required. The list of status checks to require in order to merge into this branch
        contexts: ['Build & Test']
      # Required. Enforce all configured restrictions for administrators. Set to true to enforce required status checks for repository administrators. Set to null to disable.
      enforce_admins: null
      # Prevent merge commits from being pushed to matching branches
      required_linear_history: null
      # Required. Restrict who can push to this branch. Team and user restrictions are only available for organization-owned repositories. Set to null to disable.
      restrictions: null

[m-schrepel]

I am also having this issue with branch protection rules not being applied. I modified the settings.yml many times, checked the app installation many times and it is just not working.

[dgokcin]

any updates about this issue? it is a blocker on my way right now in using probot in all the repos in my organization. @travi sorry for mentioning you but I really want to know the status of this issue since it has been almost a month.

Thanks!

[travi]

sorry for the delay in investigating this, but i can confirm that branch protection itself is working correctly. however, as a result of #591, it is more likely that applying branch protection rules will be missed since it is now only attempted after successful application of the other sections.

this is especially unfortunate in the absence of #518, since it is quite difficult to understand a failure without running your own instance and having access to logs. if you are able to identify any problems with your configuration for the other sections, that is currently your best bet for unblocking branch protection settings from being applied. i hope to find some time to implement #518 soonish, which would provide far better insight into the details of config issues resulting in this blockage, but unfortunately i cannot make strong promises at this time.

[dgokcin]

@travi no problem. I do not understand what you meant by "applying branch protection rules will be missed since it is now only attempted after successful application of the other sections." you mean like I have something in my configuration which is causing a problem? in this case what do you think that might be?

thanks

[stevie-]

Is there a way to test a settings file? Like a github-action which parses the config in the same way as the probot-settings app.

[dgokcin]

@stevie- not that I know of. @travi my minimal configuration is prepared on top of the example in the repository README. I can confirm that only settings under branch protection rules are not working. All the other sections like repo description, labels, merge preferences etc. work.

How did you confirm that the branch protection rules were working on your test? Could you please share a config file?

Thanks

# These settings are synced to GitHub by https://probot.github.io/apps/settings/

repository:
  # See https://docs.github.com/en/rest/reference/repos#update-a-repository for all available settings.

  # The name of the repository. Changing this will rename the repository
  name: POCs

  # A short description of the repository that will show up on GitHub
  description: The source code for POCs.

  # A URL with more information about the repository
  homepage: https://example.github.io/

  # A comma-separated list of topics to set on the repository
  topics: poc, testing

  # Either `true` to make the repository private, or `false` to make it public.
  private: true

  # Either `true` to enable issues for this repository, `false` to disable them.
  has_issues: false

  # Either `true` to enable projects for this repository, or `false` to disable them.
  # If projects are disabled for the organization, passing `true` will cause an API error.
  has_projects: false

  # Either `true` to enable the wiki for this repository, `false` to disable it.
  has_wiki: false

  # Either `true` to enable downloads for this repository, `false` to disable them.
  has_downloads: false

  # Updates the default branch for this repository.
  default_branch: develop

  # Either `true` to allow squash-merging pull requests, or `false` to prevent
  # squash-merging.
  allow_squash_merge: false

  # Either `true` to allow merging pull requests with a merge commit, or `false`
  # to prevent merging pull requests with merge commits.
  allow_merge_commit: true

  # Either `true` to allow rebase-merging pull requests, or `false` to prevent
  # rebase-merging.
  allow_rebase_merge: false

  # Either `true` to enable automatic deletion of branches on merge, or `false` to disable
  delete_branch_on_merge: false

  # Either `true` to enable automated security fixes, or `false` to disable
  # automated security fixes.
  enable_automated_security_fixes: false

  # Either `true` to enable vulnerability alerts, or `false` to disable
  # vulnerability alerts.
  enable_vulnerability_alerts: false

branches:
  - name: develop
    # https://docs.github.com/en/rest/reference/repos#update-branch-protection
    # Branch Protection settings. Set to null to disable
    protection:
      # Required. Require at least one approving review on a pull request, before merging. Set to null to disable.
      required_pull_request_reviews:
        # The number of approvals required. (1-6)
        required_approving_review_count: 2
        # Dismiss approved reviews automatically when a new commit is pushed.
        dismiss_stale_reviews: true
        # Blocks merge until code owners have reviewed.
        require_code_owner_reviews: true
        # Specify which users and teams can dismiss pull request reviews. Pass an empty dismissal_restrictions object to disable. User and team dismissal_restrictions are only available for organization-owned repositories. Omit this parameter for personal repositories.
        dismissal_restrictions:
          users: []
          teams: []
      # Required. Require status checks to pass before merging. Set to null to disable
      required_status_checks:
        # Required. Require branches to be up to date before merging.
        strict: true
        # Required. The list of status checks to require in order to merge into this branch
        contexts: []
      # Required. Enforce all configured restrictions for administrators. Set to true to enforce required status checks for repository administrators. Set to null to disable.
      enforce_admins: null
      # Prevent merge commits from being pushed to matching branches
      required_linear_history: null
      # Required. Restrict who can push to this branch. Team and user restrictions are only available for organization-owned repositories. Set to null to disable.
      restrictions:
        apps: []
        users: []
        teams: []

[travi]

you mean like I have something in my configuration which is causing a problem?

yes, the most likely situation would be that a call to update one of the other sections in your config is failing. this would result in the call to update branch protection not happening because it is only attempted after all of the other sections complete successfully.

in this case what do you think that might be?

this is why i mentioned it being unfortunate that #518 is not in place, since that would be able to provide feedback about the failure. solving that is high on my list, but i can't promise a timeline.

in my case, i had an issue with a color code for a label with leading zeros and worked around that in travi-test/.github@17dde15. i also implemented a more official fix for this in https://github.com/repository-settings/app/releases/tag/v2.0.3. in addition, i had some conflicts between my vulnerability alerts and security fixes settings, so i defined both as enabled in travi-test/.github@9265ad3 and travi-test/.github@48fb2ca.

Could you please share a config file?

after making the above adjustments to my account-level config that is extended by my test project, i made the following change in order to re-trigger applying the project-level config and it applied the branch protection rules successfully: travi-test/settings-test@16499fc

my minimal configuration is prepared on top of the example in the repository README

i'll try to update my test config to be closer to yours when i get a chance and see if it results in any errors

[travi]

Is there a way to test a settings file? Like a github-action which parses the config in the same way as the probot-settings app.

there isn't currently, but it certainly could be valuable. a big part of the complexity with this is that most of the details to verify are not defined by the settings app, but rather by the APIs called by the settings app, which can change independently from the maintenance that happens within this app.

if something along these lines were to be implemented, it would need to automatically account for the expectations of the APIs. currently, i believe implementing #518 would provide better ROI, so i am prioritizing that over a file validator. i'm open to input if folks have ideas around solving the latter, though.

[travi]

@dgokcin i just copy/pasted the config from your comment into travi-test/settings-test@05de64c and it applied successfully.

i think i would recommend removing the other sections from your config file so that branches is the only section left. that should enable you to confirm if you have a problem with the branch protection section specifically or if the problem is coming from one of the other sections. assuming that confirms that branch protection is not the problem, you can then add the other sections back one by one, along with making a change to the branch protection at the same time. if the branch protection setting does not apply with adding one of the sections back to the file, that is the one causing the error.

i know this is not ideal, but it is the best i can recommend until #518 is completed

[dgokcin]

after a painful hour of debugging, I found the problematic line. the enable_automated_security_fixes: false line which is also present on the README but set to true has caused probot settings to fail processing at some point and eventually causing branch protection rules to fail. I am happy to say that removing that line fixed branch protection problems in all my repositories!

[dgokcin]

@travi thank you for your help, while going over your config, I noticed _extends: .github. how exactly does this work. could you please point me towards the correct documentation? I believe it allows inheriting config from a root repo?

[travi]

the enable_automated_security_fixes: false line which is also on the README has caused probot settings to fail processing

would you be interested in sending a PR to update that property to true in the readme? probably a better recommendation anyway and is the setting that worked for me. before i set both security alerts and vulnerability fixes to true (which is really the behavior i wanted anyway), i was seeing "Vulnerability alerts must be enabled to configure automated security fixes.", so there are some requirements around those being used together.

[travi]

I noticed _extends: .github. how exactly does this work. could you please point me towards the correct documentation?

this behavior is available from the probot framework used in this app and is mentioned in the readme for the project: https://github.com/repository-settings/app#inheritance

i'm going to hide this in this thread since it is a bit off topic, but feel free to open a new discussion if more clarification is needed on that topic