Automerge stuck with multiple repositories creating PRs around the same time

[posativ]

Description

I have notices a weird but reproducible misbehavior with the automerge functionality. I have a couple of repositories that I need to orchestrate for a combined release process (aka poly repo) with a script:

1. have main/dev branch + a simple workflow action,e.g. exit 0
2. create a release branch with changes
3. create PRs from that said release branch to main/dev
4. immediately trigger auto merge
5. repeat for up to 5 repositories

Repository settings: default, merge commit, no status checks or any branch protection (tested with many configurations).

With that workflow, the automerge feature gets stuck after the first repository almost always. To recover from this, I can simply add a success to that particular commit (via API). Alternatively, sleep 30 && exit 0 as workflow action resolves this issue too.

After diving into the code I suspect that there might an issue with the notification of commit statuses: the pull request might still check if it is mergeable and thus the automerge is not triggered.

This is also reproducible with a single repository and creating many branches + PRs + auto merge requests with a script. Roughly 50% of automergeable PRs get stuck with this:

Create a sample repository, e.g. I've cloned gitea/helm-gitea and used this repository with the script below.

env GITEA_HOST=git.example.com OWNER=My_User REPOSITORY=test-repo USERNAME=My_User PASSWORD=an_application_token ./reproduce-automerge-stuck.sh

#!/bin/bash
# filename: reproduce-automerge-stuck.sh

set -eu -o pipefail

GITEA_HOST="${GITEA_HOST:-demo.gitea.com}"

USERNAME="${USERNAME:-}"
PASSWORD="${PASSWORD:-}"

OWNER="${OWNER:-posativ}"
REPOSITORY="${REPOSITORY:-automerge-issue}"

if [[ -z "$USERNAME" || -z "$PASSWORD" ]]; then
  echo "Please set USERNAME and PASSWORD environment variables."
  exit 1
fi

automerge() {
  TEST_BRANCH="test-branch-$(date +%s)"
  TEMP_DIR=$(mktemp -d)
  trap 'rm -rf "$TEMP_DIR"' EXIT

  # clone + config
  git clone --depth 1 "https://${USERNAME}:${PASSWORD}@${GITEA_HOST}/${OWNER}/${REPOSITORY}" "$TEMP_DIR/repo"
  cd "$TEMP_DIR/repo"

  git config user.name "foo"
  git config user.email "bar@example.com"

  # Create a new branch and commit changes
  git checkout -b "$TEST_BRANCH"
  echo "$(date +%s)" > "$TEST_BRANCH.txt"
  git add "$TEST_BRANCH.txt"
  git commit -m "Add test file"

  sha=$(git rev-parse HEAD)

  git push origin "$TEST_BRANCH"

  # set commit status to pending
  curl -vX POST -H "Content-Type: application/json" \
    -u "$USERNAME:$PASSWORD" \
    -d '{"context": "foo", "description": "bar", "state": "pending", "target_url": "https://example.com"}' \
    "https://${GITEA_HOST}/api/v1/repos/${OWNER}/${REPOSITORY}/statuses/${sha}"

  # CI or something
  sleep 1

  # set commit status to success
  curl -vX POST -H "Content-Type: application/json" \
    -u "$USERNAME:$PASSWORD" \
    -d '{"context": "foo", "description": "bar", "state": "success", "target_url": "https://example.com"}' \
    "https://${GITEA_HOST}/api/v1/repos/${OWNER}/${REPOSITORY}/statuses/${sha}"

  # create pr
  index=$(curl -vX POST -H "Content-Type: application/json" \
    -u "$USERNAME:$PASSWORD" \
    -d "{\"head\": \"$TEST_BRANCH\", \"base\": \"main\", \"title\": \"Test PR\", \"body\": \"This is a test PR.\"}" \
    "https://${GITEA_HOST}/api/v1/repos/${OWNER}/${REPOSITORY}/pulls" | jq -r '.number')

  # create auto merge
  curl -vX POST -H "Content-Type: application/json" \
    -u "$USERNAME:$PASSWORD" \
    -d '{"do": "merge", "merge_when_checks_succeed": true}' \
    "https://${GITEA_HOST}/api/v1/repos/${OWNER}/${REPOSITORY}/pulls/${index}/merge"
}


automerge

Gitea Version

1.24

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

No response

Screenshots

Git Version

1.24.2

Operating System

No response

How are you running Gitea?

Official Gitea Helm Chart with Kubernetes + Valkey.

Database

PostgreSQL

[wxiaoguang]

Did you see some logs like this? Or some other related logs?

Head commit id of auto merge %-v does not match sha [%s], it may means the head branch has been updated. Just ignore this request because a new request expected in the queue

[wxiaoguang]

If you can build your own binary, could you try this fix? Start automerge check again after the conflict check #34989

I think one of the problems is that when a PR is being checked for conflicts, then the automerge check will be skipped, then the automerge will never happen. As you found:

After diving into the code I suspect that there might an issue with the notification of commit statuses: the pull request might still check if it is mergeable and thus the automerge is not triggered.

I added some code to start the automerge check again after the conflict check

[posativ]

I am able to build the docker image and deploy it for testing. Unfortunately #34989 does not fix the issue. My conclusion was not correct, after some log-style debugging I now think the issue is with the merge request via API when all status checks are already completed before triggering auto merge.

With the following patch (based on #34989 fix-automerge), I can resolve the issue:

diff --git a/services/automerge/automerge.go b/services/automerge/automerge.go
index 588ce0d195..3d5741df5d 100644
--- a/services/automerge/automerge.go
+++ b/services/automerge/automerge.go
@@ -65,6 +65,12 @@ func ScheduleAutoMerge(ctx context.Context, doer *user_model.User, pull *issues_
                _, err = issues_model.CreateAutoMergeComment(ctx, issues_model.CommentTypePRScheduledToAutoMerge, pull, doer)
                return err
        })
+
+       if scheduled && err == nil {
+               log.Trace("Pull request [%d] scheduled for auto merge with style [%s] and message [%s]", pull.ID, style, message)
+               automergequeue.StartPRCheckAndAutoMerge(ctx, pull)
+       }
+
        return scheduled, err
 }

I think I can reproduce that behavior on the demo instance.

---

Now this behavior is similar to the UI with no "auto-merge" available when the statuses are already green. However I think the API should either return an error in that case or do the auto merge instead.

[posativ]

I have edited the original script and how to reproduce is basically this:

1. create branch + push changes
2. add success status to that commit
3. create PR
4. schedule auto merge via API

Now the PR is seemingly "stuck" at auto merge because it could have been merged directly.

[wxiaoguang]

With the following patch (based on #34989 fix-automerge), I can resolve the issue:

Awesome, I will merge it into that fix and (and add co-author), and try to add some tests

[wxiaoguang]

I push a commit to #34989 : ab05810

I don't know your email address, so I only left the commit message as "Co-authored-by: posativ". If you'd like to use an email in the commit, feel free to tell me and I will add it in the final squash merge.

---

Now this behavior is similar to the UI with no "auto-merge" available when the statuses are already green. However I think the API should either return an error in that case or do the auto merge instead.

I think it's fine to trigger automerge and then merge after the schedule when the PR is merge-able. Indeed there should be no difference from "after the scheduled automerge then the PR becomes merge-able immediately and gets merged" IMO since it is really merge-able. What do you think?

[posativ]

Yes, I expect the automerge to just happen in this case.

No further attribution needed from my side, thank you for your help and work!