Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nil pointer error when manually running "atlantis plan" on Gitlab after upgrading Atlantis to 0.23.2 #3196

Closed
bakeemawaytoys opened this issue Mar 8, 2023 · 20 comments
Labels
bug Something isn't working never-stale provider/gitlab regression Bug introduced in a new version

Comments

@bakeemawaytoys
Copy link

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

After upgrading Atlantis to version 0.23.2, I ran into an issue where Atlantis would error out with a "invalid memory address or nil pointer dereference" exception.

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44e05c)
runtime/signal_unix.go:837 (0x44e02c)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)
github.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)
runtime/asm_amd64.s:1598 (0x46b200)

The specific scenario is manually triggering a plan operation because auto plan did not run. Looking at the code, I think the issue was introduced in #2979. Specifically, it was introduced at the following line.

ctx.PullRequestStatus, err = p.pullReqStatusFetcher.FetchPullStatus(pull)

When fetching the status of a Gitlab merge request, a nil pointer dereference exception is thrown if Atlantis has not added a commit status to the merge request's commit. This behavior is described in more detail in #1852. If auto plan runs, then the commit status will exist and the manual plan operation succeeds. If a project has a repo-level configuration file, then auto plan will fail to run because of #3122. If I manually trigger an apply operation, then the commit status will appear even though the apply operation will fail. After that, I can manually trigger a plan operation. Another scenario where auto plan doesn't run is when Atlantis is configured to ignore draft merge requests. Removing the draft status and manually triggering an auto plan will result in the nil pointer exception.

The combination of this issue and #3122 makes version 0.23 almost unusable on Gitlab. I've had to revert my deployment to version 0.22.3.

Reproduction Steps

The simplest way to reproduce the issue is to disable auto plan by configuring Atlantis to ignore draft merge requests.

  1. Deploy Atlantis with allow-draft-prs set to false.
  2. Create a new draft merge request in Gitlab.
  3. Remove the draft status.
  4. Comment atlantis plan

Logs

Here are logs for both the creation of a merge request and the manual run of atlantis plan. The merge request contained one change to a .tf file in the root of the project.


@timestamp @message
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"server/middleware.go:45","msg":"POST /events – from 172.31.5.135:11816","json":{}}
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"events/events_controller.go:109","msg":"handling GitLab post","json":{}}
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"events/events_controller.go:492","msg":"request valid","json":{}}
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"events/events_controller.go:496","msg":"handling as comment event","json":{}}
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"logging/simple_logger.go:157","msg":"Ignoring non-command comment: "**Error: goroutine panic. This is a bug...."","json":{}}
2023-03-07 18:00:14.371 {"level":"debug","ts":"2023-03-07T18:00:14.371Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}
2023-03-07 18:00:13.983 {"level":"error","ts":"2023-03-07T18:00:13.982Z","caller":"events/command_runner.go:427","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:260 (0x44e05c)\nruntime/signal_unix.go:837 (0x44e02c)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207 (0xce2f66)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xce8744)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xce9015)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174 (0xf12a37)\ngithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290 (0xf140eb)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xef86c3)\nruntime/asm_amd64.s:1598 (0x46b200)\n","json":{"repo":"devops/development-application-resources","pull":"164"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\truntime/panic.go:890\nruntime.panicmem\n\truntime/panic.go:260\nruntime.sigpanic\n\truntime/signal_unix.go:837\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:207\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:76\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:174\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:290\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"server/middleware.go:45","msg":"POST /events – from 172.31.5.135:11816","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"events/events_controller.go:109","msg":"handling GitLab post","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"events/events_controller.go:492","msg":"request valid","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"events/events_controller.go:496","msg":"handling as comment event","json":{}}
2023-03-07 18:00:13.576 {"level":"info","ts":"2023-03-07T18:00:13.576Z","caller":"events/events_controller.go:545","msg":"parsed comment as command="plan" verbose=false dir="" workspace="" project="" flags=""","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"events/events_controller.go:577","msg":"executing command","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"logging/simple_logger.go:157","msg":"Processing...","json":{}}
2023-03-07 18:00:13.576 {"level":"debug","ts":"2023-03-07T18:00:13.576Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}
2023-03-07 17:58:41.730 {"level":"debug","ts":"2023-03-07T17:58:41.730Z","caller":"events/working_dir.go:256","msg":"ran: git merge -q --no-ff -m atlantis-merge FETCH_HEAD. Output: ","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.730 {"level":"info","ts":"2023-03-07T17:58:41.730Z","caller":"events/project_command_builder.go:351","msg":"successfully parsed atlantis.yaml file","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.730 {"level":"debug","ts":"2023-03-07T17:58:41.730Z","caller":"events/project_finder.go:180","msg":"checking if project at dir "." workspace "default" was modified","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.730 {"level":"info","ts":"2023-03-07T17:58:41.730Z","caller":"events/project_command_builder.go:359","msg":"0 projects are to be planned based on their when_modified config","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.730 {"level":"info","ts":"2023-03-07T17:58:41.730Z","caller":"events/plan_command_runner.go:97","msg":"determined there was no project to run plan in","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.723 {"level":"debug","ts":"2023-03-07T17:58:41.723Z","caller":"events/working_dir.go:256","msg":"ran: git merge-base main FETCH_HEAD. Output: 3bace385c287f62f25034aaf3745c3ff8376de55","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.722 {"level":"debug","ts":"2023-03-07T17:58:41.722Z","caller":"events/working_dir.go:256","msg":"ran: git fetch head +refs/heads/test_gitlab_runner_15_9_ecr_cache_images:. Output: From https://gitlab.example.com/devops/development-application-resources\n * branch test_gitlab_runner_15_9_ecr_cache_images -> FETCH_HEAD\n * [new branch] test_gitlab_runner_15_9_ecr_cache_images -> head/test_gitlab_runner_15_9_ecr_cache_images","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.410 {"level":"debug","ts":"2023-03-07T17:58:41.410Z","caller":"events/working_dir.go:256","msg":"ran: git remote add head https://group_43_bot:@gitlab.example.com/devops/development-application-resources.git. Output: ","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.408 {"level":"debug","ts":"2023-03-07T17:58:41.408Z","caller":"events/working_dir.go:256","msg":"ran: git clone --branch main --single-branch https://group_43_bot:@gitlab.example.com/devops/development-application-resources.git /home/atlantis/.atlantis/repos/devops/development-application-resources/164/default. Output: Cloning into '/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default'...","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.059 {"level":"debug","ts":"2023-03-07T17:58:41.059Z","caller":"events/project_command_builder.go:284","msg":"0 files were modified in this pull request","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.059 {"level":"debug","ts":"2023-03-07T17:58:41.059Z","caller":"events/project_command_builder.go:326","msg":"got workspace lock","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:41.059 {"level":"info","ts":"2023-03-07T17:58:41.059Z","caller":"events/working_dir.go:224","msg":"creating dir "/home/atlantis/.atlantis/repos/devops/development-application-resources/164/default"","json":{"repo":"devops/development-application-resources","pull":"164"}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"server/middleware.go:45","msg":"POST /events – from 172.31.5.73:2274","json":{}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"events/events_controller.go:109","msg":"handling GitLab post","json":{}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"events/events_controller.go:492","msg":"request valid","json":{}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"events/events_controller.go:499","msg":"handling as pull request event","json":{}}
2023-03-07 17:58:40.918 {"level":"info","ts":"2023-03-07T17:58:40.918Z","caller":"events/events_controller.go:602","msg":"identified event as type "opened"","json":{}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"logging/simple_logger.go:157","msg":"Processing...","json":{}}
2023-03-07 17:58:40.918 {"level":"debug","ts":"2023-03-07T17:58:40.918Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}
2023-03-07 17:58:26.780 {"level":"debug","ts":"2023-03-07T17:58:26.780Z","caller":"server/middleware.go:45","msg":"POST /events – from 172.31.5.73:30530","json":{}}
2023-03-07 17:58:26.780 {"level":"debug","ts":"2023-03-07T17:58:26.780Z","caller":"events/events_controller.go:109","msg":"handling GitLab post","json":{}}
2023-03-07 17:58:26.780 {"level":"debug","ts":"2023-03-07T17:58:26.780Z","caller":"events/events_controller.go:492","msg":"request valid","json":{}}
2023-03-07 17:58:26.780 {"level":"debug","ts":"2023-03-07T17:58:26.780Z","caller":"logging/simple_logger.go:157","msg":"Ignoring unsupported event","json":{}}
2023-03-07 17:58:26.780 {"level":"debug","ts":"2023-03-07T17:58:26.780Z","caller":"server/middleware.go:72","msg":"POST /events – respond HTTP 200","json":{}}

Environment details

Atlantis Version: 0.23.2
Gitlab Version: 15.9.2 and 15.6.8

Atlantis server-side config file:

---

allow-draft-prs: true
automerge: true
checkout-strategy: merge

gitlab-hostname: gitlab.example.com

log-level: debug

silence-allowlist-errors: true
silence-no-projects: true
silence-vcs-status-no-plans: true

write-git-creds: true

Repo atlantis.yaml file:

---
version: 3
projects:
  - dir: .
    workflow: "production"
    terraform_version: "v1.3.9"

Server side config

---
repos:
  - id: "/.*/"
    allowed_overrides: [ workflow ]
    allow_custom_workflows: false
    apply_requirements: [ "approved", "mergeable", "undiverged" ]
    delete_source_branch_on_merge: true
workflows:
  default:
    plan:
      steps:
        - run: echo "The default workflow is not supported.  Please select from 'development', 'production', 'staging', or 'uat'."
        - run: return 1
    apply:
      steps:
        - run: return 1

  development:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: development
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  production:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: production
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  staging:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: staging
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply
  uat:
    plan:
      steps:
        - env:
            name: ENVIRONMENT_NAME
            value: uat
        - run: "/usr/local/bin/initialize.sh"
        - run: "/usr/local/bin/validate.sh"
        - plan
    apply:
      steps:
        - apply

Additional Context

@bakeemawaytoys bakeemawaytoys added the bug Something isn't working label Mar 8, 2023
@bakeemawaytoys
Copy link
Author

I should clarify that I don't know the expected behavior of auto plan when the draft status is removed from a merge request. What I observed with Atlantis 0.21 is that auto plan would not run after the draft status was removed. I had to manually trigger a plan operation.

@jamengual
Copy link
Contributor

@lyoung-confluent @krrrr38 what do you think?

@ryansk
Copy link

ryansk commented Mar 14, 2023

We see this same issue. I will note that when we get the nil pointer error, closing and re-opening the merge request often allows Atlantis to function normally. Though, no matter what we do, it's not auto planning when opening an MR. We have to manually trigger it.

{"level":"info","ts":"2023-03-14T14:17:11.237Z","caller":"events/project_command_builder.go:351","msg":"successfully parsed atlantis.yaml file","json":{"repo":"repo/name","pull":"25"}}
{"level":"info","ts":"2023-03-14T14:17:11.237Z","caller":"events/project_command_builder.go:359","msg":"0 projects are to be planned based on their when_modified config","json":{"repo":"repo/name","pull":"25"}}
{"level":"info","ts":"2023-03-14T14:17:11.237Z","caller":"events/plan_command_runner.go:97","msg":"determined there was no project to run plan in","json":{"repo":"repo/name","pull":"25"}}

@jseiser
Copy link

jseiser commented Mar 18, 2023

@ryansk

Are you sure the auto-plan isnt failing, because gitlab changed how their API is returning changed ?

Link

@nitrocode nitrocode added the regression Bug introduced in a new version label Mar 18, 2023
@jseiser
Copy link

jseiser commented Mar 20, 2023

I should clarify that I don't know the expected behavior of auto plan when the draft status is removed from a merge request

@nitrocode

Any chance you could clarify this? When a Draft MR is marked ready I would expect an auto-plan to take place.

@nitrocode
Copy link
Member

It would make sense if you set a PR to ready from draft, that it would autoplan automatically.

Additional thoughts @runatlantis/maintainers ?

@jamengual
Copy link
Contributor

I guess Atlantis is expecting a push to happens so it does not know of the status change

@nitrocode
Copy link
Member

Looks like it's part of the pull request webhook permissions so atlantis is probably getting the event and not auto planning on it.

image

Some references

Events: []string{"issue_comment", "pull_request", "pull_request_review", "push"},

Events: []string{"issue_comment", "pull_request", "push"},

Events: []string{
"check_run",
"create",
"delete",
"issue_comment",
"issues",
"pull_request_review_comment",
"pull_request_review",
"pull_request",
"push",
},

It might be as easy as adding ready_for_review in the github_app_controller.go

https://docs.github.com/en/webhooks-and-events/events/issue-event-types#ready_for_review

We might want to add reopened as well but unsure if it already plans on that. I do not see it as an event type and have not tested it.

@jseiser
Copy link

jseiser commented Mar 20, 2023

If you guys do not have access to gitlab, I have an env I can test from.

@nitrocode
Copy link
Member

Unfortunately, while ready_for_review does seem to be detectable in github, it doesn't seem like gitlab sends a webhook when the status of the pr is changing from draft to ready.

https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html

There is a draft key in the json that gets toggled to true. I'm unsure how we would detect the difference between a commit pushed event and a ready mr event without a specific event_type associated with a ready pr. Open to any ideas of course

@jseiser
Copy link

jseiser commented Mar 20, 2023

Ive opened a support request with Gitlab to see what they say.

@jseiser
Copy link

jseiser commented Mar 21, 2023

https://gitlab.com/gitlab-org/gitlab/-/issues/398224

Thats the support request

@bakeemawaytoys
Copy link
Author

I finally had some time to revisit this issue. I thing I haven't mentioned yet is that I added autoplan settings to my repo-level Atlantis files. I set enabled to true and set the file pattern to auto plan for all file changes.

---
version: 3
projects:
  - dir: .
    workflow: "production"
    terraform_version: "v1.3.9"
    autoplan:
      when_modified: ["**/*"]
      enabled: true

Auto plan still did not run on 0.22 or 0.23.3 with those settings. Today, I upgraded Atlantis to 0.23.4. Auto plan is now working correctly when the autoplan settings are present in the repo-level configuration file. I assume that #3272 and possibly #3277 are the reason it is working now. There are still certain scenarios where the nil pointer error occurs. For example, if I remove or limit the when_modified pattern(s) and then create an MR with changes that don't match the pattern, the commit status is not created. If I add an atlantis plan comment to the merge request, the nil pointer error is triggered. The error still occurs in the draft status change scenario but I expected that based on the discussion above.

My Gitlab installation is on version 15.9.4 now for what it is worth. It'll be upgraded to 15.10 in the very near future.

@lanzrein
Copy link

Hello,

I have also noticed this happens however in my experience this happens when someone issues an atlantis plan command on a MR where there is no pipeline which has run. I have tested on a personal repository on gitlab.com found here : https://gitlab.com/lanzrein/test-some-stuff
I have created an MR and have not configured any pipeline. Then when querying the API (https://gitlab.com/api/v4/projects/35657329/merge_requests/2).
I get a JSON output like this :

{
  "id": 222683528,
  "iid": 2,
  "project_id": 35657329,
  "title": "Draft: test2",
  "description": "",
  "state": "opened",
  [...]
  "pipeline": null,
  "head_pipeline": null,
  [...]
}

We see that head_pipeline is null.

To replicate this issue in a setting where you use atlantis, you can create a MR and set it to Draft status. Then no pipeline will run. Afterwards, when running an atlantis plan command, you will get the nil pointer error due to the line here :

statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil)

where mr.HeadPipeline is null so we get the nil pointer error.

There was a PR #1981 that suggested to add a mechanism to detect when this is null to avoid this error which was refused because it the behavior of the Gitlab API was not very clear on this type of scenario. I have opened a support ticket asking them to explain if this behavior is intended. If it is intended maybe the proposed PR should be implemented ?

@jamengual
Copy link
Contributor

jamengual commented May 10, 2023 via email

@nitrocode
Copy link
Member

The original pr author deleted the branch so it cannot be reopened cc @marceloboeira

@jamengual
Copy link
Contributor

he offered to bring it back if we were ok with it, we can wait for his answer.

@lanzrein
Copy link

I digged a bit more and it seems there is an Epic to bring the entire Rest API to Open API v2 but there is no timeline.
https://gitlab.com/groups/gitlab-org/-/epics/8926

If there is any way I can help let me know. :)

@marceloboeira
Copy link
Contributor

@nitrocode let me see if I can bring it back

@jseiser
Copy link

jseiser commented May 26, 2023

Gitlab appears to have closed the other issues, and then opened a new one: https://gitlab.com/gitlab-org/gitlab/-/issues/321268

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working never-stale provider/gitlab regression Bug introduced in a new version
Projects
None yet
Development

No branches or pull requests

7 participants