Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime error: invalid memory address or nil pointer dereference #1852

Open
ghost opened this issue Oct 12, 2021 · 30 comments
Open

runtime error: invalid memory address or nil pointer dereference #1852

ghost opened this issue Oct 12, 2021 · 30 comments
Labels
bug Something isn't working never-stale waiting-on-response Waiting for a response from the user

Comments

@ghost
Copy link

ghost commented Oct 12, 2021

Hi,

I am trying to integrate atlantis (v0.17.4) with gitlab (free self managed ver. 11.9.6).

It is working fine till autoplan Comment output:
Ran Plan for project: preprod dir: environments/preprod workspace: default

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

  # module.eks_1.null_resource.test0 will be created
+ resource "null_resource" "test0" {
      + id = (known after apply)
    }

  # module.eks_1.null_resource.test1 will be created
+ resource "null_resource" "test1" {
      + id = (known after apply)
    }

Plan: 2 to add, 0 to change, 0 to destroy.
  • ▶️ To apply this plan, comment:

    • atlantis apply -p preprod
  • 🚮 To delete this plan click ***

  • 🔁 To plan this project again, comment:

    • atlantis plan -p preprod
  • ⏩ To apply all unapplied plans from this pull request, comment:

    • atlantis apply
  • 🚮 To delete all plans and locks for the PR, comment:

    • atlantis unlock

But fails when I do "atlantis apply -p preprod":
Error: goroutine panic. This is a bug.

runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:221 (0x449106)
/usr/local/go/src/runtime/signal_unix.go:735 (0x4490d6)
/home/circleci/project/server/events/vcs/gitlab_client.go:208 (0x96c66a)
/home/circleci/project/server/events/vcs/proxy.go:72 (0x96dbd4)
/home/circleci/project/server/events/apply_command_runner.go:101 (0xe436b4)
/home/circleci/project/server/events/command_runner.go:214 (0xe476a3)
/usr/local/go/src/runtime/asm_amd64.s:1581 (0x463f40)
Following is the request body sent by gitlab
{
  "object_kind": "note",
  "event_type": "note",
  "user": {
    "name": "***",
    "username": "***",
    "avatar_url": "***/uploads/-/system/user/avatar/503/avatar.png"
  },
  "project_id": 953,
  "project": {
    "id": 953,
    "name": "terraform",
    "description": "",
    "web_url": "***",
    "avatar_url": null,
    "git_ssh_url": "***",
    "git_http_url": "***",
    "namespace": "***",
    "visibility_level": 0,
    "path_with_namespace": "***",
    "default_branch": "master",
    "ci_config_path": null,
    "homepage": "***",
    "url": "***",
    "ssh_url": "***",
    "http_url": "***"
  },
  "object_attributes": {
    "attachment": null,
    "author_id": 503,
    "change_position": null,
    "commit_id": null,
    "created_at": "2021-10-12 15:07:52 UTC",
    "discussion_id": "82fd2f394e41ef45062f5b20021d35caa8eea8af",
    "id": 117876,
    "line_code": null,
    "note": "atlantis apply -p preprod",
    "noteable_id": 33736,
    "noteable_type": "MergeRequest",
    "original_position": null,
    "position": null,
    "project_id": 953,
    "resolved_at": null,
    "resolved_by_id": null,
    "resolved_by_push": null,
    "st_diff": null,
    "system": false,
    "type": null,
    "updated_at": "2021-10-12 15:07:52 UTC",
    "updated_by_id": null,
    "description": "atlantis apply -p preprod",
    "url": "***/terraform/merge_requests/7#note_117876"
  },
  "repository": {
    "name": "terraform",
    "url": "***",
    "description": "",
    "homepage": "***/terraform"
  },
  "merge_request": {
    "assignee_id": null,
    "author_id": 503,
    "created_at": "2021-10-12 15:06:48 UTC",
    "description": "",
    "head_pipeline_id": 15358,
    "id": 33736,
    "iid": 7,
    "last_edited_at": null,
    "last_edited_by_id": null,
    "merge_commit_sha": null,
    "merge_error": null,
    "merge_params": {
      "force_remove_source_branch": "0"
    },
    "merge_status": "can_be_merged",
    "merge_user_id": null,
    "merge_when_pipeline_succeeds": false,
    "milestone_id": null,
    "source_branch": "div-fb",
    "source_project_id": 953,
    "state": "opened",
    "target_branch": "master",
    "target_project_id": 953,
    "time_estimate": 0,
    "title": "check1",
    "updated_at": "2021-10-12 15:07:52 UTC",
    "updated_by_id": null,
    "url": "***/terraform/merge_requests/7",
    "source": {
      "id": 953,
      "name": "terraform",
      "description": "",
      "web_url": "***/terraform",
      "avatar_url": null,
      "git_ssh_url": "***",
      "git_http_url": "***",
      "namespace": "***",
      "visibility_level": 0,
      "path_with_namespace": "***/terraform",
      "default_branch": "master",
      "ci_config_path": null,
      "homepage": "***/terraform",
      "url": "***/terraform.git",
      "ssh_url": "***/terraform.git",
      "http_url": "***"
    },
    "target": {
      "id": 953,
      "name": "terraform",
      "description": "",
      "web_url": "***/terraform",
      "avatar_url": null,
      "git_ssh_url": "***/terraform.git",
      "git_http_url": "***/terraform.git",
      "namespace": "***",
      "visibility_level": 0,
      "path_with_namespace": "***/terraform",
      "default_branch": "master",
      "ci_config_path": null,
      "homepage": "***/terraform",
      "url": "***/terraform.git",
      "ssh_url": "***/terraform.git",
      "http_url": "***/terraform.git"
    },
    "last_commit": {
      "id": "5facef328360f37b26da3268d9cdc438fcc30cd4",
      "message": "check1\n",
      "timestamp": "2021-10-12T15:02:49Z",
      "url": "***/terraform/commit/5facef328360f37b26da3268d9cdc438fcc30cd4",
      "author": {
        "name": "***",
        "email": "***"
      }
    },
    "work_in_progress": false,
    "total_time_spent": 0,
    "human_total_time_spent": null,
    "human_time_estimate": null
  }
}
Installation repoConfig
repoConfig: |
  ---
  repos:
  - id: ***/terraform
    # apply_requirements: [mergeable]
    allowed_overrides: [apply_requirements, workflow, delete_source_branch_on_merge]
    allowed_workflows: [preprod_workflow]
    delete_source_branch_on_merge: true

  # workflows lists server-side custom workflows
  workflows:
    preprod_workflow:
      plan:
        steps:
        - init:
            extra_args: ["-input=false", "-no-color"]
        - plan:
            extra_args: ["-input=false", "-no-color", "-refresh=false", "-parallelism=30"]
      apply:
        steps:
        - apply:
            extra_args: ["-no-color", "-input=false", "-parallelism=30"]
atlantis.yaml at repo's root
version: 3
automerge: true
projects:
- name: preprod
  dir: ./environments/preprod
  workspace: default
  terraform_version: v1.0.2
  autoplan:
    when_modified: ["./*.tf"]
    enabled: true
  workflow: preprod_workflow

Thanks for any help!

@blytheaw
Copy link

blytheaw commented Oct 13, 2021

I believe this is the same issue I am having, which is reported here: #1804

This is really disrupting our workflow and it is unclear what we can do about it.

@gai6948
Copy link

gai6948 commented Nov 30, 2021

Having same issue, using TF Cloud as backend

@marceloboeira
Copy link
Contributor

I have had the same issue. #1931 doesn't really fix it... updating didn't change anything.

@sapslaj
Copy link
Contributor

sapslaj commented Dec 31, 2021

I have had the same issue. #1931 doesn't really fix it... updating didn't change anything.

@marceloboeira Hmm is it the same stack trace? There could be another nil pointer hiding somewhere that I can't reproduce.

@marceloboeira
Copy link
Contributor

Yeap:

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x449106)
runtime/signal_unix.go:735 (0x4490d6)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98192a)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0x982eb4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x983654)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xe505f4)
github.com/runatlantis/atlantis/server/events/command_runner.go:252 (0xe54a03)
runtime/asm_amd64.s:1581 (0x463fa0)

@marceloboeira
Copy link
Contributor

@sapslaj I realised that this started to happen after we've updated gitlab recently, but interestingly enough it doesn't happen to all repos with terraform we have.

@sapslaj
Copy link
Contributor

sapslaj commented Jan 3, 2022

Ahh yes. I was testing with GitHub instead of GitLab so this might be a different issue unrelated to what I fixed then. My bad for linking my PR to this issue without taking a closer look.

It looks like it's happening somewhere in PullIsMergable. I wonder if something changed with the GitLab update? It looks like Atlantis is on the latest go-gitlab version at least.

@marceloboeira
Copy link
Contributor

marceloboeira commented Jan 6, 2022

Yes, so we kind of found a way to overpass this bug by creating an empty pipeline on the repo. It seems that on line 208:

statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil)

the mr.HeadPipeline could be empty, leading to the invalid memory address or nil pointer dereference, since the .SHA property is expected.

I didn't have much time to check what could be an alternative here, maybe using pull.HeadCommit instead...

However, it could be other dependencies of mr.HeadPipeline like line 221 also have that issue...

isPipelineSkipped := mr.HeadPipeline.Status == "skipped"

@marceloboeira
Copy link
Contributor

A more complete log + stack trace overview:

{"level":"info","ts":"2022-01-06T14:05:29.474Z","caller":"events/events_controller.go:417","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"monitors\" flags=\"\"","json":{}}

{"level":"warn","ts":"2022-01-06T14:05:30.069Z","caller":"events/apply_command_runner.go:97","msg":"unable to update commit status: POST $$$REDACTED$$$: 403 {message: 403 Forbidden}","json":{"repo":"%%","pull":"$$"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:97\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:252"}

{"level":"error","ts":"2022-01-06T14:05:30.965Z","caller":"events/command_runner.go:377","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:221 (0x449106)\nruntime/signal_unix.go:735 (0x4490d6)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98192a)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0x982eb4)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x983654)\ngithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xe505f4)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:252 (0xe54a03)\nruntime/asm_amd64.s:1581 (0x463fa0)\n","json":{"repo":" $$$REDACTED$$$","pull":" $$$REDACTED$$$"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:377\nruntime.gopanic\n\truntime/panic.go:1038\nruntime.panicmem\n\truntime/panic.go:221\nruntime.sigpanic\n\truntime/signal_unix.go:735\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:105\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:252"}

github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
	github.com/runatlantis/atlantis/server/events/apply_command_runner.go:97
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
	github.com/runatlantis/atlantis/server/events/command_runner.go:252
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics
	github.com/runatlantis/atlantis/server/events/command_runner.go:377
runtime.gopanic
	runtime/panic.go:1038
runtime.panicmem
	runtime/panic.go:221
runtime.sigpanic
	runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
	github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
	github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
	github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
	github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
	github.com/runatlantis/atlantis/server/events/command_runner.go:252

marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Jan 6, 2022
In this particular example, `mr.HeadPipeline.SHA` panics on a nil
pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit
status.

```
runtime.gopanic
	runtime/panic.go:1038
runtime.panicmem
	runtime/panic.go:221
runtime.sigpanic
	runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
	github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
	github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
	github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
	github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
	github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull,
and guess that the pipeline was "skipped" unless the HeadPipeline is
there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this
results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Jan 6, 2022
In this particular example, `mr.HeadPipeline.SHA` panics on a nil
pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit
status.

```
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull,
and guess that the pipeline was "skipped" unless the HeadPipeline is
there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this
results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
@marceloboeira
Copy link
Contributor

marceloboeira commented Jan 6, 2022

After a lot of debugging, it seems that the root cause is a broken permission token + gitlab updating.

In resume, we have a user for Atlantis with a token created a long time ago. Over gitlab updates, this token doesn't work the same way a new token, created with the same permissions.

func (g *GitlabClient) PullIsMergeable(repo models.Repo, pull models.PullRequest) (bool, error) {
mr, _, err := g.Client.MergeRequests.GetMergeRequest(repo.FullName, pull.Num, nil)
if err != nil {
return false, err
}

This code doesn't error, yet it results on a broken mr struct.

The HeadPipeline is nil, which makes a problem for mr.HeadPipeline.SHA, I believe it is caused by the lack of permissions for the token to create/run the pipeline (might be the feature permissions changed but the token didn't?)

statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil)
if err != nil {
return false, err
}

Running locally, I could simply replace the code to run pull.HeadCommit and it has the same effect, however, as mentioned before, the other line using HeadPipeline fails:

isPipelineSkipped := mr.HeadPipeline.Status == "skipped"

runtime error: invalid memory address or nil pointer dereference
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/panic.go:221 (0x1048d26)
	panicmem: panic(memoryError)
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/signal_unix.go:735 (0x1048cf6)
	sigpanic: panicmem()
/atlantis/server/events/vcs/gitlab_client.go:221 (0x15805d7)
	(*GitlabClient).PullIsMergeable: isPipelineSkipped := mr.HeadPipeline.Status == "skipped"
/atlantis/server/events/vcs/proxy.go:72 (0x1581994)
	(*ClientProxy).PullIsMergeable: return d.clients[repo.VCSHost.Type].PullIsMergeable(repo, pull)
/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0x1582134)
	(*pullReqStatusFetcher).FetchPullStatus: mergeable, err := f.client.PullIsMergeable(repo, pull)
/atlantis/server/events/apply_command_runner.go:105 (0x1a4e6d4)
	(*ApplyCommandRunner).Run: ctx.PullRequestStatus, err = a.pullReqStatusFetcher.FetchPullStatus(baseRepo, pull)
/atlantis/server/events/command_runner.go:252 (0x1a52ae3)
	(*DefaultCommandRunner).RunCommentCommand: cmdRunner.Run(ctx, cmd)
/usr/local/Cellar/go/1.17.1/libexec/src/runtime/asm_amd64.s:1581 (0x1063e60)
	goexit: BYTE	$0x90	// NOP

In this case, the pipeline is skipped so we could simply:

	isPipelineSkipped := mr.HeadPipeline == nil || mr.HeadPipeline.Status == "skipped"

I have opened #1981 as a way to go around this and avoid the runtime panic...

@marceloboeira
Copy link
Contributor

For people coming into this issue, if the stack trace is the same, you might've updated GitLab recently which is one of the possible causes for this issue.

Solution: create a new token for atlantis with the api permissions and try that. It should go around this issue!

@narroyog
Copy link

The error on atlantis:

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x449366)
runtime/signal_unix.go:735 (0x449336)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208 (0x98598a)

after some troubleshooting using the same gitlab project, branch and commit id with two separate merge request i got two distinct results calling the GitLab API: https://gitlab.com/api/v4/projects/:id/merge_requests/:iid for the field that is used by the go-gitlab module to retrieve the value for mr.HeadPipeline

...
    "first_deployed_to_production_at": null,
    "pipeline": null,
    "head_pipeline": null,
    "diff_refs": {
        "base_sha": "844 ...

and

    "first_deployed_to_production_at": null,
    "pipeline": {
        "id": 47 ...
    },
    "head_pipeline": {
        "id": 47 ...

Which means there is not issue with the token used because i'm using the same one and i got the two results, one with data on head_pipeline field and the other one with null (that causes the panic error on atlantis)

This discards that there is an issue on the module used to access the Gitlab API (github.com/xanzy/go-gitlab) that is doing the expected work or issues on permission access caused by the token.

IF the row 208 requires the SHA of the commit for the head_pipeline, maybe that value can be found under diff_refs property head_sha

@lhw
Copy link

lhw commented Mar 30, 2022

I observed the same issue with our gitlab instance after updating atlantis to 0.19.2. I downgraded back to 0.17.4, the version we had before the update, and everything currently works again. So while its almost definitely an issue with the gitlab api, older versions do seem to be fine with the api issue.

@jamengual jamengual added the bug Something isn't working label Aug 26, 2022
@jamengual
Copy link
Contributor

is this still happening with v0.19.8?

@jamengual jamengual added the waiting-on-response Waiting for a response from the user label Aug 26, 2022
@github-actions github-actions bot added the Stale label Oct 16, 2022
@marceloboeira
Copy link
Contributor

@jamengual yes, the patch I wrote wasn't accepted so if it wasn't fixed in the gitlab client it can (and does) still happen. We recently updated gitlab on our side and started getting this error again.

#1981

@github-actions github-actions bot removed the Stale label Oct 22, 2022
@c0da
Copy link

c0da commented Jan 26, 2023

In my case, the problem disappeared after changing the role of the access token from "reporter" to "developer". I don't know if it has anything to do with the error, but I hope it helps someone!

@marceloboeira
Copy link
Contributor

@c0da for us it was always developer, but somehow we had to renew tokens for every gitlab update...

@github-actions
Copy link

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

@github-actions github-actions bot added the Stale label Mar 14, 2023
@1337andre
Copy link
Contributor

following this workaround helped me out
#2606 (comment)

$ cat .gitlab-ci.yml
dummy:
  script:
    - echo "dummy job for atlantis to trigger pipelineID"

@nitrocode nitrocode removed the Stale label Mar 14, 2023
@guillermotti
Copy link

For me removing the .terraform.lock.hcl solved the issue.

@trozborilCA
Copy link

I'm running into the same issue using atlantis v0.23.3, and gitlab version 15.9. I've rotated access tokens, removed the .terraform.lock.hcl file, attempted the gitlab-ci.yml workaround - all with no success. Here is the stack trace I'm receiving:

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44e07c)
runtime/signal_unix.go:837 (0x44e04c)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:225 (0xd2d9a6)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xd33184)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xd33a55)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xf3f6b4)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xf44d43)
runtime/asm_amd64.s:1598 (0x46b220)

Any other suggestions?

@jamengual
Copy link
Contributor

it looks like as @marceloboeira said even updating the library did not fixed anything and it broke more than just this.....we will evaluate if this + other fixes are needed and report back but we currently do not have anyone actively working on this.

@marceloboeira
Copy link
Contributor

@jamengual I used to have a "safety check" PR open for this:
#1981

Let me know if it makes sense to rebase/reopen that one.

@github-actions
Copy link

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

@github-actions github-actions bot added the Stale label Apr 29, 2023
@trozborilCA
Copy link

From what I've gathered, Atlantis is not compatible with GitLab until gitlab fixes their API response.

@github-actions
Copy link

github-actions bot commented Jun 2, 2023

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

@peimanja
Copy link

peimanja commented Jun 5, 2023

Seeing the same issue on Github enterprise server 3.8.x. In my case looks like it happens if the PR does not meet the mergable requirements and the user trying to run atlantis apply:

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:260 (0x44eb55)
runtime/signal_unix.go:835 (0x44eb25)
github.com/runatlantis/atlantis/server/events/vcs/github_client.go:431 (0xcd322b)
github.com/runatlantis/atlantis/server/events/vcs/github_client.go:500 (0xcd37f7)
github.com/runatlantis/atlantis/server/events/vcs/instrumented_client.go:185 (0xcda60b)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:76 (0xcdc2a4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:32 (0xcdcb75)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105 (0xece554)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xed36a3)
runtime/asm_amd64.s:1594 (0x469ac0)

@Mazorius
Copy link

Mazorius commented Jun 13, 2023

Is there a timeline when this will be fixed in the gitlab-client api of Atlantis?

@nitrocode
Copy link
Member

@Mazorius

There is no timeline. This project is based on community contributed PRs and it's been decided that we cannot merge #3428 without tests. If you'd like to contribute unit tests to the existing pr 3428, that would be the most helpful way to accelerate fixing this issue.

marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Aug 4, 2023
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Aug 4, 2023
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Aug 4, 2023
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
marceloboeira added a commit to marceloboeira/atlantis that referenced this issue Aug 4, 2023
In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
GenPage pushed a commit that referenced this issue Aug 4, 2023
…ty (#3428)

In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- #1852
@mars64
Copy link

mars64 commented Aug 12, 2023

fwiw, i ran into this issue on atlantis 0.24.4 on Gitlab SaaS: 16.3.0-pre 2719a0163a1

I'm not sure precisely what fixed it -- maybe all the things! I changed the three things I found in this thread:

I'm back on the road for now. Thank you to all who shared their experiences.

ijames-gc pushed a commit to gocardless/atlantis that referenced this issue Feb 13, 2024
…ty (runatlantis#3428)

In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
ijames-gc pushed a commit to gocardless/atlantis that referenced this issue Feb 13, 2024
…ty (runatlantis#3428)

In this particular example `mr.HeadPipeline.SHA` panics on a nil pointer dereference because HeadPipeline is empty.

This seems to be caused by the lack of permission to update the commit status.

```go
runtime.gopanic
        runtime/panic.go:1038
runtime.panicmem
        runtime/panic.go:221
runtime.sigpanic
        runtime/signal_unix.go:735
github.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:208
github.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable
        github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72
github.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus
        github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28
github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run
        github.com/runatlantis/atlantis/server/events/apply_command_runner.go:105
github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand
        github.com/runatlantis/atlantis/server/events/command_runner.go:252
```

The least invasive solution is to simply use the commit-hash from pull and guess that the pipeline was "skipped" unless the HeadPipeline is there.

The outcome is:

When mr.HeadPipeline is present:
- use the commit hash and status from the HeadPipeline
When mr.HeadPipeline is NOT present:
- use the commit hash from pull request struct
- assume the pipeline was "skipped"

In cases where GitLab is configured to require a pipeline to pass, this results on a message saying the MR is not mergeable.

More info:
- runatlantis#1852
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working never-stale waiting-on-response Waiting for a response from the user
Projects
None yet
Development

No branches or pull requests