Skip to content

Conversation

andeplane
Copy link
Contributor

@andeplane andeplane commented Apr 17, 2020

If function call ends up giving 503, nats-queue-worker crashes with panic: runtime error: invalid memory address or nil pointer dereference when trying to do callback.

Description

If X-Callback-Url is set, and the function call gives 503, functionRes is nil, but we try to evaluate functionRes.Header.Get("X-Duration-Seconds") in the postResult function. This leads to null pointer error and a dying pod. The call is then being retried multiple times on any new queue-worker pod starting.

This PR adds the timeout directly to the headers instead.

Motivation and Context

Crashing queue worker in the state mentioned above.

How Has This Been Tested?

I have patched the worker and tried with and without the change and verified that it does not crash with this patch. I tested this with the following images

['prom/alertmanager:v0.18.0', 'openfaas/basic-auth-plugin:0.18.11', 'openfaas/faas-idler:0.3.0', 'openfaas/gateway:0.18.13', 'openfaas/faas-netes:0.10.2-rc1', 'nats-streaming:0.17.0', 'prom/prometheus:v2.11.0', 'eu.gcr.io/cognitedata-development/queue-worker:debug2-amd64']

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I've read the CONTRIBUTION guide
  • I have signed-off my commits with git commit -s
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@derek
Copy link

derek bot commented Apr 17, 2020

Thank you for your contribution. I've just checked and your commit doesn't appear to be signed-off. That's something we need before your Pull Request can be merged. Please see our contributing guide.
Tip: if you only have one commit so far then run: git commit --amend --signoff and then git push --force.

Signed-off-by: Anders Hafreager <anders.hafreager@cognite.com>
@andeplane andeplane force-pushed the anders/fix_nullptr branch from 37e37d1 to 3738cb2 Compare April 17, 2020 19:10
@derek derek bot removed the no-dco label Apr 17, 2020
Signed-off-by: Anders Hafreager <anders.hafreager@cognite.com>
@alexellis alexellis requested a review from viveksyngh April 20, 2020 08:59
Copy link
Member

@alexellis alexellis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexellis alexellis merged commit 509b039 into openfaas:master Apr 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants