context: ease debugging of where a context was canceled?

[matthewceravolo]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/matthew/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/matthew/work"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build356255000=/tmp/go-build -gno-record-gcc-switches"

What did you do?

used context.WithTimeout() to make requests to google calendar api and outlook calendar api

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

Making requests using contexts with timeouts should cancel when the timeout is reached

What did you see instead?

Contexts with timeouts are instantly failing with "context canceled" even though the timeout is set to time.Minute. The error goes away if I remove the timeout context and use one without any limit. It also seems to be transient to some extent

[ianlancetaylor]

Sorry, but there isn't enough information here to say anything useful. Please show us your code, or tell us how to recreate the problem. Thanks.

[matthewceravolo]

import (
	"net/http"
	"os"
	"time"

	"golang.org/x/net/context"
	"golang.org/x/oauth2"
	"golang.org/x/oauth2/google"
)

var (
	backgroundContext = context.Background()
	oauthConfig       = &oauth2.Config{
		ClientID:     os.Getenv("GOOGLE_PROJECT_CLIENT_ID"),
		ClientSecret: os.Getenv("GOOGLE_PROJECT_SECRET_KEY"),
		Endpoint:     google.Endpoint,
	}
)

type calendarClient struct {
	*calendar.Service
	requestTimeout time.Duration
}

// NewCalendarClient creates a new authenticated Google Calendar client.
func NewCalendarClient(refreshToken string) (wf.Client, error) {
	client, err := newClient(refreshToken)
	if err != nil {
		return nil, err
	}

	service, err := calendar.New(client)
	if err != nil {
		return nil, err
	}

	return &calendarClient{Service: service, requestTimeout: time.Minute}, nil
}

func newClient(refreshToken string) (*http.Client, error) {
	contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
	defer cancel()
	return oauthConfig.Client(contextWithTimeout, &oauth2.Token{RefreshToken: refreshToken}), nil
}

func (client *calendarClient) GetCalendars() (*calendar.CalendarList, error) {
	contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
	defer cancel()
	return client.CalendarList.List().Context(contextWithTimeout).Do()
}

The issue is that when we call GetCalendars for example, it will instantly return "context canceled". When I use an empty context with no timeout, everything works as expected. This all began failing in the past couple of days since updates were made in the /x/net package

[meirf]

I think the behavior you're seeing is expected. I don't see a Go bug.

NewClient (called by oauthConfig.Client):

The returned client is not valid beyond the lifetime of the context.

Your code has:

func newClient(refreshToken string) (*http.Client, error) {
    contextWithTimeout, cancel := context.WithTimeout(backgroundContext, time.Minute)
    defer cancel()
    return oauthConfig.Client(contextWithTimeout, &oauth2.Token{RefreshToken: refreshToken}), nil
}

The defer cancel() causes the context to be cancelled before the client even gets to calendar.New.

"When I use an empty context with no timeout, everything works as expected. This all began failing in the past couple of days since updates were made in the /x/net package." You are describing to things that might have changed in your code (no longer using empty context, changes in x/net). If my theory above is correct: if you undo the x/net changes (to only change one variable) you'll see that any non-empty context will still have the problem you're seeing.

[bcmills]

I've seen this sort of issue crop up several times now.

I wonder if context.Context should record a bit of caller information (say, the PC of the immediate calling function) in context.WithTimeout and in calls to the CancelFunc returned by context.WithCancel. Then we could add a debugging hook to interrogate why a particular context.Context was cancelled.

Perhaps something along the lines of:

package context

// A DoneReasoner describes the reason a Context is done.
// The Context implementations returned by this package implement DoneReasoner.
type DoneReasoner interface {
	DoneReason() string
}

// DoneReason returns a human-readable description of the reason that ctx.Done() is closed,
// or the empty string if ctx.Done() is not closed.
func DoneReason(ctx context.Context) string {
	select {
	case <-ctx.Done():
	default:
		return ""
	}
	if r, ok := ctx.(DoneReasoner); ok {
			return r.DoneReason()
	}
	return ctx.Err().Error()
}

[bcmills]

(CC @Sajmani @bradfitz for Context debuggability.)

[matthewceravolo]

@meirf I think there was a misunderstanding in how I described the problem. This code has always been using two contexts like I described but only in the past two days since we pulled updates to the vendor package did it start failing. There were not any changes whatsoever related to any of this code for several months besides pulling in the package update.

Only after we pulled in the update and everything started failing, we switched to contexts without timeouts (still multiple) and started seeing things succeed. If your explanation is correct, the defer cancel was not previously working as expected

However, I believe your explanation is incorrect because the error message is returned when GetCalendars is called, not on client retrieval. We always receive a calendarClient back, so calendar.New is being called correctly

[ianlancetaylor]

Do you happen to have a record of which revision of the context package you were using previously, and which one you updated to?

[bcmills]

only in the past two days since we pulled updates to the vendor package did it start failing.

Which vendored packages did you update? (Just x/net, or also x/oauth2?)

[matthewceravolo]

I've narrowed down the precise commit where things began to fail, and there are more vendor updates here than I'd previously thought. It looks like both packages (x/net and x/oauth2) were updated and that the previous version was decently old such that it may be very difficult to narrow anything down, but here are a couple of samples from the revisions:

"checksumSHA1": "dr5+PfIRzXeN+l1VG+s0lea9qz8=", => "checksumSHA1": "GtamqiJoL7PGHsN454AoffBFMa8=",
"path": "golang.org/x/net/context", => "path": "golang.org/x/net/context",
"revision": "f5079bd7f6f74e23c4d65efa0f4ce14cbd6a3c0f", => "revision": "1e491301e022f8f977054da4c2d852decd59571f",
"revisionTime": "2017-07-19T21:11:51Z" => "revisionTime": "2018-05-30T06:29:46Z"

"checksumSHA1": "hyK05cmzm+vPH1OO+F1AkvES3sw=", => "checksumSHA1": "xaHFy3C2djwUXtiURpX3/ruY8IA=",
"path": "golang.org/x/oauth2", => "path": "golang.org/x/oauth2",
"revision": "314dd2c0bf3ebd592ec0d20847d27e79d0dbe8dd", => "revision": "1e0a3fa8ba9a5c9eb35c271780101fdaf1b205d7",
"revisionTime": "2016-12-14T09:25:55Z" => "revisionTime": "2018-06-02T17:55:33Z"

Also, I re-tested the above code by leaving everything the same and completely removing the second context.WithTimeout inside of GetCalendars such that only the client has a timeout configured, and I see the same error

[bcmills]

https://golang.org/cl/45370 is in that range and loosely matches @meirf's diagnosis. It looks like your vendor update uncovered a latent bug within your code.

[matthewceravolo]

I've confirmed that commenting out the defer cancel() inside of newClient does in fact successfully return our calendar data as expected, but it is still not matching what was diagnosed above since when the error occurs it is on GetCalendars, it isn't canceling prior to Calendar.New. I don't feel I have a great understanding of exactly what's happening here other than the pre-mature cancel seems to be killing the client's ability to make any future successful requests

Is the expectation that we exercise the client we retrieve to its fullest extent only to call cancel() at the last moment? And that if it's set on this client we shouldnt even need to bother with client.CalendarList.List().Context(), correct?

Can we rely on GC and just not bother holding onto the cancel func?

[fraenkel]

I think you have it backwards. Get rid of the context in newClient.
I assume you want to have a timeout per service call rather than across the Client use.

[matthewceravolo]

@fraenkel I'm fine with whichever, but doesn't putting it on the client prevent me from needing to duplicate this context code across all calls?

[bradfitz]

@matthewceravolo, please don't retitle the bug. We have a convention for how we use titles.

It still reads above like this is user error and you got lucky in the past, so I'd like to keep this bug about thinking about ways to make this debugging easier for other users making similar errors.

[Sajmani]

@bcmills I wouldn't use type assertion to introspect a Context because they are wrapped so often. If you want to attach a Done reason to a Context, I'd use a Context value. This is overhead, though, so I'd make it opt-in in somehow. Perhaps provide a context.WithDebug method that adds a Context value for storing debugging info. The code in WithTimeout and WithCancel can check for this debug value to decide whether to walk the call stack and record any information there. The context debug value needs to be installed at the top of the context stack to be seen by all derived contexts, so I'd enable it in the request handler (say) with a parameter like ?debugcontext=1.

[bcmills]

I wouldn't use type assertion to introspect a Context because they are wrapped so often.

Good point. A Context.Value call with an exported (but opaque) key to retrieve the reason code should be fine.

If you want to attach a Done reason to a Context, I'd use a Context value. This is overhead, though, so I'd make it opt-in in somehow. Perhaps provide a context.WithDebug method that adds a Context value for storing debugging info. The code in WithTimeout and WithCancel can check for this debug value to decide whether to walk the call stack and record any information there.

Ironically, Context.Value is O(N) with the depth of Context nesting, whereas runtime.Caller(1) is O(1): the mere act of checking whether debugging is enabled may be significantly more costly than storing the debug information unconditionally.

[Sajmani]

I would store a *debug in the context to flatten the lookup, leaving it nil when debugging is disabled. Associating that debug value with an unexported context key allows the debug info to propagate through Context implementations outside the context package.

…

On Tue, Jul 17, 2018 at 3:52 PM Bryan C. Mills ***@***.***> wrote: I wouldn't use type assertion to introspect a Context because they are wrapped so often. Good point. A Context.Value call with an exported (but opaque) key to retrieve the reason code should be fine. If you want to attach a Done reason to a Context, I'd use a Context value. This is overhead, though, so I'd make it opt-in in somehow. Perhaps provide a context.WithDebug method that adds a Context value for storing debugging info. The code in WithTimeout and WithCancel can check for this debug value to decide whether to walk the call stack and record any information there. Ironically, Context.Value is O(N) with the depth of Context nesting, whereas runtime.Caller(1) is O(1): the mere act of checking whether debugging is enabled may be significantly more costly than storing the debug information unconditionally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26356 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJSK3TSfBWY21reU6GLskizNeC3fM3mEks5uHkBugaJpZM4VNsJ2> .

[Sajmani]

@andreimatei Others have suggested similar hooks, and IIRC we even prototyped some. That might be worthwhile, but IMO debugging context cancelation is a common enough problem that I'd like to make it better for everyone by default, if we can do so with acceptable cost.

[jtolio]

I've been timidly thinking of proposing to the go team to provide the required interfaces for custom implementations to participate in propagation as efficiently as the stdlib one, so that stdlib ctx can co-exist with the cowboys. I guess I'm asking now.

Perhaps this deserves its own issue. I'd certainly be for it.

[Sajmani]

@jtolio There's prior discussion on this in #28728

[bcmills]

Capturing the stack trace at the cancel() call is likely to be uninteresting: usually cancel is called either via timer.AfterFunc (in the case of a deadline) or via defer, which may make it harder to debug if there are multiple defers in the same function.

A given WithContext call site may have many possible cancel call sites, and while it's true that the cancel call is often just a deferred function, in some of the most interesting cases it is decidedly nontrivial.

For example, a server might construct a context.WithCancel for a background operation and store the cancel function in a map, and only invoke that cancel function when either the background operation completes or an RPC comes in to explicitly cancel it. In such a case it seems very important to distinguish between “canceled because the operation completed” and “canceled by RPC”, but the WithCancel call site would not distinguish them at all.

[Sajmani]

Based on the feedback from several of you, I'll prototype an alternative that allows passing a value in via the cancel function.

…

On Fri, Jan 7, 2022 at 2:33 PM Bryan C. Mills ***@***.***> wrote: Capturing the stack trace at the cancel() call is likely to be uninteresting: usually cancel is called either via timer.AfterFunc (in the case of a deadline) or via defer, which may make it harder to debug if there are multiple defers in the same function. A given WithContext call site may have many possible cancel call sites, and while it's true that the cancel call is often just a deferred function, in some of the most interesting cases it is decidedly nontrivial. For example, a server might construct a context.WithCancel for a background operation and store the cancel function in a map, and only invoke that cancel function when either the background operation completes or an RPC comes in to explicitly cancel it. In such a case it seems very important to distinguish between “canceled because the operation completed” and “canceled by RPC”, but the WithCancel call site would not distinguish them at all. — Reply to this email directly, view it on GitHub <#26356 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACKIVXOCQENNWK5TYHTYNDTUU45W7ANCNFSM4FJWYJ3A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[andreimatei]

@jtolio There's prior discussion on this in #28728

I had missed the developments around #28728 and their implications. Those changes seem to let us avoid the extra goroutine, which is great! That is, as long as we don't want to override the Done() method. Not being able to override it means that we have to pay some extra allocations because we have to pair a stdlib WithCancel ctx with our implementation, but that's not the end of the world. With this in mind, I've given a renewed push to our attempts to implement richer cancellation, linked below.

Based on the feedback from several of you, I'll prototype an alternative that allows passing a value in via the cancel function.

I'm excited to see what comes out of it!

FWIW, here's what we're trying within CockroachDB: a colleague's PR and my commit on top of it.

[Sajmani]

I've updated my CL with a new version that prototypes adding *Cause variants of WithCancel, Deadline, and Timeout.
The purpose of this design is to provide backwards-compatibility for existing code while enabling users to add cancelation "causes" where they need them.
I welcome feedback on the API with respect to whether it addresses this issue:
https://go-review.googlesource.com/c/go/+/375977/3/src/context/context.go

I need to write more tests, particularly for cases where there are multiple causes in a context chain, and with mixed cancelation orders.
I also need to write benchmarks so we can measure slowdowns on critical paths and any new allocations.

[andreimatei]

This is wonderful!

Unless I'm missing something, your Cause(ctx1) implementation can return an error from ctx2's cancel even if ctx1 was not canceled when ctx2 was canceled. In fact, it might return an error even if ctx1.Error() == nil (i.e. ctx1 is not canceled). This is when a ctx on the chain from ctx1 to ctx2 overrode the Done() channel. Right? This seems surprising.
Have you considered adding protection against this in the way that parentCancelCtx() does?
If you don't want to do that, consider exporting cancelCtxKey so that 3rd party implementors can hijack Value(cancelCtxKey).

[Sajmani]

@andreimatei I need to write some tests with chains of contexts and multiple causes. Hopefully that will help us see whether the returned causes are sensible or surprising. I'm explicitly propagating causes from parents to children in propagateCancel and cancelCtx.cancel—the same places where the context's error gets propagated. Glad to hear this API is directionally correct. If others agree, we'll need to take this up for proposal review.

[Sajmani]

@andreimatei I added some tests:
https://go-review.googlesource.com/c/go/+/375977/4/src/context/context_test.go#798
Do these exercise the case you were describing? (I'm not sure what ctx1 and ctx2 mean in your description).

[Sajmani]

I added test for cause vs. causeless cancels and more orderings. I also renamed things so the test is easier to follow. I think the behavior is pretty clear now: the first cancel cause sticks for all children, even if it's a nil cause.

[andreimatei]

Do these exercise the case you were describing?

I don't think so. The cases I'm thinking of are about a custom implementation of context that overrides the Done() channel - e.g. an implementation that doesn't inherit the cancelation of its parent. In such cases, your Cause implementation can return a non-nil cause for a context who's Err() is nil. This is true for both Cause(<custom ctx impl>), and for Cause(stdlib ctx derived from custom ctx). I think this is surprising. In the particular case of a stdlib ctx, I think we should guarantee that, if Err() is nil, Cause() is also nil. Would you agree?

Here's a small program that demonstrates what I'm talking about.

package main

import (
	"context"
	"fmt"
	"time"
)

type uncanceledCtx struct {
	p context.Context
}

var _ context.Context = uncanceledCtx{}

func (u uncanceledCtx) Deadline() (deadline time.Time, ok bool) {
	return time.Time{}, false
}

func (u uncanceledCtx) Done() <-chan struct{} {
	return nil
}

func (u uncanceledCtx) Err() error {
	return nil
}

func (u uncanceledCtx) Value(key any) any {
	return u.p.Value(key)
}

func main() {
	ctx1, cancel1 := context.WithCancelCause(context.Background())
	uc := uncanceledCtx{p: ctx1}
	ctx2 := context.WithValue(uc, "key", "val")
	cancel1(fmt.Errorf("cancel 1"))

	fmt.Printf("uncanceled: Err: %v, cause: %v\n", uc.Err(), context.Cause(uc)) // uncanceled: Err: <nil>, cause: cancel 1
	fmt.Printf("ctx2: Err: %v, cause: %v\n", ctx2.Err(), context.Cause(ctx2))   // ctx2: Err: <nil>, cause: cancel 1
}

The output of Cause(uc) and Cause(ctx2) is surprising to me.

[Sajmani]

Ah, thanks for the reproduction case. I would state the expectation in the reverse: Cause(ctx) is only valid (defined) if ctx.Err() != nil (which is the same as when ctx.Done() is closed). If ctx.Err() is nil, then Cause(ctx) is undefined. I can add this to the spec in the CL.

We could strengthen the spec for Cause to mirror what we say for Err:

	// If Done is not yet closed, Err returns nil.
	// If Done is closed, Err returns a non-nil error explaining why:
	// Canceled if the context was canceled
	// or DeadlineExceeded if the context's deadline passed.
	// After Err returns a non-nil error, successive calls to Err return the same error.
	Err() error

So we could add (and implement) "If ctx.Err returns nil, the Cause(ctx) returns nil."
I'd prefer not to add this strengthening of the spec unless there's a good reason to do so.
I expect most uses of Cause to check <-ctx.Done() or ctx.Err() != nil before calling Cause(ctx).

[andreimatei]

I would state the expectation in the reverse: Cause(ctx) is only valid (defined) if ctx.Err() != nil (which is the same as when ctx.Done() is closed). If ctx.Err() is nil, then Cause(ctx) is undefined. I can add this to the spec in the CL.

I expect most uses of Cause to check <-ctx.Done() or ctx.Err() != nil before calling Cause(ctx).

In my opinion, we should aim to relegate ctx.Err() to the past. I think we should aim for Cause(ctx) to completely replace it for new programs (because it's much better) (*). So I personally do not agree with the premise that any Cause() call should be preceded by an Err() call (or, rather, I don't think we should design Cause() with that mindset). There may be a preceding Done() call, but not always.

Another thing I wanted to bring up is making sure that custom ctx impls can participate in Cause() determination for themselves and their children. I think with your current patch that's not possible, because the key that Cause() looks for is not exported?

(*) If I'd have my druthers, we would make Cause() a new method on ctx (even if that means that custom Ctx impls will need to add that method to compile under the new go version. I'm not very familiar with the go compatibility promise but, as far as I'm personally concerned, there's not that many custom Ctx impls out there and the changes to them would be trivial (e.g. they can delegate to Err()).
I'd also consider adding a version of the Done() method that returns the error. It'd have to be a new channel per caller; I haven't thought through the performance implications.

[bcmills]

@andreimatei, the Go compatibility policy does not allow new methods to be added to exported interfaces. So there is no plausible way for Cause to replace Err outright.

The best we could perhaps do is retrofit a top-level func Cause(Context) error function into the context package, which would use Cause if defined or Err otherwise.

[Sajmani]

Regarding compatibility, my goal with this CL is to avoid breaking anything, including existing implementations of the Context interface. Therefore I'm not adding Cause() to the Context interface.

My goal with Cause is not to replace Err, but to allow people to supplement Err with additional information.

I think you're saying that you'd like to check Cause(ctx) != nil instead of ctx.Err() != nil. If so, then indeed we'd need to make sure they are nil / not-nil at the same times. We'd probably also want to make Cause(ctx) == ctx.Err() when the user hasn't provided a cause. But this then makes it difficult to determine whether ctx.Done was closed due to cancelation or deadline exceeded (the user would need to encode this into their cause).

Regarding implementing Cause for custom contexts, users can do this easily by wrapping a context created using WithCancelCause. I'd like to keep things simple until there's a demonstrated need to support anything fancier.

[andreimatei]

We'd probably also want to make Cause(ctx) == ctx.Err() when the user hasn't provided a cause.

That's what I was thinking.

But this then makes it difficult to determine whether ctx.Done was closed due to cancelation or deadline exceeded (the user would need to encode this into their cause).

Right. But isn't that a natural thing to do, since the user has to pass an error to WithDeadlineCause(...,err)? One thing to consider is having the Cause() for a deadline be a wrapper error type (say, CtxDeadlineExceeded), which wraps the user-provided error (such that errors.Is/As still recognize the provided error). And similarly for causes stemming from cancellation.

Regarding implementing Cause for custom contexts, users can do this easily by wrapping a context created using WithCancelCause. I'd like to keep things simple until there's a demonstrated need to support anything fancier.

When people implement custom contexts, I think they primarily do so for efficiency reasons. In such cases, forcing them to wrap a stdlib ctx defeats the purpose. There's also a more philosophical contradiction here, in my opinion. Context is an interface, presumably, because you want people to be able to implement their own. But you seem to also want to tell people that some functionality is only available to the stdlib, so you have to jump through hoops to get it in a custom impl. Aren't these two things at odds with each other? There are similar nuances at play around #28728 and the performance implications of propagating cancellation across context types: in that case it's not strictly about what an implementer can do, but rather the degree to which the stdlib plays nicely with others. Pragmatically speaking, since every library out there uses the stdlib ctx, whether or not it's feasible for other to write custom implementations largely depends on how well the stdlib impl interacts with them.0ad3686 addressed the concerns to some degree, but I think not fully.
So, as a matter of principle, FWIW, my opinion is that the performance and ergonomics of 3rd party impls should be a concern whenever mucking with the context library.

---

I should say that I can nitpick and form semi-informed opinions on proposals, but ultimately I'll gladly take most improvements around the cause, compared to nothing.

[Sajmani]

All good points, @andreimatei , thank you.
I suggest at this point I write up the current state as a Proposal and document your suggestions as Alternatives to Consider.

[rsc]

Please take a look at #51365 and see if it addresses your use cases. Thanks!