proposal: Go 2: returnFrom()

[ConradIrwin]

Author background

• Would you consider yourself a novice, intermediate, or experienced Go programmer?

  Experienced

• What other languages do you have experience with?

  Significant amounts of Ruby, Javascript/Typescript and Python. Small amounts of many others.

Related proposals

• Has this idea, or one like it, been proposed before?

  Yes, #35093 is the clearest other version of this (and a few other linked issues are also similar)

  • If so, how does this proposal differ?

    It limits the change to lexically visible places, and shows how this might interact with other Go features in more detail.

• Does this affect error handling?

  Somewhat

  • If so, how does this differ from previous error handling proposals?

    It is similar to #35093. The primary focus is a new control flow mechanism. While it is not an error-handling proposal per-se, it would help with error handling in currently hard-to-support contexts.

• Is this about generics?

  Not as such, but generic code would benefit from this change particularly.

  • If so, how does this relate to the accepted design and other generics proposals?

Proposal

• What is the proposed change?

  Go should add a new keyword/builtin returnFrom(fn, ...retArgs) that lets you return early from a surrounding function or function call.

• Who does this proposal help, and why?

  The primary case it would be helpful is for iterating over custom collections.

  Iteration in go is today handled using a for loop that looks something like this:

  i := collection.Iter()
  defer i.Close()
  for i.Next() {
    if doSomething(i.Current()) { break }
  }

  It is currently necessary in go to use a for loop to iterate if you know in advance that you may not want to iterate over the entire collection; because break and return let you end iteration early.

  This has two problems:

  1. The implementation of the iterator cannot use for loops. As each call to Next() happens on a different stack, and implementors must manually track and update progress.
  2. The caller has to do any necessary cleanup (in the example here, calling i.Close()).

  Both of these problems could be solved by allowing the implementor of the iteration to own the control flow. That would mean that the lifetime of the iterator is tied to a function, and so implementors could use for loops, and defer statements to clean up as necessary. (This was proposed in the context of extending for range loops to custom types at #47707). With returnFrom() this proposal is easier to implement, and doesn't suffer the "forget to check the return value" problem.

  Two examples of this API:

    func example() error {
        // used as an equivalent to `break` to move function control out of the loop
        collection.Range(func(u *Unicorn) {
            if doSomething(u) {
                returnFrom(collection.Range) // control flow jumps to fmt.Print call
            }
        })
        fmt.Print("done")
        // used as an equivalent to `return` to end the enclosing function early
        collection.Range(func(u *Unicorn) {
            if err := doSomethingElse(u); err != nil {
                returnFrom(example, err) // example() returns the value err
            }
        })
        return nil
    }

  With this support in the language, the implementor of collection.Range() can defer the cleanup
  required itself; and if the collection used something like a slice under the hood, the implementation of the iterator would be very simple (something like this):

    func (c *Collection) Range(fn func(u *Unicorn)) {
        s := c.getResults()
        defer s.Close()
        for _, u := range s.slice {
            fn(u)
        }
    }

  This would also pave the way for generic helper functions like slices.Map that are common in other
  languages (particularly those that use exceptions for error related control flow), but which are hard to implement in go because there is currently no idiomatic way to end iteration early if something goes wrong.

    func lookupHosts(hosts []string) ([]string, error) {
        return slices.Map(hosts, func(s string) string {
            ret, err := lookupHost(s)
            if err != nil {
                returnFrom(lookupHosts, nil, err)
            }
            return ret
        }), nil
    }
  
    // package slices
    func Map[T, U any](s []T, fn func(T) U) []U{
      r := make([]U, len(s))
      for i := range s {
        r[i] = fn(s[i])
      }
      return r
    }

  That said, there are a few other cases it would also help with. You could use it to clarify returning from the parent function in a deferred function.

    func before() (i int) {
        defer func() { i = 10 }()
        return 5
    }
  
    func after() int {
        defer func() { returnFrom(after, 10) }()
        return 5
    }

  There are also a few cases where a nested function is used outside of the context of iteration. One example is errgroup (though until #53757 is fixed, this will not work).

  // returns a match, or if no matches, the first error encountered
  func find(needle string) (int, error) {
    g := errgroup.Group{}
  
    for _, c := range collections {
        bound := c
        g.Go(func () error {
            result, err := bound.Seek(needle)
            if err != nil {
                return err
            }
            returnFrom(example, result, nil)
        })
    }
  
    return 0, g.Wait()
  }

  In terms of prior art, a similar keyword return-from exists in lisp. The problem is solved in Ruby by differentiating between methods and blocks (return returns from the enclosing method even when called in a block; and break returns control to the next statement in the enclosing method when called in a block). The distinction between methods and blocks seems unnecessarily subtle for a language like go. In Python this problem was solved by adding generators to the language, but that seems very heavy-handed for a language like go.

• Please describe as precisely as possible the change to the language.

  A new builtin would be added returnFrom(label, ...retArgs). The first argument label
  identifies a funtion call, that is either the call to the function named label that lexically encloses the function literal containing returnFrom, or a call to a function named label that contains the function literal in its argument list.

  A new type would be added to the runtime package, type EarlyReturn { } which contains unexported fields; and which implements the error interface.

  When returnFrom is called, it creates a new instance of runtime.EarlyReturn that identifies which function call to return from, and with which arguments; and then calls panic with that instance.

  When panic is called with an instance of runtime.EarlyReturn it unwinds the stack, calling deferred functions as normal until it reaches the identified function call. At that point, stack unwinding is aborted and execution is resumed as if the called function had executed return ...retArgs.

  If the identified function call is not found on the stack, then the stack is unwound and the program terminates as it would with an unhandled panic. This would happen if the closure in which returnFrom is called was returned from the function it was defined in, or run on a different goroutine. The error message would be: panic: call to returnFrom(<label>) outside of <label>.

  If recover is called while unwinding in this way, then the instance of runtime.EarlyReturn is returned. If the deferred function calls does not panic then execution is resumed as it is when recovering from a panic; and the returnFrom behavior is abandoned. Similarly if a deferred function panics with a different value, then the returnFrom is abandoned and the panic happens as normal. If the instance of runtime.EarlyReturn is later passed to panic, then it works as though returnFrom had been called.

  If returnFrom is called in a deferred function while the current goroutine is panicking (either with an un-recovered panic, or another returnFrom) it does not do an early return, but instead continues panicking as though you'd run panic(recover()).

• What would change in the language spec?

  A new section under Builtins

  Return from

  The built in function returnFrom(label, ...args) takes the name of a function as the first argument and the remaining arguments correspond to the return values of that function. returnFrom can be called in two places: within a nested function literal defined inside a function named label, or in a function literal in the argument list of a call to a function named label.

  When returnFrom(label, …) is called, it creates an instance of runtime.EarlyReturn that identifies which function call to return from and with what values, and then calls panic() with the instance of runtime.EarlyReturn.

  When panic() is called with an instance of runtime.EarlyReturn it runs all deferred functions on the current stack that were added after the function call, and then acts as though return …args was run in fn immediately returning to the caller of fn. If any of those deferred functions calls recover() then the panic is aborted, and execution resumes as it normally would. If returnFrom is called in a deferred function while the current goroutine is panicking, it instead resumes stack unwinding with the current panic.

• Please also describe the change informally, as in a class teaching Go.

  When using loops you can use break or return to exit early. When using nested functions use returnFrom instead.

  If you are implementing a function that accepts a callback, then you must be aware that it could panic() or returnFrom(). As such, any cleanup that must happen after you run the callback should be scheduled using a defer statement.

• Is this change backward compatible?

  Yes

• Orthogonality: how does this change interact or overlap with existing features?

  This feature iteracts heavily with panic() and recover(), and introduces a new case for functions that defer a call recover() across a user-provided callback to consider. In most cases this will not be an issue as code should not swallow unexpected panics; though it could exacerbate a problem if this is not the case (e.g. encoding/json before 2017).

  This feature also interacts heavily with iteration. Although not in scope for this proposal, an obvious extension would be to allow for x := range collection loops if collection implements either

   type Ranger[T any] interface { Range(f func(T)) }
   type Ranger2[T, U any] interface { Range(f func(T, U) }
  

  The compiler would convert break/return statements within the loop to calls to returnFrom() as appropriate.

  Given that, I considered proposing that the syntax was break fn, ...retArgs; but I think that is more confusing in the case you're trying to return from the current function, and it is less obvious that this feature interacts with panic() and recover().

  This feature also interacts with defer as it introduces a new way for deferred functions to set the return arguments of the enclosing function. Instead of using named arguments, code can now just use returnFrom(enclosingFn, ...retArgs).

  This feature also interacts with function literals. Currently in go it is not possible to name function literals; and this proposal does not suggest changing that. This means that returnFrom will not be able to return from an enclosing function literal directly. I don't think this is likely to be a problem in practice, as if you want to name a function you can move it's definition to the top level.

  func badExample() {
    func notSupported() { ... }
  }

• Is the goal of this change a performance improvement?

  No.

  • If so, what quantifiable improvement should we expect?
  • How would we measure it?

Costs

• Would this change make Go easier or harder to learn, and why?

  Harder, it adds surface area to the language. Although the behaviour is easy to describe in the common case, it is still more than not having it.

• What is the cost of this proposal? (Every language change has a cost).

  The primary cost is an additional powerful keyword that programmers are unlikely to have encountered before. That said, it worked for go statements :).

• How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?

  All tools would need to learn about the new builtin. As the syntax resembles a function call (albeit one that handles the first argument specially), and the semantics resemble a panic() (control flow is aborted) I think the changes would be minor. Generics may complicate things, as this would be a place you're allowed to "pass" a generic function with no type parameter.

  Go vet could be updated to add handling for obviously broken calls to returnFrom as discussed below.

• What is the compile time cost?

  The compiler must identify which calls can be returned from early, and ensure that there's a mechanism to do that. This would need two parts: one to identify which stack frame to unwind to, likely by inserting a unique value onto the stack in a known place in the frame; and secondly a mechanism to resume execution at that point; possibly with a code trampoline.

  As any modifications are scoped within a single function, this change would not slow down linking, or require changes to functions that do not lexically enclose returnFrom() statements.

• What is the run time cost?

  I would imagine that it has similar performance to a panic/defer, as the stack must be unwound explicitly instead of using normal returns.

• Can you describe a possible implementation?

  To identify which call returnFrom targets, stack-frames that may be targetted by returnFrom will need a unique identifier. I had initially hoped to reuse the frame pointer, but that may not be sufficient as a given frame pointer may be re-used for later calls after the current call has returned. The combination of a frame pointer and a per-goroutine counter value stored at a known offset in the frame should be sufficient.

  To actually implement returnFrom it may be necessary to add a small trampoline, either at the end of the function if the enclosed function is identified; or after the function call if a function call is identified, that copies the values passed to returnFrom into the right place. That said, if go's scheduler is able to resume a goroutine at any point, it may be possible to set up the state of the stack and registers and then just jump back into the existing code.

• Do you have a prototype? (This is not required.)

Yes, I have implemented a version of this by syntax rewriting at https://github.com/ConradIrwin/return-from

Examples

To help understand how this could work:

// OK, returning from named enclosing function
// f() == 10
func f() int {
  func () { returnFrom(f, 10) }()
  return 5
}

// OK, returning from enclosing function call
func f(s []int) {
  slices.Range(s, func (i int) { returnFrom(slices.Range) })
}

// OK, returning from (doubly) nested enclosing function call
func f(s []int) {
  slices.Range(s, func (i int) {
    inTransaction(func () { returnFrom(slices.Range) })
  })
}

// OK, returning from a enclosing instance method
func (s *T) f () {
    func () { returnFrom(s.f) }()
}

// OK, showing why panic/recover can be helpful
// f() == 10
func f() int {
  c := make(chan interface{})
  go func () {
    defer func () { c <- recover() }()
    returnFrom(f, 10)
  }()
  panic(<-c)
}

// OK, showing how it can be used with defer
// f() == 10
func f() int {
  defer func () { returnFrom(f, 10) }()
  return 5
}

// OK, showing how it could be used with function pointers
// f() == 10
func f() int {
    g := func() { returnFrom(f, 10) }
    g()
    return 5
}

// compile time error, f does not enclose returnFrom(f)
func f() { }
func g() { returnFrom(f) }

// compile time error, nested is a function pointer (not a named function)
func f() {
  nested := func () { func () { returnFrom(nested) }() }
}

// unhandled panic, could be added to go vet
func f() func() {
  return func() { returnFrom(f) }
}
f()()

// unhandled panic, could be added to go vet
func f() {
  go func () { returnFrom(f) }()
}
f()

// unhandled panic, the call to f(true) is over before f(false) calls the callback
func f(a bool) func() {
  if a {
    return func() { returnFrom(f, func() { }) }
  }
  f(true)()
  return func() { }
}
f(false)

// OK, but may be confusing, the first argument to returnFrom
// identifies the call to the function nested; not the function
// pointed to by the variable called nested.
func f() {
   nested := func(f func()) { f() }
   nested2 := func (f func()) { }
   nested(func () { nested = nested2; returnFrom(nested) })
}
f()

// OK, example of showing why returnFrom() ignores subsequent different calls.
func wrapper(fn func()) int {
    defer func() { returnFrom(wrapper, 5) }()
    fn()
}
func f() {
  x := wrapper(func () {
      returnFrom(wrapper, 10)
  })
  fmt.Println(x == 10)
}

[ianlancetaylor]

It's quite common in Go to have anonymous function literals.

    f := func() { returnFrom(f) }

What happens if the label is ambiguous, due to a recursive call? Do we return from the innermost instance? That seems potentially quite confusing.

In general the notion of labelling a function call by naming the function being called seems confusing and non-orthogonal. What about variables of function type? Recall that Go explicitly does not permit using == on values of function type.

[ConradIrwin]

Thanks for taking the time to read through this!

I don't think returnFrom would work with anonymous function literals. Two further tweaks to the language that could make this better would either be: 1) allow nesting function statements within functions; or 2) use the same approach as javascript where assignment to a variable implicitly names functions (so the code you wrote would work).

For recursive functions I think the inner-most call would be the most obvious choice (it seems possible to further extend returnFrom(label) to allow labelled calls, but when I was writing up the proposal it seemed like it would add even more complexity for not that much reason).

function f () {
  outer: slices.Range(s, func () {
    slices.Range(r, func () {
      returnFrom(outer)
    })
  })
}

Definitely agree that naming a function call can be confusing (it looks like you are passing a function, but you aren't); but requiring every function call to be labelled seemed not great either...

[apparentlymart]

From reading the proposal I understand this as more-or-less some syntactic sugar over panic and recover, where the panic call is replaced with a returnFrom call, and the recover call is implicit in some hidden generated code at the call site of the function.

Do you anticipate that every function call would generate code to approximate the effect of calling recover() and returning early if it returns a runtime.EarlyReturn instance? Or do you imagine the compiler quietly "coloring" all functions to record whether they contain a (direct or indirect) returnFrom so that it can generate the additional code only where a downstream caller (direct or indirect) makes use of this facility?

The details of how runtime.EarlyReturn would be defined aren't clear to me:

When returnFrom(label, …) is called, it creates an instance of runtime.EarlyReturn that identifies which function call to return from and with what values, and then calls panic() with the instance of runtime.EarlyReturn.

This suggests that there are two fields of this type: some sort of identifier for the function, and some return values. Your definition of label suggests that the compiler would search up the enclosing functions lexically until it finds one of the given name. Discussion above already considered how that would interact with anonymous functions, but I wonder also about generic functions: if I have a func Example[T](), would I need to returnFrom(label[SomeType], ...), or would returnFrom(label, ...) match any instantiation of that generic function?

I'm also curious about the subsequent arguments representing return values. If the compiler knows from the first argument exactly which function we're returning from then I suppose it can statically check that there's the appropriate number of additional arguments and that they are assignable to the function's return types, in which case we wouldn't need to consider the situation where the types mismatch at runtime. In the implementation details, is there a field inside runtime.EarlyReturn of type []any that captures the return values as interface values? Or do you imagine the compiler generating an anonymous hidden type per function which captures directly its return values?

---

Stepping away from the implementation and into the programmer-visible effect, it seems like a key consideration of this proposal is that now any function call can potentially be an early return. That is admittedly already true in the presence of panic, but we ask Go programmers to understand that only as an exceptional return path used in "should never happen" cases where code is somehow buggy, and consequently that pattern is not commonly used for happy-path control flow.

I have some difficult-to-substantiate concerns about that situation. I personally currently use panic only with the intention of intentionally crashing the program, and so when I'm writing happy-path code I'm accustomed to considering only the locally-visible control flow. I can see that, due to the restriction of lexical nesting, this is restricted to situations that could also be described using goto, break, and continue within a single named function. In those cases however we do explicitly label the places where control flow might arrive, which then serves as a prompt to the reader that something special is going on. If the intent here is to support a sort of "cross-function goto" (so that function calls can behave as loops) then I think I'd lean towards your later variant of explicitly labeling the calls, so that there's at least an explicit mark at the call site that something unusual is happening.

If you haven't seen it already, it might interest you that there's an active discussion about defining a standard iterator interface that might integrate with for ... range: #54245. If Go had a feature like what's described there, do you think returning across multiple levels of call would still be compelling?

[ConradIrwin]

Do you anticipate that every function call would generate code to approximate the effect of calling recover() and returning early if it returns a runtime.EarlyReturn instance? Or do you imagine the compiler quietly "coloring" all functions to record whether they contain a (direct or indirect) returnFrom so that it can generate the additional code only where a downstream caller (direct or indirect) makes use of this facility?

No, I expect only calls that are statically determined to need it will get it (because we're re-using panic/recover this is determinable by looking lexically locally; there's no need for this information to move between compilation units).

This suggests that there are two fields of this type: some sort of identifier for the function, and some return values. Your definition of label suggests that the compiler would search up the enclosing functions lexically until it finds one of the given name. Discussion above already considered how that would interact with anonymous functions, but I wonder also about generic functions: if I have a func ExampleT, would I need to returnFrom(label[SomeType], ...), or would returnFrom(label, ...) match any instantiation of that generic function?

Either way; one of the reasons the prototype doesn't work well with generics right now is exactly this ambiguity. As a user of the feature, I'd prefer to use returnFrom(label), but we could also make returnFrom(label[SomeType]) work.

I'm also curious about the subsequent arguments representing return values. If the compiler knows from the first argument exactly which function we're returning from then I suppose it can statically check that there's the appropriate number of additional arguments and that they are assignable to the function's return types, in which case we wouldn't need to consider the situation where the types mismatch at runtime. In the implementation details, is there a field inside runtime.EarlyReturn of type []any that captures the return values as interface values? Or do you imagine the compiler generating an anonymous hidden type per function which captures directly its return values?

In the prototype version this is a hidden type per call-site; I imagine it depends what's easiest to compile, but I agree there will be no runtime-type mismatches as the compiler will check.

I have some difficult-to-substantiate concerns about that situation. I personally currently use panic only with the intention of intentionally crashing the program, and so when I'm writing happy-path code I'm accustomed to considering only the locally-visible control flow. I can see that, due to the restriction of lexical nesting, this is restricted to situations that could also be described using goto, break, and continue within a single named function. In those cases however we do explicitly label the places where control flow might arrive, which then serves as a prompt to the reader that something special is going on. If the intent here is to support a sort of "cross-function goto" (so that function calls can behave as loops) then I think I'd lean towards your later variant of explicitly labeling the calls, so that there's at least an explicit mark at the call site that something unusual is happening.

Yeah; there's some discussion related to this on making errgroup "panic-safe". In general one thing that go is not good at is expectations around panics() (other than that they might crash your program). In the kind of code I write, I think it's important that even unhandled programmer errors do not terminate programs (one web request shouldn't be able to kill others in flight for example), so I am careful that panics are handled appropriately and error logging is tied to the request that was made. This is not uniformly how the go community works (I've heard people express the ideal that panics should always cause programs to exit), so there's some tension there. It gets particularly gnarly when using go-routines, but we have internal libraries that let you start a new goroutine and ensures that panics are propagated correctly.

If you haven't seen it already, it might interest you that there's an active discussion about defining a standard iterator interface that might integrate with for ... range: #54245. If Go had a feature like what's described there, do you think returning across multiple levels of call would still be compelling?

Thanks for the pointer! I'll wade in in a bit – in general (as described in this proposal) I find the idea of an iterator interface described there to be a bit backwards. It may let you make the syntax on the consumer side shorter in the common case, but it puts a lot of burden on the implementor to do things right; I think the iterator proposal from #47707 makes a lot more sense, and if we had returnFrom too it would be easier for implementors and consumers alike.

[ianlancetaylor]

It seems very difficult to make this fully orthogonal. Consider

func F() func() {
    return func() { returnFrom(F) }
}

func G() {
    f := F()
    f() // what happens here?
}

[ConradIrwin]

As written right now it would panic at runtime: "panic: call to returnFrom(F) outside of F" (or similar). I'd expect go vet to catch this case as there is no call to the function from within F (though there may be more subtle variations that are harder to catch statically).

Similarly to a send/recv on a closed channel; trying to return from a function that has already returned is a programmer error.

[ianlancetaylor]

OK, but what we aim for in Go is that, as much as possible, language features simply work. That's why, for example, it's fine (today) to return a function closure out of a function.

Consider also code like this:

var C1 = make(chan func())
var C2 = make(chan bool)

func F() {
    go G()
    C1 <- func() { returnFrom(F) }
    <-C2
}

func G() {
    f := <-C1
    f()
    C2 <- true
}

What happens here? Presumably F doesn't return twice.

[ConradIrwin]

Thank you for continuing to explore the ideas here with me!

Your example would also panic with "panic: call to returnFrom(F) outside of F".

G is running on a different goroutine, so when returnFrom unwinds the stack (it's implemented as a panic under the hood) it will not find the call to F on the stack and will terminate the program with an unhandled panic.

I think this is "as much as possible, the feature just works" – it doesn't make sense to allow a function call to return more than once. There's definitely an editorial question of whether the benefits brought by the feature make up for the confusion – but that's why this is a proposal not a pull request after all :D.

As mentioned in the proposal I think the benefits are surprisingly far reaching for a comparatively small language change:

• You can have functions like a generic func Map[A,B any](a []A, func (A) B) b []B and have a consistent mechanism to stop iteration if an error occurs (without needing an additional func MapErr[A, B any](a []A, func (A) (B, error)) (b []B, error))
• You can implement an iterator proposal like proposal: Go 2: spec: for range with defined types #47707 (but better because there's less chance the iterator forgets to handle the boolean return)
• You can clarify editing the surrounding functions return values in a deferred func (maybe a taste thing, but assigning to named return values still reads a bit odd after many years of using go)

One example from nested in the proposal that is related to the example you just gave, it is possible (with the proposal as written currently) to do some amount of go-routine work, for example (if contrived):

// f() == 10
func f() int {
  c := make(chan interface{})
  go func () {
    defer func () { c <- recover() }()
    returnFrom(f, 10)
  }()
  panic(<-c)
  return 5
}

[ianlancetaylor]

it will not find the call to F on the stack

I don't understand how to implement that in the general case.