proposal: Go 2: introduce a way of performing function chains

[asegs]

Author background

• Would you consider yourself a novice, intermediate, or experienced Go programmer?
  Experienced with less knowledge of internals.
• What other languages do you have experience with?
  Java, Elixir, Python, JavaScript, Lisp and some other ones.

Related proposals

• Has this idea, or one like it, been proposed before?
  Yes, I believe this relates to at least three previous proposals:
  proposal: spec: allow type parameters in methods #49085
  proposal: Go 2: alternate function call syntax to improve ergonomics of chainable top-level functions #47680
  proposal: Go 2: add elixir-like pipe operator #33361
• If so, how does this proposal differ?
  This proposal requests that the question of function chaining be readdressed in the context of generics, considering several syntactic limitations and numerous requests for a reconsideration in the type parameters & methods thread which I have been involved with. In terms of specific differences, this proposal leaves implementation open ended and serves to be a dedicated place for discussion of a solution to the problem of function chaining in Go. This proposal also provides a much more diverse set of new features that would be supported by this syntax.
• Does this affect error handling?
  It shouldn't, unless implementation settles on some "in chain" style of error handling.
• Is this about generics?
  At the very least it is tangential, since generics have enabled Java style stream APIs and Ruby style "expect" APIs, both which are severely kneecapped not by generics but by Go's syntax.
  • If so, how does this relate to the accepted design and other generics proposals?
    It proposes an alternative way of providing what people are looking for with type parameters on methods, which is something that seems impossible given the current implementation of generics.

Proposal

• What is the proposed change?
  Provide some way of chaining functions, either using a new operator such as ->, |>, or .. or by using the . operator used for calling a method to also call functions.
• Who does this proposal help, and why?
  Here I will list a number of potential use cases that would be possible with this simple, backwards compatible change. Here, treat the dot as whatever operator or strategy is settled upon. These are simply meant to show abstract representations of use cases.
  • Streaming APIs
    • numbers.map(toString)
    • numbers.reduce(add, 0)
  • Ruby-style expect(foo).toBe(bar) assertions.
  • New syntax
    • 3.plus(3)
    • n.times(print, "Hello")
    • "hello world".split(' ')
  • Something close to final default methods for interfaces. This is better served if we were able to use . for both methods and functions, but if not, here is an example:

  package main
  
  import "fmt"
  
  type Animal interface {
    Speak() string
    Die() string
  }
  
  func EatenByHyena(a Animal) {
    a.Speak()
    a.Die()
  }
  
  type Dog struct {}
  
  func (d Dog) Speak() {
    fmt.Println("Bark!")
  }
  
  func (d Dog) Die() {
    fmt.Printf("Oh no, not my pupper!")
  }
  
  func main() {
    d := Dog{}
    d.EatenByHyena()
  }

It is true that several similar issues to this one seem to have been poorly received. The code examples above are in fact possible using nested function calls, which is a point that is quickly made. However, I believe that there is a reason many programming languages are supporting similar syntax, which is that the code ends up closer to readable English and requires less brain power to manually parse. I will quote two people very close to me here:

"To add information to a sentence, I don't have to go back and add to the beginning."

-My friend.

"I've yet to see a clear reason why function chaining is really bad or simply not possible, and the attitude that negate(square(add_2(x))) is equal in readability to x |> add2() |> square() |> negate() is just inane. Reading those nested Lisp style function calls forces the brain to parse recursively when reading it. In fact, when I parse that first example, I find myself going backwards, inside out, starting at x, then add2, then square, then negate. I'm not sure if this is the case for others, but if this is the natural order of reading transformations on data, then we should support that syntactically."

-Myself.

Finally, I believe that most of what is demonstrated here with syntax would also be doable with generic methods. Three reasons I am not focusing on that:

1. I have been told that such a change would either be impossible or would require major changes to how generics are implemented.
2. Since you cannot declare methods on primitive types, such a change would preclude people from using any of this syntax on ints, strings, and any other primitive value.
3. There is already an open issue.

I believe that this provides amazing value as a change, it adds much more readable syntax to any transformations on data without changing internals. Most importantly, it adds support for a number of programming patterns that are terribly underserved by the current nested function call syntax. The ability to define pseudo-default methods on interfaces is interesting to me, and I imagine more satisfying cases will emerge from this syntax.

For people who take the valid opinion of "why introduce a change that doesn't add any new functionality to the language?", I will say:

1. There is absolutely no requirement to use this if you are not interested, it would not hurt performance, it would not break old code.
2. It will result in more usable and readable developer APIs.
3. One could argue the same about generics, which are not strictly necessary but made many patterns of programming far easier.

• Please describe as precisely as possible the change to the language.
  A function defined as plus(a int, b int) could be called either as plus(3, 4) or 3.plus(4). However, instead of the dot operator, I am looking to open discussion as to an appropriate way to signify this function chaining operation. Formally put, I propose that a syntax for calling a function starting with its first argument, then the function name, then the remaining arguments (if any) inside parentheses. I believe this would be achievable without any changes other than to the preprocessor, which could map this syntax back to nested function calls.

I would like to use this issue as a place to discuss three main things: the proper operator to represent such a chain, how to appropriate handle errors in such a context (discussed in the previous function chaining issues), and how to (if at all) handle cases such as either multiple returns in the chain and cases where the starting argument should be inserted in a position other than the first one.

• What would change in the language spec?
  There would be two ways to call a function, and possibly a new operator in this context.
• Please also describe the change informally, as in a class teaching Go.
  You could call any function starting with its first argument instead of starting with the function name.
• Is this change backward compatible?
  Yes.
• Orthogonality: how does this change interact or overlap with existing features?
  This overlaps with function calls, and nested functions. This would not change how they are handled, but would provide alternatives to calling them in this fashion in specific cases.
• Is the goal of this change a performance improvement?
  No.

Costs

• Would this change make Go easier or harder to learn, and why?
  I believe easier for a few reasons. Reading code with these "pipes" would be far more linear for a new programmer, and people with very little programming background would likely be well served by code that looks closer to plain English. (or their native language) I believe this would heavily support much more developer friendly APIs which would make writing Go easier, and it would save people from cases where they want to do something and find out there is no way, which is very grating to...certain people.
• What is the cost of this proposal? (Every language change has a cost).
  Making changes to the preprocessor in the compiler to convert chained function calls into standard function calls.
• How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected?
  I am not certain, I am tempted to say none, but anything that statically analyzes Go and doesn't have the latest version to parse it with would not understand this syntax. I think this is true of all language change proposals though.
• What is the compile time cost?
  Likely very very small.
• What is the run time cost?
  None.
• Can you describe a possible implementation?
  I should be able to throw together a little parser for this if people are interested.
• Do you have a prototype? (This is not required.)
  No.

Finally, I want to comment on the first comment on another similar proposal. I saw it suggested that this whole problem could be remedied by allowing variables to shadow each other, while I think this may help it is a greater internal change than this, and it still does not support the concise and inline style of this change, while also introducing the line noise of foo := for every transformation.

I know this proposal is very similar to others, but I request that it be considered independently and with an open mind. I am intentionally trying not to limit this to any specific implementation that I have in my head, I want to take feedback from the community. I have provided some things that this supports that I find interesting - I would love others to continue to provide more of these. If anyone else has different changes that they think would support the same things proposed here, I would also love to hear them.

[Merovius]

I don't think using . is a good choice. It means foo.bar.baz() could mean three things

1. Calling the method baz fn the field bar of the struct foo (that's today the only choice)
2. Calling the function baz from the package bar with argument foo
3. Calling the function baz from the current scope with argument foo.bar

It's not a syntactical ambiguity - it's all just a call expression with a selector expression - but it's at least confusing to read. There would also be the question of what to do if multiple of these are defined - e.g. if there is both a function baz in scope and a package baz. Again, this can probably be resolved and as packages are usually lower case and lower-case identifiers are not exported, this probably won't happen a lot in practice. But we'd at least have to say what happens, if nothing else then for good error messages.

So while we can use . (these issues are solvable), it seems like a bad choice.

[edit] oh, actually your 3.plus(4) example does generate a genuine parsing problem, as 3. is also the beginning of 3.5, which is a floating point literal. I'm not sure how much of a problem it would present, but it's at least a remark to keep in mind [/edit]

[edit2] After 3.e4, should the parser expect () (calling the function e4 with argument 3), or the end of the expression (a floating point literal 30000)? So I think there is a real parsing ambiguity here [/edit2]

I also just don't like this proposal in general. I believe it leads to less readable code, not more readable code. It encourages writing long, chained statements and DSLs. I just don't like that style of code and find it hard to understand, personally. But that's just a personal aesthetic preference.

[MSE99]

chained statements and DSLs. I just don't like that style of code and find it hard to understand, personally. But that's just a personal aesthetic preference.

@Merovius We're not chaining statements but function call expressions, don't you think that this:

double := func (x int) { return x * 2 }
overFive := func (x int) { return x > 5 }
add := func (x, y int) { return x + y }

reduceSlice(
  add,
  filterSlice(
    overFive,
      mapSlice(
        double,
        []int{1, 2, 3, 4, 5}, 
      ),
  ),
)

is a lot harder to understand than this

[]int{1, 2, 3, 4, 5} 
-> mapSlice(double)
-> filterSlice(overFive)
-> reduceSlice(add)

I'm not suggesting anything here, just trying to explain that having a nice way of doing function composition together leads to more readable code, I'm also aware someone can also take the first snippet and do this to it

double := func (x int) { return x * 2 }
overFive := func (x int) { return x > 5 }
add := func (x, y int) { return x + y }

doubled := mapSlice(double, []int{1, 2, 3, 4, 5})
filtered := filterSlice(overFive, doubled)
result := reduceSlice(add, filtered)

but this approach introduces new problems:

1. You need temporary variables to hold the values
2. These variables pollute the scope and most likely won't be used much after declaration
3. The worst problem: you need to figure out a pretty good name for these temps

[Merovius]

@MSE99

We're not chaining statements but function call expressions, don't you think that this: […] is a lot harder to understand than this

My position is that neither is particularly readable. My position is that this should be put into its own function, named by whatever it is actually trying to achieve (your example is too contrived to take a stab). At that point, I don't think the syntax matters very much.

That's basically my point. The second form is undoubtedly easier to write, but (on second look, I misread. It's not even that) I think it is less readable than code which isn't chained at all, so I don't want to encourage chaining anymore than needed. Higher level functions like Map, Reduce and Filter have their use, but to my aesthetic preference, that use is exhausted after one, at most two levels. I find code chaining more than that hard to understand.

Your mileage may vary.

[asegs]

oh, actually your 3.plus(4) example does generate a genuine parsing problem, as 3. is also the beginning of 3.5, which is a floating point literal. I'm not sure how much of a problem it would present, but it's at least a remark to keep in mind

Since Go does not allow function names to start with a number or currently support anything where a floating point number can be defined with a constant and variable part, like 3.x or x.5, I do not believe that this is an issue.

I hadn't seen the 3.e4 style of literals before, but it would be trivial to create a regex for excepting this case: /\d+.e\d+/. These are the edge cases I was interested in when I made this proposal, so thank you.

Also, @Merovius, you and I have already discussed the use of the . operator here heavily, and I tend to agree that it would introduce real ambiguity. I personally lean towards ->, but since the <- operator has a very novel meaning in Go, I would like to be certain that new programmers would not be confused, so I would also be satisfied with the Elixir pipe operator, assuming it isn't copyrighted ;).

And @MSE99, I really like your choice of example here:

[]int{1, 2, 3, 4, 5} 
-> mapSlice(double)
-> filterSlice(overFive)
-> reduceSlice(add)

as it was similar to the ones I was testing when I first started this proposal. To give Go credit, this is almost possible now with generics, if methods could be generic it would work with .. However, since this seems to be impossible, this is my alternative route to achieve more or less the same thing.

[fzipp]

I hadn't seen the 3.e4 style of literals before, but it would be trivial to create a regex for excepting this case: /\d+.e\d+/. These are the edge cases I was interested in when I made this proposal, so thank you.

Don't forget hexadecimal floating point number literals like 0x1.AaBbCcDdEeFfp0

[asegs]

Regarding the case of defining the function e1 (i int), you are right that this would introduce look-ahead into the parser, just in the sense that you are scanning for an open (, and similar goes for you @fzipp.

These are more good reasons against using the . operator for function chaining, but the -> operator resolves these issues.

[asegs]

And regarding the taste discussion that @Merovius has brought up, I find another area more compelling, I think that for a large number of Go developers (just maybe not the ones on the GitHub forums) such a syntax would be valuable and highly satisfying.
However, I think the biggest issue would be IDEs, they would need to be amended to suggest the proper functions when someone types foo ->, which right now results in nothing but would now need to look up all functions beginning with something of the foo type. I'm not sure how heavily the compiler is used in an IDE, so maybe this would be a non-issue since IntelliJ supports it for Elixir, but it is something to think about.

I understand that for some this is antithetical to their taste, and that similar proposals have ranged from negative to overwhelmingly downvoted. I also want to point out that the post requesting type parameters on generic methods was around 229 upvotes and 7 downvotes last time I checked. To me, this says that people want to do these things, just with methods instead of functions.

If the official opinion of the Go contributors and core team is that generic methods are impossible given the architecture of Go, and that generic methods are highly desired by the Go user community, we should pursue some way of giving people that functionality. My proposal is one option, there may be others, such as actually solving the problem through the type system. I am not asking for my exact vision to be merged into the language. I am asking for Go to provide some way of chaining generics for the purposes shown in my proposal. I believe that my proposal is the most flexible of the options I have seen, since it supports calling functions chained on primitive values as well as something that looks very close to default final methods for interfaces. For some, this flexibility may be unattractive or seem unnecessary, which is a perspective I can't argue with. But I think it is clear that Go developers are trying to chain generics, and it isn't going well.

[MSE99]

A while back there was lots of discussions on how to improve error handling, i think with this proposal we can simplify error handling by using a Result wrapper type:

type Result[T any] struct {
  err error
  val T
}

we can then use this like:

parseToken(r)
-> extractUserID()
-> loadUserFromDB(ctx, db)
-> authenticate()
-> sendLoginAlert(ctx)
-> handleFailure(ctx)

Every function in the pipeline above accepts and returns a Result struct, if the result contains a non-nil error it won't do anything and will simply return a new result object with the same error, if the result isn't an error it will do it's processing and wrap the result inside of a new result object, the last function handleFailure(ctx) is only interested in the error, which might add additional information to the error or do some logging.

[asegs]

@MSE99 - I don't necessarily like (or dislike) this, but this is a very cool thing that looks nice with this syntax. I actually think it is really cool that you can have result Result[int] in a signature, as an aside. The only issue I have with this pattern is it somewhat constrains use of these functions to being used in sequence, unless you are willing to do something like Result.of(3), similar to Optionals in Java. But I don't mind it and I think this is really well served by generics. I almost wonder if you could do this whole pattern without this syntax using just generic interfaces? From tinkering in a scratch file, it seems like this all doable with methods.

func (r Result[int]) loadUserFromDb(...) {
 //
}

[Merovius]

@MSE99 Another way to look at it, though, is that the syntax is significantly less useful if we don't have a pretty widely used Result[T] type (and/or tuples). The same goes for most concepts borrowed from functional languages, FWIW - they usually rely on tuples and/or currying to really work smoothly.

Also, I don't generally think Result[T] on it's own - without actual sum types which statically guarantee that only one of the fields can be set - really improves error handling.

[timothy-king]

Provide some way of chaining functions, either using a new operator such as ->, |>, or .. or by using the . operator used for calling a method to also call functions.

Thank you for pointing out #47680. It is roughly exactly adding a new operator ->, |>, or ... That was closed, and this part may be a duplicate. The conversation so far makes it look like the "." option has some issues #56283 (comment) . Have things changed since the issue was closed? Do we have new evidence? New arguments?

[timothy-king]

Higher level functions like Map, Reduce and Filter have their use, but to my aesthetic preference, that use is exhausted after one, at most two levels. I find code chaining more than that hard to understand.

Personally my mileage does vary. I have written a modest amount of things like FlumeJava (paper) where 5-6 functions being chained is pretty normal. The types got long and often get in the way more than they elucidate, and naming intermediate results is not more readable. To be fair though, I have also seen aggressive applications of Map and Reduce get in the way of readability compared to a simple for loop in other contexts.

[asegs]

Nope - nothing that I know of has changed. To be fair, I don't think that issue was closed for a specific fault, just a lack of interest.

Since then, there has been considerable discussion around function chaining in the generic methods thread. I opened this to separate those conversations since there is no other place that I know of for such discussions right now.

I would like to use this to get an answer of either "yes, we will work on a way of chaining function calls" or "no, we will not do that, but we are planning to figure out generic methods". Either of those answers address my concerns.

[Merovius]

@asegs

I would like to use this to get an answer of either "yes, we will work on a way of chaining function calls" or "no, we will not do that, but we are planning to figure out generic methods".

To be clear, if the answer is "we don't want UFCS and as far as we know generic methods are infeasible", you should be prepared to accept that as well.

[asegs]

Just a repeated nitpick - what I am proposing is not UFCS. And that's fine, for me though it would still beg the question of how to satisfy everyone requesting generic methods.

[Merovius]

it would still beg the question of how to satisfy everyone requesting generic methods.

My point is that the answer "they won't be satisfied" must be acceptable. Just as people who wanted generics had to stay dissatisfied for ~10y of using Go. If we can't figure out how to solve a problem satisfactorily, we will err on the side of not solving it. That's how Go is generally developed.

[asegs]

Ok, sure. I'm just saying this is a practical change that enables a lot of useful functionality while not requiring years of waiting and major changes. If that isn't the Go philosophy, that's fine, but I believe such an approach turns people away - I can't think of another mature programming language that gives me the answer of "no, you just can't do that."

[Merovius]

I can't think of another mature programming language that gives me the answer of "no, you just can't do that."

I'm not as sure about that. One example that comes to mind for me is Python refusing Tail Call Optimization, even though many people are asking for it. To me, that's basically what language design is all about - making choices, to build a coherent product.

As I said, there is precedence. We told people for 10 years that they just can't use generics, unless and until we figure out how to implement them. We can tell them for another 10 that they just can't use generic methods, unless and until we figure out how to implement them. It's not a satisfying answer, but sometimes that's just how it is to work collaboratively in an open source project.

[asegs]

Yeah, that's a fair example, albeit one that I would like to be added to Python. I think we both have our opinions here so I suppose we can accept that. I think this case is different from generics because there is a path forward with little work required but you (as well as others) find it distasteful enough that you prefer to wait for an alternative. That's personal preference, and I appreciate that we can both be open about where we stand.

[timothy-king]

enables a lot of useful functionality

The status quo of generic functions + intermediate variables do cover the requested use cases and functionality for both this proposal and generic methods on #49085. There does seem to be a problem that this is not a more obvious conclusion. It is cheaper to write tutorials and examples explaining that this pattern can be used instead of generic methods than it is to adopt and support new syntax. Education about generic functions vs generic methods and then 'wait-and-see a couple of years' could make a good deal of sense.

[asegs]

Ok, true. Replace functionality with expressiveness or readability.

[ianlancetaylor]

I think it's clear that we will not adopt this proposal using dot as the syntactic marker. Dot is heavily overloaded in Go. We don't need to overload it further, especially given that the result is going to look ambiguous to the reader even if it is not technically ambiguous.. Of course this concern does not apply to other syntaxes.

That aside my concern is that in practice many Go functions take a context.Context as their first parameter and many Go functions return an error as their last result. As far as I can tell neither will work with this syntax. So while the syntax is clearly useful for some specific examples I'm not sure it will be widely applicable.

[asegs]

If I were to submit a few examples of what error handling/multi-return could look like in this thread, would you be open to exploring this proposal and similar ones? Or is your general opinion just that this sort of function chaining is not a good fit for Go and is never going to happen? Don't take that with an edge, it's hard to express tone over text, I'm just pretty busy with work but would spend some time drafting error handling syntax for this if I believed there was a possibility that it would be implemented.

[ianlancetaylor]

These language change proposals are evaluated by a committee, and I don't make final decisions myself.

That said, I think this proposal is a long shot. The way that a proposal like this might be added to the language would be if lots of people started calling for it, and if they showed existing Go code that would be made significantly clearer or simpler or safer by adopting this change.

Right now I see two thumbs up and two thumbs down in the emoji voting on the issue. That's not compelling. Disregarding generics which are kind of a special case, I think the last syntactic change we made to the language was support for new kinds of numeric literals in Go 1.13 (https://go.dev/doc/go1.13#language). That covered issues like #19308, which has 226 thumbs up and relatively few thumbs down. That is the kind of enthusiasm we would expect to see before adopting a syntax change.

[asegs]

That makes sense, thanks for the background. I truly wish the Go core team the best in finding a way to support generic methods, and if more conversation ever comes up around function chaining I would be happy to get involved since I think it is apparent that this change is important to me.

[timothy-king]

That aside my concern is that in practice many Go functions take a context.Context as their first parameter and many Go functions return an error as their last result. As far as I can tell neither will work with this syntax.

I think if one wants to "chain" functions smoothly, one would need to design the function signature for this.

Let's say we want to map a T to S function that is context aware, can have errors, and works over an abstract Collection of T. In practice, this realistically would look something like

func Map[T, S any](ts Collection[T], fn func(ctx context.Context, t T) (S, error)) Collection[S]

The Collection[S] would need to remember error results and the Collection[T] would provide ctx. That signature can be chained nicely and intuitively, ss := c |> Map(f1) |> Map(f2).

Chaining with these constraints while trying to adhere to existing coding styles would result in something like:

func Map2[T, S any](ctx context.Context, ts Collection[T], err error, fn func(ctx context.Context, t T) (S, error)) (context.Context, Collection[S], error)

It would work, but it would lead to much less readable code compared to the status quo. I am not too worried about the prefix of context.Context. Chaining a prefix of results (kinda like Currying) kinda feels consistent with the proposal. It would a bit odd to always ignore the prefix at the top level _, res, err := Map2(ctx, c, nil, f1) |> Map2(f2). That is maybe okay? And we can have a Source function create a (context.Context, Collection, error) so the first call looks consistent to start chaining.

What worries me the most is that very odd error parameter in the middle of the arguments. Returning a (context.Context, error, Collection[S]) seems even less intuitive and more error prone at first glance. Expecting users to get the values using _, err, res := ...; if err != nil { ... } would cause so many correctness problems . Instead, a Sink function could turn a (context.Context, Collection[S], error) into a (Collection[S], error). This adds up to a lot of extra boilerplate to be ergonomic, e.g. ss, err := Source(ctx, c) |> Map2(f1) |> Map2(f2) |> Sink().

(I was actually kinda surprised how poorly this went once I started adding arguments.)

[Linkangyis]

似乎有点影响代码可读性

[asegs]

Regarding the comment from @Linkangyis which I just Google Translated, I find that more true with the |> operator than the -> operator. However, I think that the -> operator also lends itself well to anonymous function shorthand, but I guess => also works for that. I think in simple cases, this makes code more readable, but I partially agree that if we were to add an inline syntax for error handling and using the nth returned value instead of the first always, it could become a bit difficult to read as quickly.

[ianlancetaylor]

Based on the discussion above, this is a likely decline. Leaving open for four weeks for final comments.

-- for @golang/proposal-review

[ianlancetaylor]

No further comments.