Prior difficulty persuading TC39 about F# pipes and PFA

[js-choi]

• F#-style pipes, partial function application (PFA), unary function composition (Function.pipe / flow), and currying all are techniques of a paradigm called tacit / point-free programming.
  • Tacit / point-free programming is associated with functional programming on unary functions that accommodate extra arguments by currying.
  • JavaScript is an n-ary functional programming language that lacks built-in automatic currying.
  • Tacit / point-free programming is not equivalent to functional programming.
  • Functional programming can be either tacit / point free or pointful.
  • Examples of pointful functional programming include all function/lambda literals (the vast majority), Haskell’s functor/monad do-notation, and F#’s computation expressions.
• Many people in the Committee remain strongly against encouraging widespread tacit / point-free programming in JavaScript in syntax.
  • Many of the pipe champions are not these people and are fans of tacit / point-free programming in JavaScript. However, we have to work with the ones who are not.
  • The strongest opposition to tacit / point-free programming comes from the JavaScript engine implementors, concerning performance.
    • More widespread tacit / point-free programming, especially widespread curried functions, would encourage extensive allocation and garbage collection of onetime-use unary functions.
    • The engine vendors are concerned that such changes would be difficult and complex for them to optimize for relatively little gain from their perspective.
      • This includes concerns about significantly increasing the allocation/deallocation of many onetime-use functions closures in program hot paths.
      • The engine implementers have said that they would have much difficulty with optimizing these closures by inlining.
      • This is especially difficult when the closures were created by a named function and must appear in error stack traces.
      • For example, an engine might easily inline [1, 2, 3].map(x => x + 1).
      • But given a curried add function, [1, 2, 3].map(add(1)) may be difficult to inline—especially if add can throw an error.
      • Error stack traces are observable by the developer, and any function that may throw an error must be preserved for the error stack.
    • These performance concerns are obviated if every step of a pipeline is a cached variable.
      • But this would require extensive, verbose manual caching of closures.
      • This would negate much of the advantages piping has over simply using temporary variables for each step.
      • If F#/Elixir-style pipes are available, developers will probably not bother to cache their step functions in most cases.
    • This difficulty is not present in programming languages that use function-based pipes.
      • F# & Haskell are both auto-curried, compiled, and statically typed.
        • These three traits provide a compiler with a significant amount of ahead-of-time information that can be used to optimize the program around their unary-function-based pipes.
        • JavaScript does not have these three traits and cannot optimize unary-function-based pipes nearly as well. If JavaScript adopted an F# pipe, it would be, as far as we are aware, the first language that would require the VM to allocate a new closure to evaluate the RHS of the pipe in the point-free case.
        • Notably, even F#’s guidance cautions that tacit programming should be used carefully and judiciously in F#, and it should not be used in public F# APIs.
      • Elixir is dynamically typed but inserts the pipeline value as the first argument, which obviates the requirement to allocate a new closure.
        • However, Elixir-style pipes are also limited in which parameter into which can insert—the entire language must choose to pipe through first or last parameters.
        • Elixir-style pipes are ironically confusable with calls to higher-order functions: x |> f(y) being f(x, y) or f(y, x) and not f(x)(y).
    • Furthermore, many “pipe” functions as implemented by many JavaScript functional-programming libraries like Ramda and Sanctuary avoid this concern by actually being composition functions.
      • For example, pipe(a(1), b(2), c(3)) returns a new function, which can then be called multiple times without allocating new closures, because they were allocated up front at the point the new function is created.
      • While x => x |> a(1) |> b(2) |> c(3) with F# pipes looks functionally similar, the closures need to be allocated every time that function is invoked.
      • This is avoidable if you extract a(1) or wrap it in an arrow function like |> x => a(x, 1), but now this is more verbose than Hack-style / topic-style pipes.
    • This difficulty is unlikely to change just because hardware may get faster.
      • The engine vendors care about low-end devices and they care about performance regressions.
    • The engine implementers have stressed repeatedly to us that people generally overestimate how much optimization the engines can do, including in this case.
    • This is the most serious barrier. Just like with HTML in WHATWG, the implementors have the most power.
      • The engine vendors will simply not implement what they do not agree should be implemented, and we cannot force them to.
      • The most power—perhaps rightfully—lies in the standard implementors, because they are the ones doing the work to make it shippable and real. Likewise, they are the ones who get the blame when it goes wrong.
      • If a language change makes the average website’s performance worse, then the browsers and engines will be blamed for it.
      • The browser vendors have a strong interest in being conservative with the language to avoid performance regression. The recent JSSugar proposal from one delegate from Chromium/V8 lays out some of this explicitly.
      • We cannot force browser vendors or engine implementors to do things that they disagree with or don’t want to do.
      • Such attempted compulsion to implement is why the WHATWG–W3C schism over HTML occurred in the first place, as Ian Hixie saw from the frontlines.
      • Oftentimes, the desires of many standards users (the greater developer community) have concord with the desires of standards (the browser-engine implementers).
      • At other times, they clash. This is just one of those cases. (Though note that many users also have voiced support for Hack-style / topic-style pipes.)
  • Other concerns have included that tacit / point-free programming poorly fits JavaScript.
    • Tacit / point-free programming best fits unary functional programming languages that have currying, but JavaScript is an n-ary functional programming language that lacks auto-currying.
    • Some people in the Committee believe that this mismatch also makes widespread tacit / point-free programming undesirable in JavaScript.
    • Some of them also are concerned that encouraging widespread tacit / point-free programming would cause an ecosystem schism.
• Because of this:
  • F#-style pipes and Elixir-style pipes still do not have any chance to succeed.
  • Partial function application (PFA) remains in stasis.
    • It has been proposed at least three times (2017, 2018, and 2021); see HISTORY.md.
    • Each time, it received much pushback, especially from those who are against widespread tacit / point-free programming in JavaScript.
  • My Function.pipe / flow (unary function composition) proposal also received much skepticism and has been withdrawn.
• Thus, Hack-style / topic-style pipes (the current state of this proposal) remain the only viable path forward for standardized piping on the language level.
  • We have given up on trying to standardize language features in the tacit / point-free programming paradigm.
  • There is no way around the fundamental anti-tacit barrier, and we are not interested in re-litigating it.
  • We care most about being able to linearize deeply nested expressions into fluent pipelines with linear dataflow, without resorting to temporary variables.
  • Hack-style / topic-style pipes work fine with function-based pipelines or any other API design.
  • Hack-style / topic-style pipes are also a zero-cost abstraction that sidesteps the engines’ concerns about the difficulty of optimizing hot-path onetime-use closures.
  • Hack-style / topic-style pipes’ only downside is that they aren’t tacit / point-free.
• As I said in the original 2021 version of this post, I will update the explainer with an FAQ with this explanation. I plan to close this documentation issue once I do so.

Old 2021 version

The explainer does not currently explain why the pipe champion group believes Hack pipes to be the best chance of TC39 agreeing to any pipe operator.
(See HISTORY.md for a lot of detailed background.)

During 2017, 2018, and 2021, whenever they were presented to TC39, F# pipes and partial function application (PFA) have run into pushback from multiple other TC39 representatives due to memory performance concerns (e.g., from Google V8’s team), syntax concerns about await, concerns about encouraging ecosystem bifurcation/forking, etc. This pushback has occurred from outside the pipe champion group. (See HISTORY.md for specific examples.)

It is the pipe champion group’s belief that any pipe operator is better than none (in order to easily linearize deeply nested expressions without resorting to named variables). Many members of the champion group believe that Hack pipes are slightly better than F# pipes, and some members of the champion group believe that F# pipes are slightly better than Hack pipes. But everyone in the champion group agrees that F# pipes have met with far too much resistance to be able to pass TC39 in the foreseeable future (or even ever—though I hope not).

To emphasize, it is likely than an attempt to switch from Hack pipes to F# pipes will result in TC39 never agreeing to any pipes at all; PFA syntax is similarly facing an uphill battle in TC39 (see HISTORY.md). I personally think this is unfortunate, and I am willing to fight again for F# pipes and PFA syntax, later—see #202 (comment). But there are quite a few representatives (including browser-engine implementers; see HISTORY.md about this again) outside of the Pipe Champion Group who are against encouraging tacit programming (and PFA syntax) in general, regardless of Hack pipes.

In other words, the explainer currently does not adequately explain the situation regarding the strong pushback that F# pipes and PFA syntax, since 2017, have run into from various TC39 representatives. We need to fix this sometime.

This issue tracks the fixing of this deficiency in the explainer (lack of discussion regarding the pipe champion group’s decision making, after the pushback that F# pipes and PFA syntax encountered in TC39 from outside the champion group). Please try to keep the issue on topic (e.g., comments about the importance of tacit programming would be off topic), and please try to follow the code of conduct (and report violations of it to tc39-conduct-reports@googlegroups.com). Please also try to read CONTRIBUTING.md and How to Give Helpful Feedback. Thank you!

[aadamsx]

It is the pipe champion group’s belief that any pipe operator is better than none (in order to easily linearize deeply nested expressions without resorting to named variables). Many members of the champion group believe that Hack pipes are slightly better than F# pipes, and some members of the champion group believe that F# pipes are slightly better than Hack pipes.

[But everyone in the champion group agrees that F# pipes have met with far too much resistance to be able to pass TC39 in the foreseeable future (or even ever—though I hope not).]

So minimal/F# style will ultimately never to get through TC39 based on what you have said above. I have some thoughts after this realization...

Currently, the people on TC39 board get to decide what goes into JS because the compaines they work for control the browsers. Therefore we in the JS community have no real representation or voice, and can be marginalized with just a click.

Maybe the process becomes more balanced if the corporations cede some control ( I know, good luck with that right? 😉 ).

For example, add real JS community members voted in by us on the TC39 board. And have two representatives per proposal, one community and one corporation derived respectively. This way we actually have representation and some sense of accountability -- closer participation.

Right now, this doesn't feel like participation -- it feels like subjugation of the JS community by the corporations that control TC39.

[js-choi]

So minimal/F# style will ultimately never to get through TC39 based on what you have said above.

@aadamsx: I would not yet say “definitely never”. But I would say “definitely not for a long time” and that “I plan to keep fighting for F# pipes after this but I expect continued strong difficulty”.

I hear your frustration at not feeling represented. (See also #215.) Although any individual human who can pay can join TC39 by becoming a member of Ecma, it is true that, functionally, most members of TC39 are companies.

What I would say (and which the explainer should also mention) is that, no matter how much representation a community gets in the Committee, in the end, the engine implementers will always have final say on the language.

It’s impossible to have the engine implementers, because they are the ones implementing the language. The worst case scenario would be for one browser member to refuse to implement something that got standardized and fork the language; then the standard is no longer a standard. So when an implementer says that they have deep concerns, e.g., about F# pipes’ or PFA syntax’s memory problems, then we have to really pay attention to that. There’s no point in pushing something that they will block or not implement.

Like @ljharb said a few years ago about a different proposal (tc39/proposal-cancelable-promises#70 (comment)):

One of the important priorities in TC39 is to achieve [Committee] consensus - which means that any employee of any member company can always block any proposal. It's far far better for anyone to be able to block, than to arrive at a spec that "not everyone” [not every engine] is willing to implement - this is, by a large large margin, the lesser evil, I assure you.

Having said that, I agree that it is frustrating. I can only say that I am sorry, that we the pipe champion group should acknowledge it in our explainer, that we hope you would still sometimes find Hack pipes useful, and that we’ll try our best to do right by everyone within the constraints of Committee consensus.

Anyways, this all belongs in the explainer, and we’ll add it later so we can point to it.

[mAAdhaTTah]

I also strongly disagree with the characterization of the members of the committee as separate from the community. They're all involved in designing or developing core elements of JavaScript or the web that I'm sure you've used before. Many of them have worked in functional languages, including both JS + Tab. They're not outsiders imposing outside opinions on JavaScript. They're part of making JavaScript what it is throughout the ecosystem.

That certainly doesn't mean it's a perfect system, or even necessarily a good one (I worry about Google's influence on the web, especially now w/ Edge on Chromium), but I do think language design needs to be done by experts, with community input, but not by democracy. Those outside of the committee have perspectives & experience to share but we the community are no longer exploring new territory. Despite all the back and forth, I didn't see (m)any novel arguments in favor of F#, and only a few in favor of Hack, throughout that thread. This is not to minimize the community involvement generally but to say that in fact your points were deeply considered and your input was taken into account, and Hack faced fewer obstacles and objections from the committee and would be more likely to get to consensus.

[js-choi]

Note for readers: There’s a lot of related talk about browser implementers’ concerns with F# pipes’ memory performance, starting from #215 (comment). I’m fine with that, since it feeds into frustration about TC39’s process and the feeling of F# pipes being promised and being taken away by TC39 representatives’ concerns. Maybe I should have made an issue devoted to performance concerns.

Pasting from #217 (comment): Since 2018, TC39 representatives outside of the champion group have brought up several strong concerns, several times, about F# pipes and partial-function-application syntax (both of which I plan to fight for in the future, after Hack pipes).

Memory performance is indeed one of those deep concerns, although it is not the only one.

With regards to performance, though browser implementers have said repeatedly to the group that people generally overestimate how much optimization they can do, and they feel that it probably applies to this case (e.g., whenever a curried function is declared separately, how error stack traces make inlining observable to the developer, etc.).

To be more specific, engine implementers have been concerned about F# pipes and PFA syntax encouraging the rapid allocation of new function objects in hot paths—function objects that might often not be able to be inlined. For example, the callback in [1, 2, 3].map(x => x + 1) may be inlined. But, given a curried add function, [1, 2, 3].map(add(1)) may not be inlined—especially if add can throw an error. Error stack traces are observable by the developer, and any function that may throw an error must be preserved for the error stack. It’s these sorts of concerns that have driven engine implementers’ resistance to F# pipes and PFA syntax. (@mAAdhaTTah’s comment in #215 (comment) is also a good read.)

I personally support F# pipes, and I support PFA syntax, in addition to supporting Hack pipes. But the buck stops with the engine implementers, because the engine implementers can simply refuse to implement a feature, while another engine does deploy the feature—and then the language is forked in real-world code. This would be the worst-case scenario for any standard.

Anyways, there’s more talk about engine implementers’ memory concerns, going on in the frustration issue (#215 (comment)). I’m fine with the discussion continuing there, since it’s…kind of on topic there, even though it’d be also on topic here.

All of this stuff should be documented in the explainer…which we’ll get to sometime.

[JustinChristensen]

@js-choi

I don't have the full context on the history of this, but there's been a lot of reference made recently to a first-blush impression of one of the JS engine maintainers on the performance implications of PFA and composition for a language like JS.

Given that neither of these are new ideas I would think it would be worthwhile to get some opinions from those that maintain optimizing compilers for mainstream languages that have implemented these constructs so that the JS community can better understand what the actual runtime implications will be, and where we'd be able to optimize if we went down the PFA/composition route.

Perhaps the JS standards community could request a short document from the F# compiler toolchain maintainers that explains how they've chosen to represent these constructs and where they've been able to optimize within their toolchain?

[JustinChristensen]

Really, given the size and importance of a language like JS, and the scope of the proposed changes, it shouldn't be out of bounds to get a general idea of the proposed JS engine implementation for either approach before making a decision to extend the language.

[mAAdhaTTah]

Engine implementers are not going to spend engineering time implementing or even designing for a Stage 1 proposal with an uncertain future, so that sort of things isn't typical for language design (related to #216).

[JustinChristensen]

@mAAdhaTTah

They don't need to. I suggested reaching out to the those that have already done this for other languages, and not JS engine implementers.

[JustinChristensen]

@mAAdhaTTah

Also, the precise stage of a proposal shouldn't matter in the slightest if it's being weighed as a real option against another proposal that is on the same track where one of the two is very likely to succeed. It's pretty clear that the JS community wants some form of function composition. It's just a question of which design to go with, and peeking behind the curtain at possible implementations should be done.

[mAAdhaTTah]

I explained here why that comparison wouldn't be valid.