Results of TC39 presentation

[domenic]

I'm sorry to say that the committee declined to accept this proposal as-is. In the end, the concern (largely driven by @erights, although others were sympathetic) was that escaping cannot be done in a way that is totally safe in all cases, even with the extended safe set. For example,

new RegExp("\\" + RegExp.escape("w"))

is a hazard. (It does not matter that "\\" by itself is not a valid regex fragment. The above does not error to indicate that; it just silently creates a bug.)

Note that even if you attempted to correct this by escaping all initial characters, you then have

new RegExp("\\\\" + RegExp.escape("w"))

as a bug. @erights called this the "even-odd problem."

The general feeling was that to be completely safe you need a context-dependent join operation. The feeling was then that if author code wants to do unsafe escaping, the function is easy to write, but if something is going to be standardized, it must be completely safe. The idea that other languages are not held to this standard did not convince them, I'm sorry to say.

The committee recognized that you might not be willing to do work on a different, more complicated proposal. But, if you were interested, they think that a template string tag solution would be the best to investigate. Such a solution would be able to use the context of each insertion to do a safe join operation between any two fragments, instead of depending on string concatenation. Template strings can also be twisted to work in dynamic situations (e.g. those that this proposal would cover via new RegExp(pieces.map(RegExp.escape).join("/"))) by directly calling the tag function, probably with an adapter to work through the awkwardness of the parameter form for template tags. So this would be strictly more powerful. This was also preferred (for reasons I don't really remember) to a building-block approach of e.g. RegExp.concat plus RegExp.escape (used as RegExp.concat("\\", RegExp.escape(x))).

I'm pretty disappointed by this, and am sorry you and others sunk so much work into it with such an outcome. But, what can we do.

[jdalton]

I think most regexp escape methods are pretty simple. At least the ones in Closure Lib, Dojo, Lodash, Prototype, MooTools, Mout, YUI, & escape-string-regexp are.

This looks like a case of taking something with clear utility, boxing it into a context it wasn't intended for, then dumping it because it didn't fit the box.

[WebReflection]

But, what can we do.

we can ignore this decision and agree on a single standard method we, developers, can use, pointing at the possible footgun in a comment within the shipped code.

We put such utility in a CDN and we live happily ever after.

[matthewrobb]

This is certainly disappointing but I'm glad the results were posted here in this way. I love reading meeting notes but it would be great to have this kind of update on each proposal after discussions.

I'm okay with a template tag but still, this proposal would end up needed within that implementation so why not bring it in?

Doing URL escaping in the browser doesn't save a developer from SQL injection if they did something like the above but no one questions the usefulness of those existing escape functions.

[WebReflection]

Doing URL escaping in the browser doesn't save a developer from SQL injection

exactly, same as escaping HTML, or using even eval ... if you want to hurt yourself feel free to do it.

String concatenation is a reality, RegExp can be created on the fly within a Function too ... what the hack is the point of avoiding methods focused on increasing security?

Between totally unsafe and "maybe some noob will do something silly like that" there is quite an abyss.

Disappointing to say the least. Thanks gosh I don't care about these decisions

[stuartpb]

...if something is going to be standardized, it must be completely safe.

Uh, then why is the RegExp constructor itself standardized?

RegExp.escape is a function that reduces the attack surface of an existing component - if you're savvy enough to know that you need to use it, you're savvy enough to know it has its limits. The alternative to this is going to be more users rolling their own broken versions, thinking that [(*.)] are the only characters they need to escape (because it's the only characters they've ever seen used).

Saying that "avoiding a false sense of security" is going to help anything here is just plain wrong.

There has to be somewhere the general public can complain directly to the people who voted against this. Right?

More discussion at http://discourse.wicg.io/t/regexp-escape-str/832

[stuartpb]

The trouble I see with this is that it all comes down to arguments based on the behaviors of a hypothetical "average" developer nobody involved in the discussion would have any direct experience with, almost by definition.

The point that I would make is, what real code do they see ending up broken, that would not have been broken without its presence? The TC's argument, I feel, is that not having a RegExp.escape provided by the language would somehow drive developers to look up the documentation once they encounter this problem, and become enlightened - but, from my experience, for the type of developer who would think on sight that RegExp.escape is some kind of panacea, that's not what would happen. I find "average developers" (the type who would not realize RegExp.escape would have corner cases) tend to operate more in one of two modes:

• Use something they've seen somebody else use before
• Try to figure out how to do it using other things they've seen used before

And, without being given a RegExp.escape, they would not become enlightened as to how to do it correctly. They would (and do) try to fix it using the exact problem in front of them, using only what they've seen before, with as little thought into other cases where it would be a problem as possible - and that's going to generate a hell of a lot more corner cases than RegExp.escape ever would.

[benjamingr]

Thanks for the update. This is disappointing, I'll read the meeting notes and figure out how to go forward.

I definitely don't think that a completely safe tag is that way forward though.

[erights]

Historical note: When Domenic first proposed this, we all thought that the maximally escaped form could be used reliably. While we thought that, I was in favor of it. The even-odd example took us all by surprise. I changed my mind only once I understood the unfixability of the even-odd problem.

[erights]

I definitely don't think that a completely safe tag is that way forward though.

Hi Benjamin, why not?

• First, if we had such a tag, it would subsume the functionality you desire:

RegExp.escape = (str) => RegExp.tag`${str}`;

• Second, it would help developers avoid the hazards.

I emphasize the word "help" above because the claim that we insist on something completely safe is not the way I would put it, because it leads to the misunderstandings seen earlier in this thread. Nothing will prevent developers from writing code unsafe in these regards. We're trying to make the safe way the easy way, so that a set of fairly straightforward design rules generally suffices to avoid these hazards.

The issue is simple. RegExp source text is a little mini programming language with its own escaping rules. We've already heard the following song before: Having functions that provide local escaping fit for some contexts combined with use of string append to compose together source text from fragments. This has led to the nightmare that pervades the net, of escaping errors and injection attacks in html, css, sql, and more. We took a memory safe language, JavaScript, and by use of these manual-escape-and-append patterns, we reiterated the buffer-overflow-like vulnerability at a higher level of abstraction.

Lisp, when embedded languages are composed of S-Expressions, is largely free of this entire class of vulnerability because they compose structure to create structure. All the relevant source text is first passed through a reliable reader and low-level universal parser before this composition happens. Lisp quasi-quote makes this especially pleasant. Of course, we do not have a universal textual syntax like S-Expressions that we're starting from. We created the template string mechanism (and E's earlier quali-literal mechanism) as a universal framework for solving the context-dependent escaping problem for embedded languages, each with their own surface syntax.

[erights]

Btw, in case it wasn't clear, I do not propose that the tag be named RegExp.tag. This is merely a standin for whatever the tag would actually be named.

[stuartpb]

Historical note: When Domenic first proposed this, we all thought that the maximally escaped form could be used reliably. While we thought that, I was in favor of it. The even-odd example took us all by surprise. I changed my mind only once I understood the unfixability of the even-odd problem.

I feel like you've let this flaw blindside you, then. The nature of this kind of channeling attack has been a known issue with escapement for years - while I'm not going to argue against the value of a better model that checks individual RegExp components for validity, I will argue that it's short-sighted to say such a solution is the only adequate one. The more an end-developer (particularly one who's busy and only wants lo learn what's necessary to do their job) has to learn and adapt to make their solution "correct", the more likely it is that they're going to disregard that "correctness" in favor of doing it the maybe-sometimes-wrong way they already understand.

RegExp.escape is something we already get. It's something we already want. It's something we already use, and already works with all the code we already wrote. It's something we already know how to use correctly to prevent channeling attacks - without it in the language, I'm more likely to implement a broken solution, because there's a lot more effort involved in going out and looking up how to polyfill this than in taking half-a-second, deciding "yeah, I'm pretty sure it doesn't matter", and not escaping the string at all. If I had a RegExp.escape at hand that I could take, check I'm not concatenating with anything that might include uncontrolled backslashes, and drop in with 13 keystrokes, this would be a non-issue.

[benjamingr]

@erights

Well, the thing is that RegExp is not the same sort of language as SQL or HTML in this regard, there have been zero "RegExp injection" attacks (in this sense), there is no security guarantee to uphold here. I've been very in favour of contextual escaping for languages where not doing so presents a security risk.

The problem is that making a tag severely complicates this proposal. I mind but am willing to do the extra work on my side, but this means it will much less likely get implemented in browsers, will take a lot more of the TC's time and so on.

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it - I feel like it would require a lot of effort to solve a problem a tiny bit better and it would create a less simple API. There is no security risk here like SQLi or XSS - just the lack of support of truly esoteric use cases.

We have scanned the top million websites, have gone through tens of thousands of NPM packages and explored a total of over a billion lines of code. I did not see a single reasonable case that was not caught by the original "SyntaxCharacter" proposal.

This trade between pragmatism and theoretical soundness is something I think ES needs to do here, just like C#, Python, Java, Perl, Ruby and pretty much every other language did before it. Just because a theoretical edge case exists doesn't mean we have to entertain it or focus our efforts based on it.

[erights]

I see here two interesting arguments for advancing RegExp.escape

• Composing a dynamic number of strings, as in Domenic's example.
• The extra work and delay for getting RegExp.tag specced and implemented.

I understand the force of the first argument and agree that it needs to be addressed. But I do not accept the second argument. A short term fix that become a long term hazard is a common development strategy and often appears in libraries that become de facto standards. When no good long term solution is known at the time, they often become de jure standards as well.

However, when a better long term solution is known at the time, and subsumes all the functionality and convenience of the short term fix, I think the committee would be failing the community to admit the short term fix merely because the good long term solution is more work. Admitting it into the de jure standard means we're likely to be stuck with it forever. We're forever stuck with enough of these already.

On the convenience point, compare

RegExp.escape(str)

and

RegExp.tag`${str}`

Once both exist, is there any reason whatsoever to prefer writing the first to the second?

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it

Thank you. I'll be happy to help.

[WebReflection]

I am really having hard time trying to understand what's the tag supposed to solve compared to escape ... this is valid ES6, right ?

new RegExp("\\\\" + RegExp.tag`${str}`)

How does the user that would apparently fail with the initial example, avoid failing with the tag too?

Thanks for any sort of clarification, I know TC39 has always reasons but this time I really can't figure out the what, the where, the when, and the how.

Best Regards

[benjamingr]

@erights

So, while a safe RegExp.tag method is interesting and I'm willing to continue exploring it
Thank you. I'll be happy to help.

I'd love that, I think it would be interesting to apply the same formal analysis techniques you use in your work to show that the escaping is sound. I'm not sure it's something I'd put in the world's most popular programming language or that it would make a good API for people - but if you could point me in the right direction and tell me where to start this sounds like an interesting thing to do regardless of whether or not we build a language API out of it.

---

The arguments for RegExp.escape as I see it are:

• It's a de-facto standard, as @jdalton pointed out there are libraries with tens of millions of downloads that do this (with the less-strict escape set).
• It's a real problem programmers have been having for years, and we can't up with non-contrived cases it does not solve (and boy we tried). Moreover exploring a huge number of code bases does not show anyone doing anything remotely close to "\\" + escape("w") - so it's a problem that exists in theory.
• It's a simpler API for users, a fixed escape set is easy to reason about and we're only claiming that escape(s) matches s which matches the vast majority of use cases.
• It's shimmable very easily, so everyone using non ES2015 code bases can use it. I get this might not a hold of weight for the TC but it means a lot to a lot of developers.
• Composing a dynamic number of strings. This is probably the most common use case for RegExp.escape actually - building a routing table or a search pattern for multiple strings.

An idea was brought up here before (I think by @cscott) to support a second argument in RegExp.escape allowing more escaped characters (this is the standard API in some other languages, namely PHP). This means that the RegExp.escaped (tag) proposal can be easily built on top of it. Would this meet your concerns about contextual escaping?

[benjamingr]

@benjamingr please don’t do that; that is a much worse outcome than never having it at all.

I feel like if I go through here I will spend another 5 years waiting and the API people are actually asking for (.escape) won't happen. I fail, I fail the users who have been rooting for me, I fail the ecosystem.

On the other hand if I make a PR to Node and coordinate with WHATWG I get the API I've been asking for since 2015 in about a week. The interaction with WHATWG has been significantly easier than TC39, browsers would (most likely) also want this and Node sure does.

So please help me understand why I would rather go through TC39 for something that is borderline platform and not language anyway (in other languages and ecosystems it's half-and-half)?

I like you and (full honesty) I like Mark. Both of you have helped me in the past with a bunch of things when I needed help both in private and in public. What I don't like is the process and how skewed it is towards concerns that our users in Node frankly don't care about as you can see from this (and the other repo issues).

Please give me a reason to trust the committee on this or faith this will progress at some point.

[ljharb]

The presentation I made that reactivated the proposal at stage 1 is the entire committee giving consensus to reconsider it, which woudn't have happened if anyone's position remained immovable. I'm sorry I haven't made the time to do more work on it since then, but I can assure you that, at least, I am confident it has a real shot at advancement.

[benjamingr]

@ljharb ok, let me know how it goes