Refusing a mix of numeric-only and BIDI domains

[vorner]

Hello

Some time ago I was trying to figure out if the domains below were rejected by the Rust url crate, it is tracked here. It seems this is maybe accidentally disallowed by the standard. I was recommended to raise it here.

It's a bit old so I don't remember the exact details and would have to dig them up, I tried to describe it in this comment. I think the issue was the combination of numeric only label and BIDI label.

Now, my question is, should these be valid URLs? They certainly are valid domains, even though it might be discouraged to allow them and the URLs are (were at least when it was reported; I could provide new ones if needed) alive and reachable. Note that they are considered malware URLs, so be careful when handling them.

Parsing failed: invalid international domain name, http://mail.163.com.xn----9mcjf9b4dbm09f.com/iloystgnjfrgthteawvo/indexx.php
Parsing failed: invalid international domain name, http://shdedgelanimailnoticeborad.count.mail.163.com.xn----9mcjf9b4dbm09f.com/sitemap.html
Parsing failed: invalid international domain name, http://count.shdedgelanimailnoticeborad.count.mail.163.com.xn----9mcjf9b4dbm09f.com/bvv
Parsing failed: invalid international domain name, http://count.shdedgelanimailnoticeborad.count.mail.163.com.xn----9mcjf9b4dbm09f.com/index.php
Parsing failed: invalid international domain name, http://count.shdedgelanimailnoticeborad.count.mail.163.com.xn----9mcjf9b4dbm09f.com/iloystgnjfrgthteawvo/index.php

[TRowbotham]

These domains are considered invalid because they don't meet the criteria from RFC 5893 Section 2. Specifically, the label "163" fails criteria 1, which requires the first character of a label to have a Bidi property of L, R, or AL. The digits 0-9 have a Bidi property of EN (European Number) 0030..0039 ; EN # Nd [10] DIGIT ZERO..DIGIT NINE according to the DerivedBidiProps.

The domain to ASCII algorithm sets the CheckBidi option to true, which causes the result of Step 2 to return a failure value due to not meeting the above criteria, which is then rejected in Step 3 and ultimately leads to the host parser returning a failure, which then causes the the URL parser to abort.

RFC 4920 Section 1 states:

If any step of the ToASCII operation
fails on any label in a domain name, that domain name MUST NOT be
used as an internationalized domain name.

So, the URL spec is doing the right thing here. The only 2 options for making these domains valid in terms of this spec, as far as I can tell, would be setting the CheckBidi option to false or allowing the options for the Unicode ToASCII steps to be user configurable.

[vorner]

I agree that the current spec disallows these URLs as invalid.

My question was more in the line of „Was the spec's/author's intention to disallow them, or did the spec got written in a way that it disallows them by accident?“

Several sections in there, pointed out in this comment seem to suggest that allowing such URLs as compatibility with existing deployments was considered.

So, in other words, my question is not „Are they invalid“, but „Should they be invalid“?

[domenic]

This might be the same underlying issue as #438.

[annevk]

I think @TRowbotham has the correct analysis here and indeed it very much depends on how CheckBidi is used.

To simplify from OP:

• xn--mhb is fine in all browsers.
• 1.xn--mhb errors (though does not error in Gecko, presumably it has CheckBidi set to false; also doesn't error in Chromium, presumably due to an erroneous ASCII fast path).

However, https://www.rfc-editor.org/rfc/rfc5893.html#section-2 (which UTS46 invokes) also says:

In a domain name consisting of only LDH labels (as defined in the
Definitions document [RFC5890]) and labels that satisfy the rule,
the requirements of Section 3 are satisfied as long as a label
that starts with an ASCII digit does not come after a
right-to-left label.

But that seems contradictory as a label that starts with an ASCII digit can never fulfill The Bidi Rule due to ASCII digits not having the correct Bidi property (they have EN according to https://unicode.org/reports/tr9/):

The first character must be a character with Bidi property L, R,
or AL. If it has the R or AL property, it is an RTL label; if it
has the L property, it is an LTR label.

I'm not sure what to make of this.

I would appreciate input from @achristensen07 @valenting @markusicu @macchiati @alvestrand. I would be somewhat inclined to set CheckBidi to false given that it matches most implementations, is more likely to match deployed content, and the bidi requirements appear contradictory, but I'm open to suggestions.

[annevk]

This might also be more complicated still as, e.g., يa is rejected by all browsers. Which I think is due to mixing L and R character properties and would not be rejected without CheckBidi being true?

And then 1.ي is only rejected in Chromium and WebKit. So Chromium can again be explained through an erroneous ASCII fast path. And it seems that Gecko has a different CheckBidi behavior when it comes to ASCII digits at least, perhaps due to the above contradiction. Or perhaps there is another check related to character properties unrelated to CheckBidi.

(All of this is only concerned with the ToASCII code path, for what it's worth.)

[alvestrand]

This logic took quite a while to work out, including actually coding up the BIDI rule and running it through all the possible combinations of directions to make sure I had them all covered.....

The "bidi rule" in RFC 5893 section 2 applies to a single label. So a label (not a domain name) can either obey the rule or not.

The guarantees in the last two paragraphs are about the properties of a whole domain name. They are not part of the rule.

The practical consequence is that if you want sanity in your display, you can never have <RTL-label>.3com.com - because that would probably display as 3.<RTL-label>com.com, which is confusing.

So 1.ي should not be rejected, but 1.ي.3com should be. (inspect the order of the characters in that one!)

[annevk]

@alvestrand so do I understand it correctly that IDNA2008 doesn't take a stance as to whether all labels in a domain need to obey The Bidi Rule?

https://www.unicode.org/reports/tr46/#Validity_Criteria does which might explain the difference.

https://www.rfc-editor.org/rfc/rfc5891.html#section-4.2.3.4 seems to only enforce The Bidi Rule upon individual labels containing characters whose property is R whereas UTS46 enforces it upon all labels in a domain as long as CheckBidi is true.

I do think The Bidi Rule is somewhat confusing if that is the case as it itself states

The following rule, consisting of six conditions, applies to labels
in Bidi domain names.

which easily leads one to think it applies to all labels and has to be obeyed.

Also, it's not clear to me how from The Bidi Rule enforced only upon labels containing characters whose property is R the guarantee follows that labels starting with an ASCII digit do not come after the RTL label.

[alvestrand]

You are correct. IDNA2008 states only rules about single labels - this was a result of discussing the various ways in which labels can be put together into domain names.

There is a very explicit discussion of "what can happen if you concatenate labels into domain names" in https://www.rfc-editor.org/rfc/rfc5893#section-5 - it ends with "Rather than trying to suggest rules that disallow all such undesirable situations, this document merely warns about the possibility, and leaves it to application developers to take whatever measures they deem appropriate to avoid problematic situations."

TR46 was written by people who have far less DNS experience than the people who were involved in RFC 5891. The two groups did not agree at the time TR46 was first written, and while my impression is that TR46 has been revised to be more in line with IDNA2008 over time, I am not surprised that there are still cases where trying to interpret the two as saying exactly the same thing will fail.

[annevk]

@alvestrand okay, but that still doesn't address my last paragraph about the purported guarantees from IDNA2008.

[alvestrand]

The point is that no single entity can make that guarantee, as described in section 5.
Remember that IDNA2008 intends to impose requirements on people registering labels; it does not impose requirements on those who use domain names.

If you want to require that a certain application rejects domain names that don't obey the requirements, that's an application spec, not a DNS spec. Section 5 (and the "it follows" parts of section 2) are intended to give guidance on how to decide to reject such names.

(I suspect that I'm a victim of knowing what I intended when I wrote it, and being unable to see where it's unclear; to me, I'm just repeating what I already wrote in the RFC. But I still hope it's understandable.)

[annevk]

The point is that no single entity can make that guarantee, as described in section 5.

So why say they are guarantees?

Given that IDNA2003 was implemented by user agents it does seem somewhat irresponsible that IDNA2008 didn't try to address them at all, but I guess that's water under the bridge.

I guess I need input from @achristensen07 @valenting @markusicu @macchiati as to what exactly we'd like to enforce here. Banning numeric labels in domains containing RTL labels seems bad so I assume we want to change that part of UTS46.

Enforcing The Bidi Rule for labels containing a character whose property is R seems realistic and implemented by all user agents.

We could additionally try to enforce the second "guarantee" by not allowing a numeric label after an RTL label, but not sure.

[zackw]

I'm concerned that the URL spec might be moving in the direction of rejecting domain names that are actually in use in the wild. Gonna bring over my comment from #438:

Here are some examples of URLs that I have personally observed in the wild (during my research, which involves Web crawling) to contain hostnames which are formally invalid per some RFC or other, but do not rise to the level of a 'serious problem' (as the IDNA2008 RFC uses that term), and which I think should probably be accepted by the URL standard, if only for interop's sake:

http://r2---sn-gvbxgn-tt1s.googlevideo.com/
http://r9---sn-i3b7sn7d.googlevideo.com/
http://lgbt_grani.livejournal.com/
http://www.mi-ru_mo.bbs.fc2.com/
http://-friction-.tumblr.com/

None of these actually involve xn--, though. Should I file a new bug?