Username with German or French letters gets encoded and passes validation ASP.NET Core Identity

[Ponant]

On .Net Core 3.0 Preview 3, Asp.NET Core Identity Template, register a user with username
bernhard@günter.com
Expected
Validation responds that this is not an allowed character for a username since the default is
public string AllowedUserNameCharacters { get; set; } = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-._@+";
Result
bernhard@xn--gnter-kva.com is stored in the db as username and email and displayed in the UI!

That is pretty unexpected for an old framework and a handicap in the current implementation.
Probably this has been encountered in the past??

[Ponant]

A first issue comes from the template having the [EmailAddress] attribute. If that is commented then günter is not formatted and the response is as expected "User name 'bernhard@günter.com' is invalid, can only contain letters or digits.", which btw is a wrong error description since other characters are allowed by default (-._@+) .
The problem remains when someone wants to collect an email address and puts in the [EmailAddress] he will collect a wrong email which can have lots of consequences.
Furthermore it would be good to have a UserNameAttribute class that would validate the username according to the allowed characters instead of going to the server.

[Ponant]

The same issue happens with [DataType(DataType.EmailAddress)]

[blowdart]

That's odd. @HaoK ?

[HaoK]

There's two kinds of validation going on, the MVC/model data validation attributes which control the views/UI validation on the forms, and then there's a set of identity specific properties that do things inside of the identity methods.

The templates use email as the username and set it as so, which is why that happens. It also does it in two steps, it updates the email ( which works), and then tries to update the username which fails.

The relevant code which clearly can result in out of sync username/emails:

                var setEmailResult = await _userManager.SetEmailAsync(user, Input.Email);
                if (!setEmailResult.Succeeded)
                {
                    var userId = await _userManager.GetUserIdAsync(user);
                    throw new InvalidOperationException($"Unexpected error occurred setting email for user with ID '{userId}'.");
                }

                // In our UI email and user name are one and the same, so when we update the email
                // we need to update the user name.
                var setUserNameResult = await _userManager.SetUserNameAsync(user, Input.Email);
                if (!setUserNameResult.Succeeded)
                {
                    var userId = await _userManager.GetUserIdAsync(user);
                    throw new InvalidOperationException($"Unexpected error occurred setting name for user with ID '{userId}'.");
                }

[Ponant]

I disagree with the arguments.

1. The [DataType(DataType.EmailAddress)] or [EmailAddress] will encode the umlaut character ü silently and transform it somewhat to some combination of dashes and dots and some alphanumeric (see above), and that will end up in the database and ui for both username and email since now the former is encoded with dashes and the like, so it passes validation, see:

   private async Task ValidateUserName(UserManager<TUser> manager, TUser user,
            ICollection<IdentityError> errors)

A proper attribute or whatever logic would stop me from sending an encoded umlaut to the server. Somehow we should make sure that the user input is properly sent to the db and later the ui.

2. This violation occurs when the umlaut or a French accent is in the domain name. güter@gmail.com would not pass the validation on the client, even if js validation is turned off

3. A round trip to the server is needed to validate the UserName, I am now working on having something validating client-side following the rules of AllowedUserNameCharacters

4. The InvalidUserName key the resx file has a value which is not consistent with the default values of the AllowedUserNameCharacters of the IdentityOptions

It would be good to have feedback from some people who I think are used to German, e.g.
@leastprivilege or French @PinpointTownes

[kevinchalet]

The [DataType(DataType.EmailAddress)] or [EmailAddress] will encode the umlaut character ü silently and transform it somewhat to some combination of dashes and dots and some alphanumeric (see above)

It's actually called punycode.

It would be good to have feedback from some people who I think are used to German, e.g.
@leastprivilege or French @PinpointTownes

That's a rather strange bug, indeed (and not something I've seen before, tho' I must admit I don't use accents in my email addresses, which is often not allowed by major email providers anyway).

Are you sure the username/email are not punycoded client-side by the browser before being sent to the server? It may explain why the validation check doesn't reject the punycoded username/email if they are received already encoded.

[Ponant]

@PinpointTownes I will do more checks later as it is late here; all I can say for now is that I will receive punycoded data only if I have these EmailAddress attributes. This holds with or without validation js scripts. Otherwise the data will go with the umlaut to the server and UserValidation will reject the username (as expected). Does this tell you something?
Edit:
Case when js validation is removed, while Chrome and Firefox accept the umlaut, Edge refuses to post.

[Ponant]

I think this should be moved to corefx as it does not seem to be related to Identity. I will update this thread once the bug is isolated.
https://github.com/dotnet/corefx/issues/36675

[kevinchalet]

I will do more checks later as it is late here; all I can say for now is that I will receive punycoded data only if I have these EmailAddress attributes. Does this tell you something?

These attributes affect the way email input boxes are represented in HTML5 (i.e as type=email inputs rather than type=text).

A quick search shows that it is indeed a browser-side conversion, that automatically happens for email fields: https://bugs.chromium.org/p/chromium/issues/detail?id=410937

It's also interesting to note that the W3C HTML WG updated the spec to allow non-ASCII chars in email inputs. I wouldn't be surprised if the behavior you're seeing was changed in a future version of mainstream browsers.

[Ponant]

@PinpointTownes , then why I do not have this problem with a fresh non core asp.net identity but still on chrome?

[Ponant]

@PinpointTownes , ok I just found out that the old .Net framework outputs type=text while .Net Core type=email. I also found that FF does send the data correctly to the browser, so it seems to be a chromium issue.

[Ponant]

the bug you link to dates back to 2014, so I still think that should be fixed here to accommodate for chrome since the response from chromium has been a "won't fix".

[kevinchalet]

ok I just found out that the old .Net framework outputs type=text while .Net Core type=email. I also found that FF does send the data correctly to the browser, so it seems to be a chromium issue.

Yep, that's because the MVC 5.x templates use Html.TextBoxFor for the email/username field. If you switch to Html.EditorFor, it will take the EmailAddress attribute into account and you'll see a similar behavior.

the bug you link to dates back to 2014, so I still think that should be fixed here to accommodate for chrome since the response from chromium has been a "won't fix".

At best, Identity's UI could try to proactively "punydecode" the username/email fields back to their real Unicode representation and validate them after that. Whether it's a good idea or something rather fragile is what @blowdart will have to determine 😄