Improving UTF-8 Support

[tankerkiller125]

The following kinds of bugs regarding UTF-8 are known and require the following kinds of changes/steps:

Problems:

• UTF-8 Usernames are unsupported (utf8-url encoded username url support (simple solve) framework#1949 and not support Arabic letters framework#1990) (fixed via https://github.com/flarum/nicknames)
  • User slugs are currently hard coded to be the username (fixed with Slug Drivers)
• Discussion slugs can not be UTF-8 and transliteration does not work for some languages/cultures (CJK) (Slug transliteration  framework#194) (fixed with slug drivers)
• UTF-8 search is awful (Can't search anything in chinese or japanese framework#2003)
  • In some languages a single character is a single word or even sentences (min search length is now a constant that extensions can override)
• Scrubber date info is hard coded format, will need to localize that.

[askvortsov1]

UTF-8 mentions: Convert current <@{username}> mention to <@U:{UserID}> similar to discord, also allows group mention in the future

I really like this, but my only concern is that it wouldn't be understandable while in the text editor. Perhaps we could save a mapping of usernames <=> ids somewhere, still show the usernames in the text editor, and swap it out on save? Essentially, just cuz we store it that way shouldn't mean that we display it in the text editor that way. Also, ref #254

Unfortunately this one is a limitation of the database engine itself often. The best solution around this is to support different search drivers that extension developers can build/tie into.

I agree, this should resolve itself naturally with #286

And that brings us to slugs. I've given this a bit of thought, and would like to propose the following structure:

We have an extender, with use:

(new Extend\Model)->slugDriver(Flarum\User\User::class, DRIVER_CLASS); // User can be any model here that implements SluggableModelInterface

The driver interface looks like this:

interface SlugDriver {
    public function to (AbstractModel model): string
    public function from (string slug): AbstractModel
}

The sluggable model interface looks like this:

interface SluggableModelInterface {
    toSlug(string driverId = DEFAULT): string
    fromSlug(string slug, string driverId): AbstractModel
}

Alternatively, we could allow registration of sluggable models via an extender, or bring the SluggableModelInterface functionality out into some form of service class.

We would provide a UI in admin for site admins to choose which slug drivers to use for each supported model (if more than one is registered for that model)

In all sluggable models, we would also include a computed slug for the model in the serialization so that we can keep slug generation logic completely to the backend.

For the API, we can use a parameter sent in get requests to single-item endpoints to determine if the model instance should be retrieved via a slug driver, otherwise defaulting to an ID (to keep a bit of compliance with JSON API? This should also resolve flarum/framework#358 and flarum/framework#1356

[dsevillamartin]

@askvortsov1 For this we could also do something like Discord, but that'd require the editor to be WYSIWYG then (at least for some functions). I agree that just showing @datitisev and replacing that in the backend with the <@U:ID> would be easier and better for both users & API requests that create posts with mentions.

[tankerkiller125]

I agree that @username in the editor makes the most sense (for now) but I feel like at some point WYSIWG will be required for some functionaility in order to resolve a comment/issue @luceos brought up about not displaying usernames to general users (because we allow username logins) personally I think that issue would be better resolved by simply disabling username logins altogether though (I can't think of any sites that allow username logins anymore).

Also @askvortsov1 I'm pretty much happy with this proposals for slugs, and it's pretty much what I had in mind.

[tankerkiller125]

I've updated the list based on some community feedback. However I believe that what's now listed is pretty finalized at this point. I plan to start at least some work here in the coming days to resolve some of these issues.

Edit: I'll start work on this once Mithril 2 rewrite is near completion/complete. This will be my focus for next beta.

[tankerkiller125]

Alright, I'm just going to write down my vision of the way that his would work. Looking for feedback on this idea, it mostly addresses the slugging problems.

Overall the basic idea is that each model that needs slug would use a trait or implement something that would let core know that it's a sluggable model. Further there would be a function (e.g. slugging()) that would inform the slugging handler which database column will be used for slugging purposes (so for discussions that would be the title, users the display name). We may also want to allow extension developers to override model default slugging column in some cases. (So the CJK community might want to use the ID rather than the title for example).

Each slug handler would be defined as a driver (like the mail system) and extension developers would be able to write their own driver that the admin would then be able to select on the dashboard. Admins should be able to select a driver on a model by model basis (so they could do something like discussions use php-intl slugging, users use no conversion slugging, etc.).

Once the admins have changed the slugging method they should be able to use a "Update slug" button that would update all stored slugs. However that button should warn them if they have more than say 1K discussions/users and show them the proper command to use in SSH to do it from there. We should also warn admins that there may be SEO impacts if they update the slugs.

Important Note: Slugs would be stored in the database and JS would pull slugs from this. We would need to find a semi-compatible way of pulling data from the JSON-API using the slugs rather than id.

[maicol07]

I would like to leave a suggestion about this feature. At the moment username can only follow strict rules and there are no ways at the moment (other than editing Flarum core) to change them to something more permissive.
Since usernames includes also special characters in other websites, Flarum should be able to manage them in future betas

[tankerkiller125]

@maicol07 We do support special characters in Display Names which in the default version of Flarum isn't used much. If you look on twitter for example the actual username does not support UTF-8 but the display name does. One of the things that we in theory will be fixing as part of this UTF-8 project will be display/username support and other things that need fixing. Also of note your concern is listed in the checklist.

[askvortsov1]

For changing usernames, I believe that's possible as an admin side action, and the fof username change request extension is a great package for enabling user side requests. Tbh I don't think it's needed as a core feature given those 2

[zerosonesfun]

I can't think of any sites that allow username logins anymore

Twitter? Or am I thinking about that wrong? Maybe that’s not technically a username and/or, I’m sure being Twitter they find other ways to harden security? I get trying to keep usernames more private for security. But, I do think some sites, even some big ones, still have username logins and display usernames. You just need strong passwords. Maybe 2 factor eventually.

Anyway... just saw that comment and thought I’d foolishly for whatever reason go to bat for sites that display usernames and allow username logins. 🙃

I’m actually a fan of going with email only logins or whatever is decided. Or, let the user decide how secure they want to be. A switch in the admin to allow or not allow username logins. I guess that could be an extension. I also guess I’m getting a little off topic.