The user directory doesn't seem to be tokenizing user IDs entirely correctly.

[anoadragon453]

Description

Looking at the contents of my homeserver's user_directory_search table, I'm seeing the tokenization of some user ID parts to be incomplete:

synapse=# select * from user_directory_search limit 5;
            user_id            |                   vector                    
-------------------------------+---------------------------------------------
 @negritoofcourse:matrix.org   | 'matrix.org':2 'negritoofcours':1A,3B
 @freenode_jin1200:matrix.org  | 'freenod':1A 'jin1200':2A,4B 'matrix.org':3
 @freenode_Xenguy__:matrix.org | 'freenod':1A 'matrix.org':3 'xenguy':2A,4B
 @morethanabitoff:matrix.org   | 'matrix.org':2 'morethanabitoff':1A,3B
 @curtisthe:matrix.org         | 'curtisth':1A,3B 'matrix.org':2
(5 rows)

For instance, we see @curtisthe:matrix.org has curtisth without the e as a token, same with @negritoofcourse:matrix.org and the various freenods.

It looks like there might be an off-by-one error somewhere in the user_directory_background_update process.

Doesn't seem to be a huge impact as people haven't reported unusable results, but probably a good thing to fix.

Synapse v1.14.0.