[Feature] Raw text support for TokenizingTextBox

[huynhsontung]

Describe the problem this feature would solve

Currently, TokenizingTextBox only accepts tokens. It can be expanded to support raw text as well. With this addition, TokenizingTextBox can be used as chat box or comment box where mention feature is available.

Describe solution

The control will accept raw text like a regular TextBox. When a specified token opening signifier is matched (e.g. @ for mention), it will start suggesting token. Once the control starts suggesting, it only stops when:

1. User selects a suggested token.
2. No matching token is found. Adding more text won't produce a match. Current query is considered raw text.
3. Another opening signifier is detected. Previous query is considered raw text. Start suggesting for the current query.

The control should also provide suggestions when the user go back and edit raw text. When editing, only the string between the signifier and the caret, not including whitespace, is queried.
For example, with | as caret position, Hello Miles Luna how are you? when change to:

• Hello @|Miles Luna how are you? it will start suggesting from empty query.
• Hello @Miles| Luna how are you? only Miles should be queried for suggestions. Token selection will replace Miles.
• Hello @Miles L|una how are you? no suggestion as there is whitespace between Miles and L

Feature	Priority
Get and set raw text	Must
Change token opening signifier	Must
Parse raw text to populate tokens	Should
Support multiple signifiers at the same time (e.g. @ for mention, # for tag)	Should
Support editing token's raw text representation (e.g. @Miles Luna can be shortened to @Miles while still being the same token)	Should

Describe alternatives you've considered

• AutoSuggestBox

Expected result

Hello, 'huynhsontung! Thanks for submitting a new feature request. I've automatically added a vote 👍 reaction to help get things started. Other community members can vote to help us prioritize this feature in the future!

[Kyaa-dost]

@huynhsontung Thanks for the detailed description. Is this something you can work on if approved by the team?

Looping in @marcpems and the team as well for discussion.

[huynhsontung]

@Kyaa-dost Of course. I would love to take a stab at it as I do need this control for my app. I have not looked at the source code for TokenizingTextBox so I can't really tell what to expect.

[michael-hawker]

@huynhsontung a lot of the base control code should already support this, as we support having unparsed tokens as text that can be manipulated. This just wasn't one of the core scenarios for this type of usage, so we didn't test or expand making it easier to use in this way.

It's probably a matter of improving a bit of the handling around the token separation and processing of text loaded (as we do some of that today for copy/paste). I don't think it'd be much work for you to add into the control in a PR.

I.e. your top three requirements should already be supported by the control to a degree. The bottom two about mixing types of tags or having 'shorter' tags would be new features to figure out how to add.

Let us know if you need any assistance in understanding the current code base, thanks!

[huynhsontung]

After some considerations, since I wanted to have this control to work as a message text box, I would also need to add the following features:

• Multi-line support
• Spell check support
• Option to disable the remove token button (as tokens are not the main focus of this control anymore)
• Option to disable query button

Especially multi-line support would be tricky due to the token height is much greater than raw text. One way around this is to make the token control just big enough to wrap the raw text (without pushing other elements around it) when TokenItemTemplate is not set and the remove token button is disabled. But at this point, I wonder if this is too much change to the current TokenizingTextBox? Would these requirements warrant its own control?

Also, excuse me for the late update, I only have the chance to work on this recently.

[michael-hawker]

No worries @huynhsontung thanks for the updated info on your thoughts about requirements. I know the current control supports wrapping the tokens around onto multiple lines, but if you had a long line of text and wanted to wrap that and insert a token inline with the wrap, that wouldn't work as well.

All good things to think about. It may be the type of thing where a V2 of the control is needed to support more in-depth scenarios. Not sure if that's something we'd do in the toolkit or look at working with the WinUI team to bring to the platform post-WinUI 3. There's certainly a lot of overlap in what the two scenarios need, but also they work a bit differently.

FYI @marcpems for any of his thoughts on these scenarios.

I'm going to move to 7.1 for now, as I think this needs a lot of work to pull off and we're closing up 7.0.