Skip to content

Fix separators for word tokenizer #935

Closed
@Zruty0

Description

@Zruty0

Currently, the separators property for WordTokenizer is a string, that is parsed as a comma-separated list of characters. It was appropriate for command-line parsing, but not appropriate for API use.

A better solution is to have a char[].

Metadata

Metadata

Assignees

Labels

good first issueGood for newcomersup-for-grabsA good issue to fix if you are trying to contribute to the project

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions