Skip to content

Add option case_sensitive to featurizers #5759

Closed
@tabergma

Description

@tabergma

Description of Problem:
Currently, some of our tokenizers support the option case_sensitive. If the user decides to set this option to False all featurizer will use the lowercased tokens. This might not be ideal. For example, some of the features in the LexicalSyntacticFeaturizer do not work if all tokens are lowercased.
It might be better to move the option case_sensitive to the featurizers itself. Each featurizer can be configured separately.

Overview of the Solution:
Remove the option case_sensitive from all tokenizers.
Add the option case_sensitive to featurizers for which it makes sense.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:rasa-oss 🎡Anything related to the open source Rasa frameworktype:enhancement ✨Additions of new features or changes to existing ones, should be doable in a single PR

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions