Skip to content

Commit 57eb8d7

Browse files
authored
Add ByteLevel normalizer to documentation
1 parent da1cc3b commit 57eb8d7

File tree

1 file changed

+8
-3
lines changed

1 file changed

+8
-3
lines changed

docs/source-doc-builder/api/normalizers.mdx

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
<tokenizerslangcontent>
44
<python>
5-
## BertNormalizer
5+
## ByteLevel
66

7-
[[autodoc]] tokenizers.normalizers.BertNormalizer
7+
[[autodoc]] tokenizers.normalizers.ByteLevel
88

99
## Lowercase
1010

@@ -53,11 +53,16 @@
5353
## StripAccents
5454

5555
[[autodoc]] tokenizers.normalizers.StripAccents
56+
57+
## BertNormalizer
58+
59+
[[autodoc]] tokenizers.normalizers.BertNormalizer
60+
5661
</python>
5762
<rust>
5863
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.
5964
</rust>
6065
<node>
6166
The node API has not been documented yet.
6267
</node>
63-
</tokenizerslangcontent>
68+
</tokenizerslangcontent>

0 commit comments

Comments
 (0)