Support reading tiktoken tokenizer.model file #31656

itazap · 2024-06-27T12:52:43Z

Use existing TikTokenConverter to convert tiktoken tokenizer.model file.
depends on loading without config.json file #32356

add case to convert_tiktoken_tokenizer
add internal model
add test

Workflow changes

tokenization_utils_base.py': when loading a model, the slow tokenizer is loaded first. If the tokenizer.model file is not SPM, then an error of type google.protobuf.message.DecodeError is thrown, or a RunTime error on loading ModelProto. So, the first step is to catch these errors relating to SPM and set the tokenizer=False to indicate failure.
tokenization_utils_fast.py: check if slow_tokenizer=False, if so, try to convert from tiktoken.
convert_slow_tokenizer.py: use TikTokenConverter to convert.

Note: the reason we catch errors is because there is no way to differentiate the tokenizer.model file as SPM or TikToken with the current standards for hub files. So, we always try to convert from SPM, if we fail, we try with TikToken.

@ArthurZucker

HuggingFaceDocBuilderDev · 2024-06-27T13:31:23Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

src/transformers/tokenization_utils_fast.py

ArthurZucker

Very nice! A nit about special tokens mostly

src/transformers/convert_slow_tokenizer.py

src/transformers/tokenization_utils_fast.py

tests/models/llama/test_tokenization_llama.py

src/transformers/convert_slow_tokenizer.py

src/transformers/tokenization_utils_base.py

src/transformers/tokenization_utils_fast.py

tests/models/llama/test_tokenization_llama.py

src/transformers/tokenization_utils_fast.py

ArthurZucker

a few nits but I like that it's easy to load a tiktoken based model with PreTrainedTokenizerFast! Good work

src/transformers/convert_slow_tokenizer.py

src/transformers/testing_utils.py

src/transformers/tokenization_utils_base.py

src/transformers/tokenization_utils_fast.py

ArthurZucker

Great addition, we have one "last" decision to make and good to go!

src/transformers/convert_slow_tokenizer.py

src/transformers/utils/import_utils.py

tests/models/llama/test_tokenization_llama.py

ArthurZucker

LGTM I'll let you handle the tests(CI) and pushing the new docker!

docs/source/en/tiktoken.md

src/transformers/tokenization_utils_base.py

ArthurZucker

Thanks 🤗 a few nits but LGTM otherwise

setup.py

src/transformers/convert_slow_tokenizer.py

ArthurZucker

Thanks LGTM! 🤗

docker/consistency.dockerfile

* use existing TikTokenConverter to read tiktoken tokenizer.model file * del test file * create titktoken integration file * adding tiktoken llama test * ALTNATIVE IMPLEMENTATION: supports llama 405B * fix one char * remove redundant line * small fix * rm unused import * flag for converting from tiktokeng * remove unneeded file * ruff * remove llamatiktokenconverter, stick to general converter * tiktoken support v2 * update test * remove stale changes * udpate doc * protect import * use is_protobuf_available * add templateprocessor in tiktokenconverter * reverting templateprocessor from tiktoken support * update test * add require_tiktoken * dev-ci * trigger build * trigger build again * dev-ci * [build-ci-image] tiktoken * dev-ci * dev-ci * dev-ci * dev-ci * change tiktoken file name * feedback review * feedback rev * applying feedback, removing tiktoken converters * conform test * adding docs for review * add doc file for review * add doc file for review * add doc file for review * support loading model without config.json file * Revert "support loading model without config.json file" This reverts commit 2753602. * remove dev var * updating docs * safely import protobuf * fix protobuf import error * fix protobuf import error * trying isort to fix ruff error * fix ruff error * try to fix ruff again * try to fix ruff again * try to fix ruff again * doc table of contents * add fix for consistency.dockerfile torchaudio * ruff * applying feedback * minor typo * merging with push-ci-image * clean up imports * revert dockerfile consistency

itazap force-pushed the tiktoken_file_support branch 2 times, most recently from cb46268 to 633cf73 Compare July 1, 2024 09:07

itazap commented Jul 11, 2024

View reviewed changes

src/transformers/tokenization_utils_fast.py Outdated Show resolved Hide resolved

itazap requested review from ArthurZucker, amyeroberts and LysandreJik July 11, 2024 12:06

itazap marked this pull request as ready for review July 11, 2024 12:06

huggingface locked and limited conversation to collaborators Jul 16, 2024

huggingface unlocked this conversation Jul 16, 2024

ArthurZucker reviewed Jul 16, 2024

View reviewed changes

itazap force-pushed the tiktoken_file_support branch from b542797 to aed803b Compare July 22, 2024 10:18

itazap requested a review from ArthurZucker July 22, 2024 10:35

itazap force-pushed the tiktoken_file_support branch from 9d252fb to 716ed8d Compare July 22, 2024 12:13

ArthurZucker reviewed Jul 22, 2024

View reviewed changes

itazap requested a review from ArthurZucker July 22, 2024 17:11

ArthurZucker reviewed Jul 23, 2024

View reviewed changes

itazap requested a review from ArthurZucker July 23, 2024 14:23

ArthurZucker reviewed Jul 30, 2024

View reviewed changes

itazap requested a review from ArthurZucker July 31, 2024 09:47

ArthurZucker approved these changes Aug 1, 2024

View reviewed changes

itazap force-pushed the tiktoken_file_support branch 2 times, most recently from 00f6995 to 49de881 Compare August 7, 2024 10:03

itazap added 7 commits August 22, 2024 15:10

use existing TikTokenConverter to read tiktoken tokenizer.model file

ce21f32

del test file

744ab3d

create titktoken integration file

ef5df3f

adding tiktoken llama test

341db42

ALTNATIVE IMPLEMENTATION: supports llama 405B

cc11541

fix one char

418d3d7

remove redundant line

66f38a7

itazap added 11 commits August 22, 2024 15:11

updating docs

c08c80f

safely import protobuf

55ee7e5

fix protobuf import error

3ff09bf

fix protobuf import error

815f1b6

trying isort to fix ruff error

6b568c8

fix ruff error

d0ca96f

try to fix ruff again

7b34bed

try to fix ruff again

4be52ba

try to fix ruff again

d647025

doc table of contents

a8de4ca

add fix for consistency.dockerfile torchaudio

747f6f1

itazap force-pushed the tiktoken_file_support branch from b851bb4 to 747f6f1 Compare August 22, 2024 13:11

itazap requested a review from ArthurZucker August 22, 2024 13:12

itazap and others added 2 commits August 26, 2024 13:42

Merge branch 'main' into tiktoken_file_support

5ede660

ruff

eab392d

ArthurZucker approved these changes Aug 28, 2024

View reviewed changes

itazap added 3 commits August 28, 2024 20:25

applying feedback

97b1e23

minor typo

1cdd8ae

merging with push-ci-image

1c9ee23

itazap mentioned this pull request Aug 28, 2024

support loading model without config.json file #32356

Merged

itazap requested a review from ArthurZucker August 28, 2024 20:53

clean up imports

bf3e795

ArthurZucker approved these changes Aug 29, 2024

View reviewed changes

docker/consistency.dockerfile Outdated Show resolved Hide resolved

revert dockerfile consistency

08dde7a

itazap merged commit e48e5f1 into main Sep 6, 2024
26 checks passed

itazap deleted the tiktoken_file_support branch September 6, 2024 12:24

itazap mentioned this pull request Sep 10, 2024

use diff internal model in tests #33387

Merged

pcuenca mentioned this pull request Sep 17, 2024

Fix Llama 3 TikToken conversion #33538

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support reading tiktoken tokenizer.model file #31656

Support reading tiktoken tokenizer.model file #31656

itazap commented Jun 27, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Jun 27, 2024

ArthurZucker left a comment

ArthurZucker left a comment •

edited

Loading

ArthurZucker left a comment

ArthurZucker left a comment

ArthurZucker left a comment

ArthurZucker left a comment

Support reading tiktoken tokenizer.model file #31656

Support reading tiktoken tokenizer.model file #31656

Conversation

itazap commented Jun 27, 2024 • edited Loading

Workflow changes

HuggingFaceDocBuilderDev commented Jun 27, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker left a comment • edited Loading

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker left a comment

Choose a reason for hiding this comment

itazap commented Jun 27, 2024 •

edited

Loading

ArthurZucker left a comment •

edited

Loading