🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code.
-
Updated
Apr 29, 2025 - Python
#
sentencepiece
Here are 29 public repositories matching this topic...
使用sentencepiece中BPE训练中文词表，并在transformers中进行使用。
-
Updated
Jun 24, 2023 - Python
A Robustly Optimized BERT Pretraining Approach for Vietnamese
natural-language-processing vietnamese pytorch transformer pretrained-models bert vietnamese-nlp fairseq roberta bert-embeddings sentencepiece
-
Updated
Jul 25, 2024 - Python
Extremely simple and understandable GPT2 implementation with minor tweaks
-
Updated
Dec 6, 2019 - Python
Learning BPE embeddings by first learning a segmentation model and then training word2vec
-
Updated
Dec 18, 2022 - Python
BERT implementation of PyTorch
-
Updated
Mar 16, 2020 - Python
A Gradio-based demo application for comparing state-of-the-art OCR models: DeepSeek-OCR, Dots.OCR, HunyuanOCR, and Nanonets-OCR2-3B.
python ocr pillow torch accelerate supervision gradio opencv-python nanonets torchvision sentencepiece huggingface-transformers huggingface-spaces flash-attention-2 hunyuan qwen2-5-vl dots-ocr deepseek-ocr easydict
-
Updated
Nov 28, 2025 - Python
Bengali language Tokenizer (SentencePiece)
tokenizer bengali unsupervised-learning sentencepiece bengali-natural-language-processing bengali-tokenizer
-
Updated
Oct 20, 2019 - Python
dataset, train, inference
-
Updated
May 19, 2024 - Python
A Gradio-based demonstration application for the Tencent HunyuanOCR model, focused on optical character recognition (OCR) tasks such as text detection, extraction, and coordinate formatting from images. Users can upload images, customize prompts (e.g., for Chinese/English text).
python torch accelerate gradio kernels opencv-python peft torchvision sentencepiece huggingface-transformers huggingface-spaces huggingface-hub vision-language-model easydict hunyuanocr
-
Updated
Dec 4, 2025 - Python
NMT with RNN Models: (1) in Vanilla style, (2) with Sentencepiece, (3) using Pre-trained models from FairSeq
-
Updated
Sep 19, 2021 - Python
A Gradio-based demonstration for the Tongyi-MAI/Z-Image-Turbo diffusion pipeline, enhanced with a curated collection of LoRAs (Low-Rank Adaptations) for style transfer and creative image generation. Users can select from pre-listed LoRAs or add custom ones from Hugging Face repositories.
python numpy cuda torch image-generation lora gradio kernels text-to-image peft torchvision pilllow diffusion-models sentencepiece diffuser huggingface-transformers huggingface-spaces huggingface-diffusers z-image-turbo
-
Updated
Dec 24, 2025 - Python
Kontext-Photo-Mate-v2 is an advanced image manipulation application built on top of the FLUX.1 Kontext AI model, featuring multiple LoRA adapters for fine-grained perspective and object editing.
flux photoshop camera-calibration transformers pillow torch nvidia accelerate image-manipulation gradio peft photoeditor kontext sentencepiece trl huggingface-transformers gradio-interface huggingface-spaces
-
Updated
Dec 31, 2025 - Python
-
Updated
Jan 25, 2025 - Python
Escape unknown symbols in SentecePiece vocabularies
-
Updated
Feb 12, 2025 - Python
A framework for building Sentencepiece tokenizer from a dataset
machine-learning natural-language-processing deep-learning tensorflow tokenizer keras transformer unigram bpe sentencepiece large-language-models llm pldr-llm
-
Updated
Oct 23, 2024 - Python
Neural sentence boundary detection for Meitei Mayek (Manipuri) using SentencePiece tokenization and a CNN-based spaCy pipeline.
nlp tokenizer spacy sentence-segmentation manipuri low-resource-languages fastapi sentencepiece meitei-mayek
-
Updated
Dec 19, 2025 - Python
Automated WikiGame-playing 'bot'. Achieved via SentenceTransformer Word Embeddings.
-
Updated
Nov 12, 2025 - Python
A TensorFlow-based Transformer model for English-to-Russian neural machine translation. Features subword tokenization with SentencePiece, hyperparameter optimization via genetic algorithm, and a Flask web interface for real-time translations.
flask machine-learning natural-language-processing deep-learning tensorflow genetic-algorithm transformer neural-machine-translation sentencepiece english-to-russian
-
Updated
Sep 6, 2025 - Python
pretrained models and a training code for sentencepiece
-
Updated
Jul 27, 2023 - Python
Improve this page
Add a description, image, and links to the sentencepiece topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the sentencepiece topic, visit your repo's landing page and select "manage topics."