Search results

Add GPT 4.5 to Tiktoken Tokenizer

enhancementNew feature or requestNew feature or request

Tokenizers

Status: Open.

#7404 In dotnet/machinelearning;

· luisquintanilla opened on Feb 27, 2025

 ·  

ML.NET 5.0

Add max number of token to count to the TokenCount function

enhancementNew feature or requestNew feature or request

in-pr

Tokenizers

Status: Open.

#7391 In dotnet/machinelearning;

· pgroene opened on Feb 19, 2025

 ·  

ML.NET Future

.NET AI Fundamentals Feedback / Feature Requests

enhancementNew feature or requestNew feature or request

onnxExporting ONNX models or loading ONNX modelsExporting ONNX models or loading ONNX models

Tokenizers

untriagedNew issue has not been triagedNew issue has not been triaged

Status: Open.

#7383 In dotnet/machinelearning;

· luisquintanilla opened on Feb 11, 2025

Context Aware Tokenization

enhancementNew feature or requestNew feature or request

Tokenizers

Status: Open.

#7372 In dotnet/machinelearning;

· tjwald opened on Jan 23, 2025

 ·  

ML.NET Future

Batch Tokenization Support

enhancementNew feature or requestNew feature or request

Tokenizers

Status: Open.

#7371 In dotnet/machinelearning;

· tjwald opened on Jan 23, 2025

 ·  

ML.NET 5.0

Whisper Tokenizer support

enhancementNew feature or requestNew feature or request

Tokenizers

Status: Open.

#7353 In dotnet/machinelearning;

· MithrilMan opened on Dec 23, 2024

 ·  

ML.NET Future

Consider Rust BPE tokenizer implementation

Tokenizers

Status: Open.

#7336 In dotnet/machinelearning;

· tarekgh opened on Dec 14, 2024

 ·  

ML.NET Future

Expose Encoder in TiktokenTokenizer

enhancementNew feature or requestNew feature or request

in-pr

needs-further-triage

Tokenizers

Status: Open.

#7313 In dotnet/machinelearning;

· razshare opened on Nov 15, 2024

 ·  

ML.NET Future

[Tracking] Clean up item related to Tokenizers

Tokenizers

Status: Open.

#7268 In dotnet/machinelearning;

· tarekgh opened on Oct 10, 2024

 ·  

ML.NET Future

Update Microsoft.ML.Tokenizers - o1 models

enhancementNew feature or requestNew feature or request

Tokenizers

Status: Open.

#7241 In dotnet/machinelearning;

· luisquintanilla opened on Sep 19, 2024

 ·  

ML.NET 5.0

[Tokenizers] How to load HF based tokenizers e.g. SmolLM

documentationRelated to documentation of ML.NETRelated to documentation of ML.NET

enhancementNew feature or requestNew feature or request

questionFurther information is requestedFurther information is requested

Tokenizers

Status: Open.

#7197 In dotnet/machinelearning;

· nietras opened on Jul 22, 2024

 ·  

ML.NET 5.0

NER Remaining Work

Deep Learning

documentationRelated to documentation of ML.NETRelated to documentation of ML.NET

Tokenizers

Status: Open.

#6910 In dotnet/machinelearning;

· luisquintanilla opened on Dec 15, 2023