Skip to content

Tags: agentstation/tokenizer

Tags

v0.0.8

Toggle v0.0.8's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.8

v0.0.7

Toggle v0.0.7's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.7

Fixes:
- Fix Ruby syntax error in Homebrew cask caveats
- Use single quotes to avoid quote escaping issues

v0.0.6

Toggle v0.0.6's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.6

Major Changes:
- Migrate from deprecated Homebrew formulas to casks
- Auto-generate and install shell completions

Features:
- Shell completions automatically installed via Homebrew
- Completions properly uninstall with brew uninstall
- No manual setup required for completions
- Docker image support with multi-arch builds
- SBOM (Software Bill of Materials) generation

Improvements:
- Follow GoReleaser best practices for 2024+
- Smoother Homebrew installation experience
- Pre-built binaries properly handled by casks
- Enhanced build reproducibility

v0.0.5

Toggle v0.0.5's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.5

Features:
- Add shell completion support (bash, zsh, fish, powershell)
- Add Docker image support with multi-arch builds
- Add SBOM (Software Bill of Materials) generation
- Enhanced build reproducibility with -trimpath flag
- Add mod_timestamp for consistent builds
- Include checksums in release notes footer

Improvements:
- Better version tracking with dynamic Go version
- Enhanced release notes with Docker instructions
- Optimized build flags for security and reproducibility

Fixes:
- Configure Docker buildx for multi-arch support
- Install syft in CI for SBOM generation

v0.0.4

Toggle v0.0.4's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.4

- Fix empty input handling in scanner
- Add builtBy build variable for tracking build source
- Fix GoReleaser GO_VERSION template issue
- Update Homebrew formula to use template variables
- Add comprehensive version tracking
- Performance optimizations maintained

v0.0.3

Toggle v0.0.3's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.3 - Improved CLI UX

Features:
- Intuitive default behaviors:
  - `tokenizer llama3 "text"` encodes by default
  - `cat file | tokenizer llama3` streams automatically
  - `tokenizer llama3` shows help
- Support for `tokenizer llama3 help` command
- Cleaner release naming (v0.0.3 instead of tokenizer-v0.0.3)

Improvements:
- Updated all documentation to reflect simplified usage
- Added markdown table for download options
- Better help text when no arguments provided

The CLI is now more intuitive while maintaining full backwards compatibility.

v0.0.2

Toggle v0.0.2's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Release v0.0.2

Features:
- Automatic Homebrew formula updates via GoReleaser
- Homebrew installation now available: brew install agentstation/tap/tokenizer

Changes:
- Fix GoReleaser v2 configuration for Homebrew
- Update README with Homebrew installation instructions
- Enable automated formula updates on each release

This release establishes automated distribution through Homebrew.

v0.0.1

Toggle v0.0.1's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
Initial release of tokenizer

Features:
- Llama 3 tokenizer implementation with exact compatibility
- CLI tool with encode, decode, stream, and info commands
- Streaming API for large file processing
- High-performance BPE implementation with caching
- Comprehensive test suite with 100+ test vectors
- Multi-platform binary releases via GoReleaser
- Full development tooling (Makefile, pre-commit hooks, CI/CD)
- Extensive documentation with gomarkdoc
- Homebrew formula available

Supported platforms:
- Linux (amd64, arm64, arm)
- macOS (amd64, arm64)
- Windows (amd64)
- FreeBSD (amd64, arm64)

This release provides a solid foundation for tokenization tasks with a focus on performance and correctness.