Tags: agentstation/tokenizer
Tags
Release v0.0.6 Major Changes: - Migrate from deprecated Homebrew formulas to casks - Auto-generate and install shell completions Features: - Shell completions automatically installed via Homebrew - Completions properly uninstall with brew uninstall - No manual setup required for completions - Docker image support with multi-arch builds - SBOM (Software Bill of Materials) generation Improvements: - Follow GoReleaser best practices for 2024+ - Smoother Homebrew installation experience - Pre-built binaries properly handled by casks - Enhanced build reproducibility
Release v0.0.5 Features: - Add shell completion support (bash, zsh, fish, powershell) - Add Docker image support with multi-arch builds - Add SBOM (Software Bill of Materials) generation - Enhanced build reproducibility with -trimpath flag - Add mod_timestamp for consistent builds - Include checksums in release notes footer Improvements: - Better version tracking with dynamic Go version - Enhanced release notes with Docker instructions - Optimized build flags for security and reproducibility Fixes: - Configure Docker buildx for multi-arch support - Install syft in CI for SBOM generation
Release v0.0.4 - Fix empty input handling in scanner - Add builtBy build variable for tracking build source - Fix GoReleaser GO_VERSION template issue - Update Homebrew formula to use template variables - Add comprehensive version tracking - Performance optimizations maintained
Release v0.0.3 - Improved CLI UX Features: - Intuitive default behaviors: - `tokenizer llama3 "text"` encodes by default - `cat file | tokenizer llama3` streams automatically - `tokenizer llama3` shows help - Support for `tokenizer llama3 help` command - Cleaner release naming (v0.0.3 instead of tokenizer-v0.0.3) Improvements: - Updated all documentation to reflect simplified usage - Added markdown table for download options - Better help text when no arguments provided The CLI is now more intuitive while maintaining full backwards compatibility.
Release v0.0.2 Features: - Automatic Homebrew formula updates via GoReleaser - Homebrew installation now available: brew install agentstation/tap/tokenizer Changes: - Fix GoReleaser v2 configuration for Homebrew - Update README with Homebrew installation instructions - Enable automated formula updates on each release This release establishes automated distribution through Homebrew.
Initial release of tokenizer Features: - Llama 3 tokenizer implementation with exact compatibility - CLI tool with encode, decode, stream, and info commands - Streaming API for large file processing - High-performance BPE implementation with caching - Comprehensive test suite with 100+ test vectors - Multi-platform binary releases via GoReleaser - Full development tooling (Makefile, pre-commit hooks, CI/CD) - Extensive documentation with gomarkdoc - Homebrew formula available Supported platforms: - Linux (amd64, arm64, arm) - macOS (amd64, arm64) - Windows (amd64) - FreeBSD (amd64, arm64) This release provides a solid foundation for tokenization tasks with a focus on performance and correctness.