Tokenizer 1.18.4
Fixes and improvements
- Fix segmentation fault on
cli/tokenize
exit - Ignore empty tokens during detokenization
- When writing to a file, avoid flushing the output stream on each line
- Update
cli/CMakeLists.txt
to mark Boost.ProgramOptions as required
(This is the first release to be created on GitHub. See the release note of previous tags in CHANGELOG.md.)