Open
Description
For browsermt/bergamot-translator#202, it was suggested that a view for a vector/range backed by a binary representaion be substituted to eliminate conversions between vector and binary representation.
However, decodeWithByteRanges
assumes a vector
to be provided:
marian-dev/src/data/sentencepiece_vocab.cpp
Lines 259 to 281 in 62bac85
We need to provide something that can work with ranges/iterators instead to avoid the additional copy. Since this is a function only used by bergamot we need not worry about breaking any backwards compatibility. Consistency with the remaining may still be a concern, in which case we can provide an overload which internally calls the range-based method.