Add get methods for HuggingFaceTokenizer fields

[ylwu-amzn]

Description

We need to get the tokenizer config like model maxLength. But now the HuggingFaceTokenizer class doesn't have get method for this. Suggest add get methods for HuggingFaceTokenizer fields.

One minor suggestion, return builder for configure method

      public void configure(Map<String, ?> arguments) {
            for (Map.Entry<String, ?> entry : arguments.entrySet()) {
                options.put(entry.getKey(), entry.getValue().toString());
            }
        }

change to

      public Builder configure(Map<String, ?> arguments) {
            for (Map.Entry<String, ?> entry : arguments.entrySet()) {
                options.put(entry.getKey(), entry.getValue().toString());
            }
            return this;
        }

References

• list reference and related literature
• list known implementations

[frankfliu]

Can you more more context about this request? In which use case you need to get the maxLength of a tokenizer?

[ylwu-amzn]

We want to do some checking if user's input exceeds max length or not, if yes, we will throw our own exception with readable error message.

[frankfliu]

How can you check the maxLength before tokenize?

[ylwu-amzn]

We have our own translator, https://github.com/opensearch-project/ml-commons/blob/main/ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/SentenceTransformerTranslator.java#L28

We want to add some checking in processInput, if the input text has more tokens than max length, we will throw exception

[frankfliu]

The number of tokens is purely depends on the tokenizer, sometimes the single word will have 2 tokens. the word count is not the same as token count. My questions is how can you know if an input text exceed the max token length.

The right way is to add a function: Encoding.hasOverflowTokens(), you can check if if the output tokens exceed the maxLength. But this approach has performance hit.

[frankfliu]

I created a PR to return if the output tokens exceed the max length: #2957

Please take a look and see if this resolve your issue.

[ylwu-amzn]

Cool, thanks @frankfliu

[frankfliu]

I added this PR: #2958

But be ware that in most cases getMaxLength() will return -1, which means if it truncation kicks in, it fallback to use maxModelLength. While maxModelLength is set by your code (or default to 256).

The right way of using maxLength is you manually set it (that's why we don't have a getter to begin with)

[ylwu-amzn]

I added this PR: #2958

But be ware that in most cases getMaxLength() will return -1, which means if it truncation kicks in, it fallback to use maxModelLength. While maxModelLength is set by your code (or default to 256).

The right way of using maxLength is you manually set it (that's why we don't have a getter to begin with)

Got it, make sense, I see huggingface tokeizer support options, we will try that