Fixing the text part from tokenizer endpoint.

[Narsil]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[drbh]

Simple demo using bytes vs chars for intuition. My understanding is that we may need to slice intra-character, and .chars() does not allow this. This PR resolves this

fn main() {
    let input = "こんにちは世界"; // "Hello World" in Japanese
    let start = 2;
    let stop = 5;

    // Method 1: Using bytes
    let text_bytes: String = String::from_utf8_lossy(&input.as_bytes()[start..stop]).to_string();

    // Method 2: Using chars
    let text_chars: String = input.chars().skip(start).take(stop - start).collect();

    println!("Original: {}", input);
    println!("Using bytes: {}", text_bytes);
    println!("Using chars: {}", text_chars);
    
    // Original: こんにちは世界
    // Using bytes: ��
    // Using chars: にちは
}

[drbh]

lgtm