Add optional detailed token counts

[awsr]

• Add option to also show token counts split up by segment markers (BREAK and, if enabled, line break).
  • Detailed display format is `[#, #, #] {sum}/{max_length}"
• Minor changes:
  • Change "tokenizer busy" state display to "--/--"
  • Add error state display "??/{max_length}"

PR created as draft to await feedback regarding other tokenizers (I don't have any models set up that use different tokenizers, or available free space at the moment).

[vladmandic]

i don't think we need this as tunable. if it works, it should be on and thats it.

[vladmandic]

this assumes that tokenizer works with list as input, that's the only thing i'm not sure about for all different tokenizers, but ok for now and if needed, it can be changed later.

[vladmandic]

this should be guarded with warn_once so its not flooding the log?
(similar to how preview reports first error but then stays silent)

[vladmandic]

minor nitpicks in comments.
big one is that entire concept with BREAK only works on some models and only if enhanced prompt parsing is enabled, so its a mistake to present broken down token count if that is not the case.

see check in

sdnext/modules/processing_prompt.py

Lines 85 to 90 in c9e21a5

	if (prompt_attention != 'fixed') and ('Onnx' not in cls) and ('prompt' not in p.task_args) and (
	('StableDiffusion' in cls) or
	('StableCascade' in cls) or
	('Flux' in cls and 'Flux2' not in cls) or
	('Chroma' in cls) or
	('HiDreamImagePipeline' in cls)

[awsr]

Yeah, it's a little confusing to untangle which instances are valid for BREAK because processing_prompt.py makes calls to prompt_parser_diffusers.py which also makes calls to prompt_parser.py. For example, parse_prompt_attention in prompt_parser.py includes a check to replace instances of \n with BREAK based on the setting. I've got no idea if that's because other parsers automatically do that or if BREAK doesn't work with the other parsers. I'm still looking into it.

[vladmandic]

any case where prompt parser is not native (so either fixed, a1111 or compel), do not trigger your logic. lets not go into total border cases. sdnext has native and native is what matters.

what i highlighted is that BREAK only has meaning for some MODEL TYPES, not just if parser is set to native or not.

[awsr]

Reverted using only manual looping because I found the definition of valid inputs according to tokenizers (and, by extension, transformers):

        def _is_valid_text_input(t):
            if isinstance(t, str):
                # Strings are fine
                return True
            elif isinstance(t, (list, tuple)):
                # List are fine as long as they are...
                if len(t) == 0:
                    # ... empty
                    return True
                elif isinstance(t[0], str):
                    # ... list of strings
                    return True
                elif isinstance(t[0], (list, tuple)):
                    # ... list with an empty list or with a list of strings
                    return len(t[0]) == 0 or isinstance(t[0][0], str)
                else:
                    return False
            else:
                return False