API: token descriptors considered harmful

[ruv]

When creating a text translator that generates intermediate code, or code in some not native format (but still using the native FVM), or code for an independent target VM, we must define at least the following parts:

1. The token compilers (like lit,, compile,, compilers for control-flow, etc).
2. The token translators (it's enough to load the definitions for token translators against the new token compilers).
3. The recognizers (load their definitions against the token compilers and token translators).
4. The higher-level words that generate code (load their definitions against the token compilers and token translators):
   • the words related to control-flow (like if, else, then, begin, again, etc);
   • syntax constructs like ['], [char], s", c", to, is, action-of, postpone;
5. The words related to the return stack (like >r, r>, r@, 2>r, 2r>, 2r@, rdrop, 2rdrop, etc).
6. Local variables (if any).

If the Recognizer API automates the postpone action via a reproducer action, it prevents the item (2) to be implementable at all, because it uses the system's compile,, see:

• API: Against a reproducer in a token descriptor for postpone #19

If the Recognizer API dictates that only a special system-specific method shall be used to define a token translator from two or three actions, then the Forth text interpreter itself depends on STATE, because it uses STATE to select a one or another action. This means that your special text translator still depends on STATE too, and you have to provide actions that you don't actually needs. See also:

• API: interpretation and compilation semantics are enough for user-defined descriptors #16

With token translators (instead of token descriptors), the Forth text interpreter by itself is free of STATE.
See also:

• Diagram for illustrating an unnecessary coupling between the Forth text interpreter loop and the Recognizer API, (2022-09-09).
• Comment re unnecessary coupling (2022-09-08).
• Comment re STATE in the outer loop (2024-12-04).