Discrepancy in expansion of undefined command sequences

[dginev]

This issue goes towards further parity with the native tex interpretation, towards the 1.0 milestone of latexml.

Minimal example:

\documentclass{article}
\edef\expandedondef{\tricky~\tricky}
\def\tricky{tricky!}

\begin{document}
\expandedondef
\end{document}

In pdflatex this produces an empty document with an error mentioning \tricky is undefined at time of \edef execution. In latexml, no error is ever displayed, and what's worse - the new "late binding" of \tricky is used in the final output, questioning if \edef ever expanded the macro.

[dginev]

This particular issue should have the same implication for \protected macros, which is what led me here in the first place.

[dginev]

Here is a second example showing how this general mismatch compounds in other cases, such as \expandafter:

\documentclass{article}
\begin{document}
\def\go{\def\idea{idea}}
\def\gotwo{\def\newidea{new idea}}
% error: undefined \idea
\expandafter\go\idea

% all good:
\gotwo\newidea
\end{document}

The PDF output contains only "new idea", while latexml also has the initial "idea" present in the output, as the first \expandafter does not consume the undefined \idea command sequence, but inputs it as if it were legitimate.

One could think this is a tangential problem, as it only plays a role when there is an error in the TeX source. However, it has subtle additional implications - as can be seen with babel tests failing when I attempt a fix. Namely, the lack of explicit error messages for these cases could hide issues that are caused by latexml's incomplete TeX interpretation, making certain kinds of mismatches extremely difficult to debug (as they are silently divergent from pdflatex).

The key passage from the TeX B book is item 367. on page 154, case othercases. My understanding is that all calls to tex's expand, in particular all calls to get_x_token will both 1) consume an undefined token and 2) produce an error message, when attempting to expand it. Currently latexml silently returns an undefined token from $gullet->readXToken, as if it were a valid input. I'll attempt to sculpt a good PR here, will have to wrestle with debugging the failing tests first...

[dginev]

Fixed via #1188 , and opens up the path to #1082