hbox inside equation is problematic

[Udi-Fogiel]

see https://tex.stackexchange.com/questions/709365/inline-equation-breaks-when-adding-hebrew-text-with-the-witharrows-package for an example. Here are more minimal example

\documentclass{article}
\usepackage[bidi=basic, hebrew,provide=*]{babel}
\babelfont{rm}{David CLM}
\begin{document}
שלום $\hbox{$abc$ שלום}$
\end{document}

\documentclass{article}
\usepackage[bidi=basic, hebrew,provide=*]{babel}
\babelfont{rm}{David CLM}
\begin{document}
שלום $\hbox{abc שלום}$
\end{document}

[seloumi]

If \hbox start with hebrew letter we will get a correct result.

\documentclass{article}
\usepackage[bidi=basic, hebrew,provide=*]{babel}
\babelfont{rm}{DavidLibre}

\begin{document}

\begingroup
\everyhbox{‏}% Right-To-Left Mark (U+200F)
שלום $\hbox{$abc$ שלום}$

שלום $\hbox{abc שלום}$
\endgroup

\end{document}

[jbezos]

@Udi-Fogiel @seloumi Thanks. I usually take a short break when TeXLive is about to be published (I don't like to make any last minute changes, unless it's a severe bug). When it's frozen, I'll start investigating.

[jbezos]

The title summarizes very well the whole point here: \hbox inside equation is problematic. I copy here what I wrote in tex.stackexchange.com:

I’m trying to find a general solution, but below there is as a workaround [...]. Why is this happening? Well, it’s related with the fact \hbox’es [...] are used internally for quite a lot of purposes and there is no way to know if they contain real text or serve to build some special combination of letters, symbols, graphics, etc. Babel attempts to guess how to deal with them, but it may fail.

[Udi-Fogiel]

it’s related with the fact \hbox’es [...] are used internally for quite a lot of purposes and there is no way to know if they contain real text or serve to build some special combination of letters, symbols, graphics, etc. Babel attempts to guess how to deal with them, but it may fail.

I think it would be much easier to solve it upstream (at least in the case of big packages such as tikz). \hboxes meant for intrinsic LTR material could be flagged using a reserved attribute, or use \hpack instead of an \hbox so that the code that babel add to the hpack_filter callback won't affect this box.

[jbezos]

I think it would be much easier to solve it upstream (at least in the case of big packages such as tikz). \hboxes meant for intrinsic LTR material could be flagged using a reserved attribute, or use \hpack instead of an \hbox so that the code that babel add to the hpack_filter callback won't affect this box.

Indeed, in an ideal world— 🙂. But even \put in the humble picture can be used to place either text or graphics (or both). So, I’m thinking of a helper function like the following, which can be useful in the meanwhile:

\documentclass{article}
\usepackage[bidi=basic, hebrew, provide=*]{babel}
\babelfont{rm}{David CLM}

\makeatletter
\newcommand\localebox[1]{%
  {\def\bbl@insidemath{0}%
   \mbox{\foreignlanguage{\languagename}{#1}}}}

\begin{document}

אחת שתיים $\localebox{$abc$ שלום}$

אחת שתיים $\localebox{abc שלום}$

אחת שתיים $\mbox{abc שלום}$ % Just to compare

\end{document}

[Udi-Fogiel]

@jbezos I worked on a port of ConTeXt's code to OpTeX, and it also works with LaTeX and Plain (the requirement is a subset of luaotfload). The code only implements character level bidi, it is not about tabulars or page layout. You can find it here https://github.com/Udi-Fogiel/UAX9-LuaTeX. I thinght about mentioning it as this code works with the examples here and the one in #301.

[jbezos]

@Udi-Fogiel 👍 Great! One of the reasons for requiring the bidi option was to open babel to different options, without imposing any in particular. I’ll take a look at it and make some tests.

[jbezos]

@Udi-Fogiel A question. How fast is it? At first glance, it seems it translates more or less directly the Unicode implementation of the algorithm, with lots of loops, something I avoided for efficiency reasons.

[Udi-Fogiel]

@jbezos I did not test it with long documents yet, but I think it is slower than your code. In ConTeXt itself it is probably even slower as the data file is huge.

The code originates from a port of minibidi.c from Arabeyes to Lua made by Khaled Hosny:
https://github.com/khaledhosny/context-bidi

Even after making the original code running, it had many problems so I eventually decided to use the more up to date code from ConTeXt.

I also tried to port your code to OpTeX, and it almost worked smoothly, I don't remember exactly what broke but I think it was related to code written outside the Lua module in TeX, so it got a bit convoluted...

[Udi-Fogiel]

BTW, any suggestions for a package name? CTAN does not like uax9, and bidi, unicode-bidi and luabidi are taken.

[jbezos]

Naming things is always a headache for me! 🙂

[jbezos]

@Udi-Fogiel It passed my tests with RTL as the main text and LTR as secondary one. On the other hand, embedded Arabic in LTR text is not correctly rendered (without markup). The reason is the directionality of characters is not enough and the font with the proper language system is also required. This is something babel already does, but I’m not sure how easy it will be to integrate your method with babel. I have to investigate.

[Udi-Fogiel]

@jbezos Can you give an example with the expected output?

[Udi-Fogiel]

If you have one with Hebrew it will be better, but Arabic is fine as well.