LaTeX: optionally apply a second forceful wrapping of long code lines

[jfbu]

Closes #8849

To try it out:

latex_elements = {
    'sphinxsetup': "verbatimforcewraps",
}

This is a bit experimental as I need to confirm that Pygments LaTeXFormatter output, as we use it, always comply with the pre-conceptions which are summarized in the latex code comments.

For this input:

DryGASCON128k56:

.. code-block:: shell

   $ python3 -m drysponge.drygascon128_aead e 000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F3031323334353637 000102030405060708090A0B0C0D0E0F "" ""
   28830FE67DE9772201D254ABE4C9788D

.. code-block:: shell

   $ python3 -m drysponge.drygascon128_aead e "000102030405060708090A0B0C0D0E0F101112131415161718191A1B1C1D1E1F202122232425262728292A2B2C2D2E2F3031323334353637" 000102030405060708090A0B0C0D0E0F "" ""
   28830FE67DE9772201D254ABE4C9788D

the following output is obtained in PDF:

Non highlighted strings like in first example can break at each character. Highlighted strings only every four characters. That here it breaks without one or two characters going in margin or one or two spaces at end is luck. Here is with removing one character before on same line, we see now it breaks one character short of ending line.

Actually I could make even highlighted strings have breakpoints at each character. But it makes me pain to let the computer suffer with lots of extra work.

[jfbu]

This breaks with Unicode in the code-blocks.... I got a failure from our own sphinx.pdf due to the examples with directory trees, with a test where the "force wrap" method is to be used always. But then it worked with using latex_engine = 'lualatex', i.e. all code-blocks of our own docs were forced to use the experimental code and no error was raised. (even with the feature activated, not a single code-block in our pdf docs by itself triggers the experimental code, because all are already wrapping fine if needed).

[jfbu]

The idea here is that we split the argument of the highlighting macro (which could for example do \colorbox) in small bits of 4 tokens, so that highlighting and breakpoints can both cohabit.

But in case of Unicode code points and latex_engine is "pdflatex", this will break because each of #1, #2, #3, #4 is an octet so we are possibly cutting apart a multi-bytes Unicode character. It would be lots of complications to make it work.

However if Unicode code-points are frequent in source it is better to use 'lualatex', 'xelatex', and 'uplatex'. I did tests with 'lualatex' and it works fine (although much slower than pdflatex).

[sebastien-riou]

@jfbu
Thanks, that seems to fix it indeed. I cannot really try that on my side (by lack of skills to modify my sphinx installation). looking forward to see it in a release.

[jfbu]

I have added Unicode support for the 'pdflatex' engine as well as a way to customize when forceful algorithm is triggered. In particular it is possible to have all codelines be systematically hard-wrapped at the available width, if desired.

Now, the output looks like this (I removed letters/digits to change length, so examples are only typographical):

In the above I added some non-ascii letters, to check with 'pdflatex' engine. I used

latex_elements = {
    'fontenc': "\\usepackage[X2,LGR,T1]{fontenc}",
    'sphinxsetup': "verbatimforcewraps",
}

to allow Greek and Cyrillic. As the font changes, the character width also, and as a result, we see a letter sticking a bit in margin, but this is exceptional. If we typeset the same using Unicode engine, all is exactly aligned:

and I used for this

latex_engine = 'xelatex'

latex_elements = {
#    'fontenc': "\\usepackage[X2,LGR,T1]{fontenc}",
    'sphinxsetup': "verbatimforcewraps",
}

Even with 'pdflatex' all will be aligned with non-ascii letters not from Greek or Cyrillic alphabets which trigger a font substitution.

To hard-wrap all code-blocks at the maximal width:

latex_elements = {
    'sphinxsetup': "verbatimforcewraps, verbatimmaxunderfull=0",
}

[jfbu]

The 3 in 4=3+1 is in truth the value of verbatimmaxoverfull key to 'sphinxsetup'

[jfbu]

The Changes only indicates in this PR that #8849 is fixed but makes no mention so far of the new sphinxsetup keys verbatimforcewraps, verbatimmaxoverfull, verbatimmaxunderfull. Perhaps a line should be added to "Features added" to says something like "possibility to hard-wrap code lines at the linewidth (see verbatimforcewraps)". Else people reading only change log on new release might think #8849 is fixed without user setting, but for security I have set the default of verbatimforcewraps to inactive. (it will probably break people who add extra decoration to the code-blocks such as raw LaTeX mark-up inside the code-blocks, for example in the comments, using Pygments texcomments)

[jfbu]

@jfbu
Thanks, that seems to fix it indeed. I cannot really try that on my side (by lack of skills to modify my sphinx installation). looking forward to see it in a release.

@sebastien-riou
To test you can start from some project and go the latex build directory. Overwrite the sphinx.sty there with the sphinx.styfrom this PR. You also need to open the .tex main document and replace the line \usepackage{sphinx} by the line \usepackage[verbatimforcewraps]{sphinx}. Finally, from the command line in this directory emit make all-pdf or directly pdflatex <name of main tex file>.

Due to potential frailty the verbatimforcewraps will not be default at 3.5.0 release. Once this release is done you need to add to your conf.py

latex_elements['sphinxsetup'] = "verbatimforcewraps"

[tk0miya]

LGTM with nits.

[tk0miya]

As you commented by yourself, it would be better to mention new configurations here.

[jfbu]

Thanks for reviewing, done. I will merge after testing completes. The behaviour is "opt-in" so no change to users if not using the feature.