Fix #8780: long words in narrow columns may not be hyphenated #8781

jfbu · 2021-01-28T23:38:58Z

This is done by adding '\hskip0pt\relax\n' whenever a paragraph starts.

Relates

LaTeX: long words in narrow columns may not be hyphenated #8780, Fix #8072: Directive hlist not implemented in LaTeX #8779

edited I am removing a screenshot which contained an inaccurate assertion. See updated screenshot in next comment #8781 (comment)

As explained (in an edit) at #8780 the hyphenation in the first table worked because it used tabulary. The second table also uses tabulary but with the cell containing a list, it fails without extra intervention.

Tabular and longtable do not hyphenate without intervention the first word of a paragraph in a table cell.

This phenomenon is not related to tables, but to narrow width. So it will also occur with #8779 adding support for hlist's in LaTeX, which a bit like table have columns of sometimes narrow width.

This being said, I am hesitant about this PR. It seems costly to add the \hskip0pt\relax at start of each paragraph (I didn't use \hspace{0pt} for reasons of needing to get rid of influence of space token in source after it, possibly, in case of an item list for example this can cause extra whitespace). I had to change many test files to reflect the change.

Also, for this PR to be safe it must be certain that the \hskip0pt\relax is really followed always with words: if for example it is first in a list item and then comes a list, that nested list will suffer a vertical shift.

Besides, perhaps we should not merge this at all but only mention the work-around that I had used to fix a problem with our pdf docs. The trick is simply to define a substitution this way:

.. |LaTeXHyphenate| raw:: latex

                    \hspace{0pt}

and use it manually where needed, like this

|LaTeXHyphenate|\ longwordatstartofaparagraphinnarrowwidthcontext

I have no strong opinion. And I hesitated to ask for merg in 3.x or master branch.

This is done by adding '\hskip0pt\relax\n' whenever a paragraph starts.

jfbu · 2021-01-28T23:46:09Z

sphinx/writers/latex.py

-            self.body.append('\n')
+            # the \hskip0pt\relax is to allow hyphenation of first word of
+            # a paragraph in narrow contexts such as in a table cell
+            self.body.append('\\hskip0pt\\relax\n')


It is crucial that this is never inserted before "vertical" material, in particular before a list. Else it will create extra vertical space. I hope that visit_paragraph() will always correspond to starting an actual paragraph in TeX sense of the word. But with substitutions I am not sure.

Adding this rather for example at start of each list item like I first considered (due to relevance to hlist of #8779) can not work because the item might itself be a list and adding this will, as I mentioned, then create extra whitespace due to entering horizontal mode with no reason, only to leave it immediately for a "vertical" structure.

Took me some time to change all test files, but if there is a doubt here, better to not merge this and find another way.

Sorry, this has a bug of suppressing in output a blank line which is needed by TeX to recognize paragraphs. I am correcting this stupidity in next commit, which will abstract the modification into \sphinxAtStartPar.

jfbu · 2021-01-28T23:49:31Z

tests/roots/test-latex-table/expects/complex_spanning_cell.tex

 table having …
 \begin{itemize}
-\item {} 
+\item {} \hskip0pt\relax


I thought about modifiying visit_list_item but as the item might be a nested sublist, the insertion must come from actual paragraph. Notice here that the blanks in the sequence {} \hskip0pt\relax introduces no extra space at all in pdf output.

Now \hskip0pt\relax is modified into \sphinxAtStartPar, and here it would be on next line so the \item {}<space> line remains exactly as in current Sphinx. So this will not modify things for those who parse output file for this specific line for whatever purpose.

Define it to do nothing with lualatex engine

jfbu · 2021-01-29T09:54:04Z

The \sphinxAtStartPar line is added as first line of almost all text paragraphs. Not in footnotes, but there is already there \sphinxAtStartFootnote which indirectly also allows TeX to hyphenate first word.

To review this, the thing to check is whether this insertion of \sphinxAtStartPar will really only happen before things that start a paragraph in TeX sense. Indeed it triggers TeX entering into "LR mode" (LaTeX vocabulary), or "horizontal mode" (TeXbook) and if immediately after that there is for example a list environment, this will cause TeX to finish the horizontal mode, hence output in pdf a blank line, resulting in extra vertical space. Also, if a blank line in tex file follows the \sphinxAtStartPar this will cause a blank line in output.

jfbu · 2021-01-29T10:22:06Z

Here is updated test file
index.rst.txt

With current Sphinx it produces this

With this PR it will give this

and

Here, the hor- is still too long to fit, so it gets shifted down vertically from being first list item, but this is extreme situation and we can't do much about it. If the column had been a bit wider, it would have fitted:

note: some captions in the screenshots say that lists in table cells trigger usage of tabular or longtable and not tabulary. However if one uses tabularcolumns directive one can again force usage of tabulary even with a list in a cell.

jfbu · 2021-01-29T10:52:56Z

Relates to #3042 but does not fix it.

Else, a non-hyphenatable long word as first word in a narrow column in a longtable/tabular (with column type e.g. p{1cm} from tabularcolumns directive) gets shifted downwards vertically in PDF output. Memo: 1. I did not find other cases where such a vertical shift may occur (I tried with deeply nested lists and artificial words such as 'A'*32) with LaTeX mark-up produced by Sphinx, 2. but with the support of hlist directive via PR sphinx-doc#8779 using multicols environment, there is again this situation of downwards shift of non-hyphenatable long first words. But it occurs whether or not \sphinxAtStartPar is used (\nobreak does not modify this).

tk0miya

Very cool!! +1 for merging. If your worry came from the vertical material of LaTeX, you don't need to care it because the children of paragraph node is all inline nodes (basically).

tk0miya · 2021-01-29T15:07:45Z

sphinx/writers/latex.py

-            self.body.append('\n')
+            # the \sphinxAtStartPar is to allow hyphenation of first word of
+            # a paragraph in narrow contexts such as in a table cell
+            self.body.append('\n\\sphinxAtStartPar\n')


The children of paragraph node are regularly text nodes or inline nodes. It is defined at the doctree specification.
https://docutils.sourceforge.io/docs/ref/doctree.html

Almost of the inline nodes are decorated text. But, some of them are not; for example, math, image, problematic and raw. Additionally, some extensions might generate broken doctree under the paragraph node.

A math node is fine (via the math role), an image (via a substitution) also is no problem. As per raw latex, also inserted by a substitution then if the raw latex starts some list environment it will be shifted one line down, but it seems then a user error to do this, the raw directive should have been used directly to insert the LaTeX "display" material in place of "inline" material. I can not think of a natural example causing a problem, after all docutils "paragraph" concept pretty well matches with latex "paragraph" concept I hope.

Agreed. From my perspective, docutils' paragraph model and LaTeX's horizontal mode are similar. (But I did not mention that yesterday because I'm not familiar with LaTeX's mode :-p)

tk0miya · 2021-01-29T15:10:02Z

doc/latex.rst

+  in a narrow context (like a table cell). For ``'lualatex'`` which
+  does not need the trick, the `\sphinxAtStartPar` does nothing.
+
+  .. versionadded:: 3.5.0


I don't know this change will break PDF generation. But it would be better to add big change into the next major release. It's planned for this April.

To the best of my knowledge the change can not break PDF generation. As far as I can tell, at worst it could add an extra blank line if the first child is implemented via a "vertical" environment: typically a list, but this seems excluded: in my understanding a list can not be first child of a paragraph node. An image is not a problem if it is not embedded into "figure" (for example I image inserted from a substitution). I tested that math like equation or align do not seem to be affected. At any rate there is no way I can see this can break PDF build whatever nodes follow. The only thing it can cause is this extra blank line. But I can not construct an example from rst source creating this.

To the best of my knowledge the change can not break PDF generation.

Okay. I understand and agreed this does not break PDF generation. Let us see what will be reported.

(for example I image inserted from a substitution)

Fortunately, reST's substitution only allows inline elements. Nobody can't insert non "inline" elements under the paragraph node without hacks (by raw node or extensions).

A substitution definition block contains an embedded inline-compatible directive (without the leading ".. "), such as "image" or "replace".
https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#substitution-definitions

jfbu · 2021-01-29T19:57:26Z

Thank you @tk0miya always for reviewing and advice! What this code does is simply to add in LaTeX output at the start of (most) paragraphs a macro \sphinxAtStartPar which does almost nothing (it inserts horizontal glue of zero width). This is a trick to allow hyphenation of first word, especially in tables (tables rendered by tabulary do exactly this same trick already from the latex code of package tabulary, but if user has employed tabularcolumns with some p{} specifier, this would be lost). With lualatex the added macro does nothing because lualatex engine does not have this limitation of original (La)TeX.

I think this can not be breaking change, thus now merging into 3.x!

tk0miya · 2021-01-30T00:08:37Z

🎉

…inx-doc#8781 This is cosmetic as the blank line starting varwidth environment used for merged table cells in latex output changed nothing to PDF. Nevertheless I extended a unit test to have a multi-paragraph merged cell using varwidth. What is important is that \sphinxAtStartPar line itself is never followed by blank line.

…r_8781 Let latex writer line trimming from depart_entry() work as before #8781

Fix sphinx-doc#8780: long words in narrow columns may not be hyphenated

185a2d2

This is done by adding '\hskip0pt\relax\n' whenever a paragraph starts.

jfbu added type:bug builder:latex labels Jan 28, 2021

jfbu commented Jan 28, 2021

View reviewed changes

Fix suppression of needed whiteline, encapsulate into \sphinxAtStartPar

d6e11b8

Define it to do nothing with lualatex engine

jfbu mentioned this pull request Jan 29, 2021

LaTeX renders a table as collapsed by default #3042

Open

jfbu mentioned this pull request Jan 29, 2021

Fix #8072: Directive hlist not implemented in LaTeX #8779

Merged

tk0miya approved these changes Jan 29, 2021

View reviewed changes

jfbu merged commit 879bf54 into sphinx-doc:3.x Jan 29, 2021

jfbu deleted the latex_hyphenation_of_first_word branch January 29, 2021 20:08

jfbu mentioned this pull request Jan 30, 2021

Let latex writer line trimming from depart_entry() work as before #8781 #8787

Merged

jfbu added a commit that referenced this pull request Jan 30, 2021

Merge pull request #8787 from jfbu/latex_repair_celllinetrimming_afte…

5969163

…r_8781 Let latex writer line trimming from depart_entry() work as before #8781

This was referenced Feb 6, 2021

LaTeX: table spacing sometimes broken on 3.x branch from \sphinxAtStartPar mark-up #8838

Closed

3.5.0 release plan #8798

Closed

Revert "Add \nobreak inside \sphinxAtStartPar" #8840

Merged

github-actions bot locked as resolved and limited conversation to collaborators Jul 16, 2021

Uh oh!

Fix #8780: long words in narrow columns may not be hyphenated #8781

Fix #8780: long words in narrow columns may not be hyphenated #8781

Uh oh!

Conversation

jfbu commented Jan 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relates

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfbu Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfbu commented Jan 29, 2021

Uh oh!

jfbu commented Jan 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfbu commented Jan 29, 2021

Uh oh!

tk0miya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfbu commented Jan 29, 2021

Uh oh!

tk0miya commented Jan 30, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jfbu commented Jan 28, 2021 •

edited

Loading

jfbu Jan 29, 2021 •

edited

Loading

jfbu commented Jan 29, 2021 •

edited

Loading