french typography rules not respected

[legrostdg]

Explain the problem.

It is related to #7976. When typing latex, I'm used to use babel this way: \usepackage[french]{babel}. With this, the typography rules (most notably thin nonbreaking space before ":;!?") are respected. The default template set \usepackage[bidi=default]{babel} and then \babelprovide[main,import]{french}, which for an unknown reason results in no space before ":;!?". Replacing \usepackage[bidi=default]{babel} by \usepackage[french,bidi=default]{babel} in the default template fixes the issue...

Pandoc version?

pandoc 2.17.1.1, debian sid

[legrostdg]

Looking at the babel website, it seems the correct way to set a language is in \usepackage{babel} options, isn't it? Or maybe there is some problem in babel-french where it only recognize this way to set a locale?

https://latex3.github.io/babel/guides/which-method-for-which-language.html
https://latex3.github.io/babel/guides/using-babelprovide-to-modify-or-extend-locales.html "All the examples assume: \usepackage[english]{babel}"

[jgm]

I don't know. We changed to use this method of setting the babel language in commit 3d8f011
@hseg can you comment on this issue?

[legrostdg]

Without \usepackage[french]{babel}, babel-french does not seem to be enabled at all: If I edit the default template to use any babel-french command after \babelprovide[main,import]{french}, it fails miserably.

[jgm]

It does have some effect: e.g. \tableofcontents gives you "Table des matières." Can you say specifically what is missing, besides the punctuation spacing you noted? The special treatment of ;, ?, and ! is disabled by the following lines in the default template:

% get rid of language-specific shorthands (see #6817):
\let\LanguageShortHands\languageshorthands
\def\languageshorthands#1{}

For background on that, see #6817.
However, it looks like commenting out these lines is not enough by itself to make the punctuation spacing work; one also has to add french to \usepackage{babel}.

[jgm]

Possibly helpful: https://latex3.github.io/babel/guides/which-method-for-which-language.html

[jgm]

This answer uses the method we use (including for latin alphabet languages): https://tex.stackexchange.com/questions/513618/howto-multilanguage-babel

[jgm]

This is also informative: https://tex.stackexchange.com/questions/549682/why-can-arabic-not-be-set-as-the-main-language-with-babel-in-the-package-declara/549712#549712

[hseg]

Am away from my dev box, so can only answer based on looking at manuals. Note that the `provide=*` option is equivalent to a `\babelprovide` command for the main language (see 1.13). Also, note the form of language import I implemented forcibly imports the relevant `.ini` file (instead of the `.ldf`?). I suspect this might be the issue - iiuc, the latter is the more fully-featured definition. We *could* try to pass the languages as multiline package options to the initial babel import, unsure if that'd regress my original problems. If doing this, note main language is the *last* one.

[hseg]

Ah, and I see you've provided the necessary clarification on ini vs ldf 😅.

[jgm]

unsure if that'd regress my original problems.

@hseg do you have a test case we could use to check this?

[hseg]

The file I used is on my dev box, so no. Do recall offhand the cases I checked, though: - basic embedding (ie getting rtl1 rtl2 LTR4 LTR3 rtl5 rtl6) - weak directionality - eg numbers, parens at directionality boundaries - especially for the above, consider international phone numbers at the end of an ltr context - the plus sign should be at left edge https://www.w3.org/International/articles/inline-bidi-markup/bidi_examples might have more cases worth checking I see from my original patch that I tried Greek, but don't recall what I tested that with Am 8. September 2022 06:40:36 GMT+03:00 schrieb John MacFarlane ***@***.***>:

> unsure if that'd regress my original problems.

***@***.*** do you have a test case we could use to check this?

…

-- Reply to this email directly or view it on GitHub: #8283 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[legrostdg]

It does have some effect: e.g. \tableofcontents gives you "Table des matières." Can you say specifically what is missing, besides the punctuation spacing you noted? The special treatment of ;, ?, and ! is disabled by the following lines in the default template:

% get rid of language-specific shorthands (see #6817):
\let\LanguageShortHands\languageshorthands
\def\languageshorthands#1{}

For background on that, see #6817. However, it looks like commenting out these lines is not enough by itself to make the punctuation spacing work; one also has to add french to \usepackage{babel}.

With \usepackage[french]{babel} I can at least get the correct punctuation with shorthand: off or by using xelatex. But, yes, that's suboptimal. A french pandoc user should get correct french typography rules by default, the same way he does with latex :-)

When I saw that the punctuation was not respected I tried using \frenchsetup{} in the default template, which is used to customize babel-french behaviour according to the doc. Without \usepackage[french]{babel} I get !Undefined control sequence. ... \frenchsetup

[jgm]

Let me summarize my current understanding of the sitution. Babel is in the process of moving from the old .ldf language definitions (which would be loaded by \usepackage[french]{babel} and the new .ini definitions (which are loaded by \babelprovide[main,import]{french}. This transition is needed because the .ldf definitions don't work well when you combine multiple languages in a document. For many languages (e.g. Hebrew) you really need the .ini definitions. For others, perhaps (e.g. French) the old .ldf definitions may be more full-featured. I would like to know more about this, though. Is there a reason the punctuation spacing is not supported in the French .ini definition? Is this something that is going to be added, or has it been removed for some reason?

What should pandoc do, if anything? Well, we could move to a more complicated system in which \usepackage[langs]{babel} is used for certain languages and \babelprovide for others. I actually don't know if this would cause problems, nor do I know how we'd determine which languages to treat in which way. Note that, even if we did this, the punctuation spacing wouldn't work unless we also removed the lines disabling shorthands (see #6817 -- as noted in that issue, shorthand:off wasn't sufficient to deal with the issue there). This would cause other problems. Perhaps there is a way to keep the special treatment of punctuation while disabling other shorthands?

In the mean time, your best bet is to create a custom template that modifies the default template just slightly, changing the line

\usepackage[bidi=default$if(babel-lang)$,$babel-lang$$endif$]{babel}

and removing the lines

$if(babel-lang)$
\babelprovide[main,import]{$babel-lang$}
$endif$

as well as

% get rid of language-specific shorthands (see #6817):
\let\LanguageShortHands\languageshorthands
\def\languageshorthands#1{}

You can specify the modified template on the command line with --template, or make it a local default by moving it to the templates subdirectory of your user data directory.

[jgm]

I'd be curious to get feedback from other French-speaking pandoc users on the impact of this issue.