\textasciigrave with courier triggers textcomp info/warning and font substitution but glyph appears to be available in that encoding

[jfbu]

Brief outline of the bug

I can not understand

Package textcomp Warning: Symbol \textasciigrave not provided by
(textcomp) font family pcr in TS1 encoding.
(textcomp) Default family used instead on input line 13.

and the font substitution to cmtt, as shown via \showoutput in log (....\TS1/cmtt/m/n/10 ` )

But if I select manually TS1/pcr/m/n and execute \textasciigrave, then there the actual glyph (at slot 96 in TS1 encoding) seems to make its way to output, same as if using \char96. (....\TS1/pcr/m/n/10 `)

There is no such problem with \textquotesingle.

Minimal example showing the bug

\RequirePackage{latexbug}
\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[warn]{textcomp}
\usepackage{courier}
\begin{document}

\ttfamily

\makeatletter
\f@encoding/\f@family/\f@series/\f@shape

\textasciigrave % (info in log, cmtt font substitution used)

\textquotesingle

\usefont{TS1}{pcr}{m}{n}
\edef\x{\f@encoding/\f@family/\f@series/\f@shape}
{\normalfont\ttfamily\x}

\textasciigrave \char96 % (no warning, and is same glyph as \char96)
% wasn't it supposedly missing in TS1 encoding ?
% that is what warning said earlier

\textquotesingle \char39

\showoutput

\thispagestyle{empty}
\end{document}

Log file (required) and possibly PDF file

textasciigrave.log

textasciigrave.pdf

[jfbu]

Although I can not test until a few days on a TeXLive 2018 install, I tried with a git checkout of LaTeX2e <2018-12-01> (hoping the procedure emulates enough of what is at stake, and that I don't have to do things with actual font glyphs related files) and indeed, there was no such behaviour then. The log contains ....\TS1/pcr/m/n/10 ` and never a TS1/cmtt font substitution. Checking in adobe reader, only "Nimbus Mono" is used no "SFTT" which does appear if build with current (or also with the final TL2019) LaTeX. I am attaching the log of this build with LaTeX2e <2018-12-01> (basename ends with tl2018 but this is not real TeXLive 2018).
textasciigrave_tl2018.log

In real life this issue arises because courier is still default configuration at sphinx-doc/sphinx for pdflatex, and console output (and log file) are cluttered with the textcomp warnings (it is used with warn option in order to avoid build breaking error messages in some unrelated cases) from occurences of the backtick in code listings or inline literals (which happens a lot in the documentation of sphinx itself). The reason it is escaped to \textasciigrave is to avoid ligatures and curly quotes rendering across all engines pdftex, luatex, xetex, and because it seemed as if it was the right thing to do.

[jfbu]

Seems problem is not here at release-2019-10-01-PL3 and appears at release-2020-02-02.

[FrankMittelbach]

You may not believe how many hours I spent looking through hundreds of of nfssfont tables to come up with some sort of reasonable suset classification for TS1 fonts. Basically there is none. With just CM and a few early build postscript fonts (through virtual files) life was fairly easy and we had just 5 sub-encodings that reasonably described the available fonts.
Nowadays we have hundreds of fairly high-quality fonts out there and the diversity on supported and unsupported fonts is overwhelming. In the end I settled for 9 sub-encodings and sorted a huge number of fonts into these sub-encodings.
This was done conservatively, that is always (or nearly always) making sure that all glyphs from thesub-encoding are actually available (to avoid TOFU) but allowing to err on some glyphs and substitute even though not necessary for that glyph, if the font had a strange selection that didn't fit in any sub-encoding.

The old PS fonts are of that nature and source2e documents this on page 436:

The next set of glyphs is special to TeX fonts (and available with a few older PS fonts supported in the virtual fonts), but not any longer in the majority of fonts provided through autoinst, so we pretend there aren’t available in sub-encoding 2 and below.

The \textasciigrave falls into that category.

Concerning the old PS fonts (of which pcr is one) those are sorted into subencoding 5 and the documentation states

Most older PS fonts (supported in T E X since the early nineties when virtual fonts became available) are sorted under this sub-encoding. But in reality, many of them don’t have all glpyhs that should be available in sub-encoding 5. Instead they show little squares, i.e., they produce “tofu” if you are unlucky.

But the coverage is so random that it is impossible to sort them properly and if we tried to ensure that they only typeset those glyphs that are really always available then that would put them all into sub-encoding 9 so that’s a compromise really.

So bottom line this works as designed.

Basically you have a couple of options:

• explicitly change the behavior of \textasciigrave, e.g., to take it unconditionally from TS1
• claim that pcr has more symbols than it has and accept that then some will show up as TOFU if used
• use a more modern font with better glyph coverage, e.g., qcr (the TeX Gyre alternative for Courier)
  In other words do one of these:

\DeclareTextSymbolDefault{\textasciigrave}{TS1}
\DeclareEncodingSubset{TS1}{pcr} {2} 
\usepackage{tgcursor}

The last alternative also has the advantage that you get a proper glyph and not a faked one (as the one in pcr) that doesn't fit into the font at all as it is taken from CM (maybe?) and just placed into the the font virtually.

[FrankMittelbach]

sorry should have been

\DeclareEncodingSubset{TS1}{pcr} {1} 

of course.

[jfbu]

Thanks for the detailed explanations. I will try out your advices to work around this change occuring at LaTeX2e 2020-02-02.

The \textasciigrave in my context may come in 2 ways: either in verbatim like context via the services of package upquote as the ` character is made active; elsewhere, i.e. in regular text body, by default rendered using times font, or in what we call "inline literals", and there \textasciigrave may be explictly inserted. (not with luatex/xetex). Thus we have the same issue with ptm, it is not only pcr. The goal for us is to get straight quotes whenever the user had a backtick in the (non-latex) source (which, in non-literal contexts, should be itself rst escaped to \`, but that is another irrelevant point).

Problem is that user may change the font configuration and the solution should be robust against that.

[jfbu]

The warning message reads very counter-intuitive. It says that \textasciigrave is not provided by pcr or ptm in TS1 encoding. But if user explicitly select that encoding, \textasciigrave triggers no warning and no font substitution.

[FrankMittelbach]

The warning message reads very counter-intuitive. It says that \textasciigrave is not provided by pcr or ptm in TS1 encoding. But if user explicitly select that encoding, \textasciigrave triggers no warning and no font substitution.

true but what would you say at a time the code simply doesn't know? you can't really check, for example many font put random ligatures into textpertenthousand and in other places so even if there is a glyph it may not be the right one.

As far as LaTeX is concerned the glyph is not available --- perhaps it should not say TS1 encoding but just that it is not available.

[FrankMittelbach]

Thus we have the same issue with ptm, it is not only pcr. The goal for us is to get straight quotes whenever the user had a backtick in the (non-latex) source (which, in non-literal contexts, should be itself rst escaped to \`, but that is another irrelevant point).

Problem is that user may change the font configuration and the solution should be robust against that.

well the warning is actually correct for both pcr and tpm because what you get is a substituted glyph (from CM) if you use the font, only it is already substituted in the font (and now LaTeX tells you so). Look at the glyph that shows up.

Basically both pcr and ptm are (bad) early days attempts to get PS Courier and Times. Much better to not use them (and the obsolete psfonts packages times and courier but instead use the well supported TeX Gyre implementations instead.

[jfbu]

Well, as I comment in the sphinx issue the substituted glyph now matches better with the looks of \textquotesingle. So the pdf result is different than what it was prior to 2020-02-02 but arguably better. The problem is mainly with many console output warnings, one for each instance. Using the TeX Gyre Cursor, one gets rid of warnings, and the backtick glyph matches the uptick and it matches arguably better with the other characters in the font (it is not so heavy as in cmtt, but paradoxically I do like the heavier glyphs).

We are aware that times is very old, and not configured for math fonts. Problem is that it has always been default in Sphinx (we now use FreeFont for Unicode engines). As math is rendered with CM, it matches with the html+mathjax output. But now I remember I have been wanting to switch to newtxtext+newtxmath for a long time (which uses the tex gyre fonts at least since a few years). The problem then is documenting the extra dependencies for our users. We already advise for many years people needing non-ascii alphabets to use luatex or xetex engines. As a result upgrading for TeX Gyre (especially as we won't need its otf fonts, a priori) may be potentially an extra burden on our classic user base, only to solve a problem with one glyph. Nevertheless, as I much prefer the maths from txfonts, now newtxmath, to the Knuth fonts, I have for long time been willing to do that.

[FrankMittelbach]

Well, option 1 or 2 above should give you the old behavior of using the the font as if it has the glyph inside. But in some sense you now really get what you ask for: you ask for warnings and indeed the glyph is substituted so you get a warning which you should have gotten in the past already (though I can understand why you don't want it :-))

[jfbu]

Hi Frank, not to hammer too much on why I came here, in the past there was no warning and the output was not the same. So the change is not only in these warnings (which come by the dozens when compiling Sphinx's own documentation sphinx.pdf) but also in the pdf output. A change for the better, I agree!

[FrankMittelbach]

well yes you are right (not sure why I confused myself thinking there is none) there is a change but you can revert it with option 1 or 2. if you want to go with the font's virtual backticks. Still --- I think of the change as a correction of incorrect output.

[jfbu]

Moving in real life context to TeX Gyre and txtt fonts at next major release. This will fix it for us. People will encounter this issue from a backtick in verbatim if they use upquote package together with courier for monospace. (not closing only because I feel it would not be polite) (I always considered strange that people raising issues are allowed to close them)

[FrankMittelbach]

(not closing only because I feel it would not be polite) (I always considered strange that people raising issues are allowed to close them)

well that puts the burden on me to be impolite and close something without making a change based on the issue :-) I can see it being useful to close as a user when you think your issue has been dealt with or is no longer an issue. On the other hand. for 2e we have adopted the policy to keep issues open after they have been fixed as long as the fix has not be made it into a release on CTAN. Something many people do not understand (e.g. github that cloes the issue for me if I merge a pull request for it) ... different policies.

Ok so we agree that is is a change that has some side effects but nothing we want/can do about other than turning the clock back?

but I'll keep it open and maybe write a line or two in ltnews about the topic

[jfbu]

Ok so we agree that is is a change that has some side effects but nothing we want/can do about other than turning the clock back?

agreed, also because if I wanted to duplicate your hours through nfssfont tables to gather the expertise needed to actually understand exactly what happens, it would take me years :-)

[FrankMittelbach]

it was a rather lengthy and difficult task I can tell you that ... :-)

[stale]

This issue has been automatically marked as stale because it has not had recent activity.

[FrankMittelbach]

There is now some documentation on the problem in ltnews33 in the develop branch