Advice re. building/packaging fonts & font-support for 8-bit engines

[cfr42]

I don't know if this is the (or a) right place to ask, so tell me to get lost otherwise. [I have zero experience with GitHub discussions.]

I basically want to ask if

• I should be using l3build at all;
• if I should combine l3build with something else (either make or something simpler like a shell script)
  • if so, whether one should call the other or not.

I have a bunch of font packages (which correspond to those I've put on CTAN, but the latter are all badly out of date). These all depend on nfssext-cfr. nfssext-cfr has a fairly minimal build.lua and an experimental test file. The package doesn't use dtx/ins (though it could, if that matters).

So I have a directory nfssext and then I have a bunch of sub-directories: nfssext-cfr, electrumadf etc. (Not a bundle so there's no top-level build script.) I'm currently trying to figure out how to set up electrumadf in a way which makes some kind of sense (at least to me). I started with this one because it is simpler than some of the other cases: fewer fonts, a single .sty and relatively simple configuration.

So electrumadf basically consists of the standard kinds of files you'd expect from an end-user perspective: enc, map, tfm, vf, pfb, fd, sty and documentation (pdf, readme & manifest). It uses fontinst, finstmsc, plttf, vptovf etc. to generate the font files. In order to create the package, I need to

1. unpack the dtx -> some etx, mtx + sty and a couple of tex
2. gather additional files -> some etx, mtx, possibly enc
   • really, the enc should be generated, but to do that I'd have to isolate the build directory from TL's tree, which I obviously can't do, so I think I just specify these by hand
3. compile *-drv.tex, convert *.pl to *.tfm and *.vpl to *.vf and compile *-map.tex
4. compile *.dtx and a test *.tex
5. bundle everything up for CTAN.

As far as I understand it, l3build and docstrip can do 1, 2, 4 and 5. But I don't think they can do 3 - at least, not easily.
- I don't think I can use l3build to compile *-{drv,map}.tex since they are not docs and the format differs from the documentation (?)
- I could just execute things directly from build.lua, but I don't know how to ensure things are done in the correct order (?)

make could do all of it (in theory), but not anywhere near as conveniently. Moreover, I cannot set up testing with make which would be anywhere near as good as l3build. (That's not so important for electrumadf, but it matters rather more for nfssext-cfr where visual comparisons would be much harder, even if I had fonts with all the relevant features. Since I don't, it is impossible. With l3build I can easily fake them, but I need the logging output and I want it normalised.)

On the other hand, make can easily handle 3, which I'm not sure l3build can. In particular, make can easily handle the dependencies and build recipes. (I'm not complaining l3build can't do this. I don't see any reason it should. Anymore than I'd expect make to handle TeX-specific testing.)

So should I use both make and l3build?

1. If so, should I simply do this manually? That is, run l3build unpack, then run make and then run l3build test etc.?
2. The 'obvious' option would be to have make run l3build and handle all the dependencies, but, as far as I can tell, l3build always recreates targets even if the sources have not changed, whereas make relies on this not happening. (Or perhaps I should say that docstrip always recreates targets rather than l3build.)
3. So I could have l3build call make, but I'm not sure how to handle the sequencing and doing so may still be liable to problems of illusory newness.

Or should I just handle 3 above by hand and/or shell script? (I have scripts to do all kinds of stuff l3build now does better, but not, apparently, this.)

Or only use l3build for tests?

-- $Id: build.lua 10081 2024-06-17 02:58:24Z cfrees $
-- Build configuration for electrumadf
os.setenv ("PATH", "/usr/local/texlive/bin:/usr/bin:")
os.setenv ("TEXMFHOME", ".")
os.setenv ("TEXMFLOCAL", ".")
os.setenv ("TEXMFARCH", ".")
--
module = "electrum"
ctanpkg = "electrumadf"
maindir = "../.."
checkdeps = (maindir .. "/nfssext-cfr")
unpackfiles = {"*.ins"}
-- the next line is wrong because these aren't documentation
-- and need a different format
typesetfiles = {"*.tex"}
auxfiles = {"*.aux"}
sourcefiles = {"*.dtx","*.ins"}
checkengines = {"pdftex"}
checkformat = "latex"
checksuppfiles = {""}
manifestfile = {"manifest.txt"}
bakext = ".bkup"
binaryfiles = {"*.pdf", "*.zip", "*.vf", "*.tfm", "*.pfb", "*.tar.gz"}
-- vim: ts=2:sw=2:tw=80

Or I could unravel the dtx/ins since I only just barely learnt to do this and the process isn't really endearing docstrip to me, but, though that would make things more consistent, I don't think it would actually help.

[I have read most of the discussions. I also searched for relevant bugs, either open or closed. And I searched TeX SE. I found one question about l3build and packaging fonts, but that didn't involve supporting 8-bit engines or, therefore, generating all the files required to do so. I have tried to read the l3build documentation but some of it might just as well be written in ancient Aramaic. I just don't know enough to know what it is telling me. I don't even know enough to know what it is I don't know except in a few very limited cases. If I'm really not meant to be trying to do this without learning Lua first, as opposed to trying to figure out bits of it along the way, just say so. In that case, it would help a lot to know which bits of Lua are required.]

[josephwright]

This feels to some extent like what we have to do for the kernel: we need there to do non-standard tasks relatively early on, in that case creating format files. That means coding up some of the work yourself in Lua, as the simple 'fill in the blanks' idea runs out of steam, but I don't think it's impossible. (Unpacking with the tree isolated is also something the kernel needs, so again that aspect for .enc files sounds doable.)

[cfr42]

This feels to some extent like what we have to do for the kernel: we need there to do non-standard tasks relatively early on, in that case creating format files. That means coding up some of the work yourself in Lua, as the simple 'fill in the blanks' idea runs out of steam, but I don't think it's impossible.

Can you tell me where I can see how this is done?

(Unpacking with the tree isolated is also something the kernel needs, so again that aspect for .enc files sounds doable.)

I'm not convinced this is doable. The problem is that I'd need the TL tree to both be accessible and not. That is, I need other files to be found by the very same process. The only way I can really think to do it would be to have the TL tree with electrumadf not installed, but that doesn't seem practical. Ideally, I guess I'd have some kind of minimised tree, but that seems unduly complicated.

[I think I also don't really understand the use of 'unpacking' in l3build. 'Unpack' = unpack + build? This is a conceptual question rather than a technical one, I think. It isn't that I don't understand the target description.]

[josephwright]

If you look at build-config.lua for the LaTeX2e repo, you'll see we use checkinit_hook() and docinit_hook() to add a call to fmt(), a local function which builds formats. This does the hard work and can turn searching on-and-off inside it (whcih we need as the doc version needs searching). In the end, each time you call an external process you can arrange to turn on or off tree searching by setting TEXINPUTS, etc., as required.

[josephwright]

[I think I also don't really understand the use of 'unpacking' in l3build. 'Unpack' = unpack + build? This is a conceptual question rather than a technical one, I think. It isn't that I don't understand the target description.]

The unpack target is really just needed for .ins/.dtx combos - anything more complex currently is handled in the two hooks I've mentioned - but we could add something to unpack to allow generic extensions.

[cfr42]

Thanks.

https://github.com/latex3/latex2e/blob/develop/build.lua

https://github.com/latex3/latex2e/blob/develop/build-config.lua

If you look at build-config.lua for the LaTeX2e repo, you'll see we use checkinit_hook() and docinit_hook() to add a call to fmt(), a local function which builds formats. This does the hard work and can turn searching on-and-off inside it (whcih we need as the doc version needs searching). In the end, each time you call an external process you can arrange to turn on or off tree searching by setting TEXINPUTS, etc., as required.

It's more a question of how fontinst etc. works. If in creating the files it finds a needed .enc isn't available, it writes one. If it already exists, it doesn't. But fontinst is part of the same tree as electrumadf, so I don't see an easy option for isolation here. It doesn't know that the .enc it finds is part of the package which the current one will replace.

[cfr42]

I came up with this which possibly works, though probably not properly.

-- $Id: build.lua 00000 2024-06-28 17:30:48Z fake $
-- Build configuration for testadf
ctanpkg = "testadf"
module = "test"
vendor = "arkandis"
dofile("../nfssext/fontinst.lua")
sourcefiles = {"*.dtx", "*.ins", "afm/*.afm"}

where fontinst.lua includes

-- $Id: fontinst.lua 10123 2024-07-01 05:24:17Z cfrees $
os.setenv ("PATH", "/usr/local/texlive/bin:/usr/bin:")
os.setenv ("TEXMFHOME", ".")
os.setenv ("TEXMFLOCAL", ".")
os.setenv ("TEXMFARCH", ".")
--
nifergwall = 0
ntarg = "fnttarg"
function gwall (msg,file,rtn)
        file = file or "current file"
        msg = msg or "Error: "
        rtn = rtn or 0
        if rtn ~= 0
                then
                        nifergwall = nifergwall + rtn
                        print (msg .. file .. " failed (" .. rtn .. ")\n")
                end
end
function finst (patt,dir,mode)
  dir = dir or "."
  mode = mode or "nonstopmode"
        local cmd = "pdftex --interaction=" .. mode
        local targs = {}
        targs = filelist(dir,patt)
        for i,j in ipairs(targs)
                do
                        local errorlevel = tex(j,dir,cmd)
                        gwall("Compilation of ", j, errorlevel)
                end
end
function fontinst (dir,mode)
  dir = dir or unpackdir
        mode = mode or "errorstopmode --halt-on-error"
        if not direxists(dir) then
                print("Missing directory. Unpacking first.\n")
                local errorlevel = unpack()
        end
  for i,j in ipairs(familymakers)
                do
                        local errorlevel = finst(j,dir,mode)
                        gwall("Compilation of driver ", j, errorlevel)
                end
        if nifergwall ~= 0 then return nifergwall end
        for i,j in ipairs(mapmakers)
                do
                        local errorlevel = finst (j,dir,mode)
                        gwall("Compilation of map ", j, errorlevel)
                end
                if nifergwall ~= 0 then return nifergwall end
        for i,j in ipairs(binmakers)
        do
                local targs = filelist(dir,j)
                for k,m in ipairs(targs)
                        do
                                targ = dir .. "/" .. m
                                for line in io.lines(targ)
                                        do
                                                if string.match(line,"^pltotf [a-zA-Z0-9%-]+%.pl [a-zA-Z0-9%-]+%.tfm$") then
                                                        local errorlevel = runcmd(line,dir)
                                                        gwall("Creation of TFM using " .. line .. " from ", j, errorlevel)
                                                else
                                                        print("Ignoring unexpected line \"" .. line .. "\" in " .. j .. ".\n")
                                                        nifergwall = nifergwall + 1
                                                end
                                        end
                                end
        end
        if nifergwall ~= 0 then return nifergwall end
        local targs = filelist(dir,"*.vpl")
        for i,j in ipairs(targs)
                do
            local cmd = "vptovf " .. j
                        local errorlevel = runcmd(cmd,dir)
                        gwall("Creation of virtual font from ", j, errorlevel)
                end
        return nifergwall
end
target_list[ntarg] = {
        func = fontinst,
        desc = "Creates TeX font file",
        pre = function(names)
                if names then
                        print("fontinst does not need names\n")
                        help()
                        exit(1)
                end
                return 0
        end
}
--
auxfiles = {"*.aux"}
bakext = ".bkup"
binaryfiles = {"*.pdf", "*.zip", "*.vf", "*.tfm", "*.pfb", "*.tar.gz"}
binmakers = {"*-pltotf.sh"}
checkengines = {"pdftex"}
checkformat = "latex"
checksuppfiles = {""}
cleanfiles = {"*.log", "*.pdf", "*.zip", "*.vf", "*.tfm", "*.fd", "*.vpl", "*.pl", "*-pltotf.sh", "*.map", "*.tar.gz"}
familymakers = {"*-drv.tex"}
manifestfile = {"manifest.txt"}
mapmakers = {"*-map.tex"}
sourcefiles = {"*.afm", "*.dtx", "*.ins", "*.otf", "*.ttf"}
vendor = vendor or "public"
tdslocations = {
        "fonts/afm/" .. vendor .. "/" .. module .. "/" .. "*.afm",
        "fonts/enc/dvips/" .. module .. "/" .. "*.enc",
        "fonts/map/dvips/" .. module .. "/" .. "*.map",
        "fonts/opentype/" .. vendor .. "/" .. module .. "/" .. "*.otf",
        "fonts/tfm/" .. vendor .. "/" .. module .. "/" .. "*.tfm",
        "fonts/truetype/" .. vendor .. "/" .. module .. "/" .. "*.ttf",
        "fonts/type1/" .. vendor .. "/" .. module .. "/" .. "*.pfb",
        "fonts/vf/" .. vendor .. "/" .. module .. "/" .. "*.vf",
        "source/fonts/" .. module .. "/" .. "*.etx",
        "source/fonts/" .. module .. "/" .. "*.mtx",
        "source/fonts/" .. module .. "/" .. "*-drv.tex",
        "source/fonts/" .. module .. "/" .. "*-map.tex",
        "tex/latex/" .. module .. "/" .. "*.fd",
        "tex/latex/" .. module .. "/" .. "*.sty"
}
typesetdeps = {"*.fd", "*.pfb", "*.tfm", "*.vf", "*.enc", "*.map", "*.sty"}
typesetsourcefiles = {"*.pfb"}
unpackexe = "pdflatex"
unpackfiles = {"*.ins"}
-- vim: ts=2:sw=2:tw=80

I'm really surprised it works at all, but it does create the vf, tfm, fd and map files. If the search path is set up to block it finding the rc file, it even creates an enc. [Conveniently in this case, if the rc is kept out of the TL tree and local/home/arch are blocked, the system is technically misconfigured and the programme decides the enc is missing, even though it is not.]

l3build fnttarg

returns

Missing directory. Unpacking first.

This is pdfTeX, Version 3.141592653-2.6-1.40.26 (TeX Live 2024) (preloaded format=pdflatex)
restricted \write18 enabled.
entering extended mode
(./testadf.ins
LaTeX2e <2023-11-01> patch level 1
L3 programming layer <2024-04-11>
(/usr/local/texlive/2024/texmf-dist/tex/latex/base/docstrip.tex
Utility: `docstrip' v2.6b <2022-09-03>
English documentation    <2023-10-10>

**********************************************************
* This program converts documented macro-files into fast *
* loadable files by stripping off (nearly) all comments! *
**********************************************************

********************************************************
* No Configuration file found, using default settings. *
********************************************************

)

Generating file(s) ./test.sty ./xxx-drv.tex ./xxx-map.tex ./reglyph-xxx.tex ./s
upp-xxx.etx ./t1-xxx.etx

Processing file testadf.dtx (sty) -> test.sty
(drv) -> xxx-drv.tex
(map) -> xxx-map.tex
(reglyph) -> reglyph-xxx.tex
(supp-xxx) -> supp-xxx.etx
(t1-xxx) -> t1-xxx.etx
Lines  processed: 1702
Comments removed: 91
Comments  passed: 2
Codelines passed: 1595

)
No pages of output.
Transcript written on testadf.log.
This is pdfTeX, Version 3.141592653-2.6-1.40.26 (TeX Live 2024) (preloaded format=pdftex)
restricted \write18 enabled.
entering extended mode
(./xxx-drv.tex
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/base/fontinst.sty
No file fontinst.rc.
) (./xxxr8a.afm)
Metrics written on xxxr8a.mtx.
(./xxxr8a.mtx)
Raw font written on xxxr8a.pl.
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinetx/8r.etx) (./xxxr8a.mtx
)
Transformed metrics written on xxxr8r.mtx.
(./xxxr8r.mtx)
Raw font written on xxxr8r.pl.
(./xxxr8a.afm)
Metrics written on xxxr8a.mtx.
(./xxxr8a.mtx)
Raw font written on xxxr8a.pl.
(./supp-xxx.etx) (./xxxr8a.mtx)
Transformed metrics written on xxxr8s.mtx.
(./xxxr8s.mtx)
Raw font written on xxxr8s.pl.
(./reglyph-xxx.tex (./xxxr8s.mtx)
Reglyphed metrics written on xxxr8sr.mtx.
) (./xxxr8r.mtx) (./xxxr8sr.mtx)
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinmtx/newlatin.mtx
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinmtx/llbuild.mtx
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinmtx/ltcmds.mtx))
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinmtx/lubuild.mtx)
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/latinmtx/ltpunct.mtx))
(./t1-xxx.etx) (./t1-xxx.etx) (./t1-xxx.etx) (./t1-xxx.etx)
Virtual font written on xxxr8t.vpl.
Font definitions written on t1xxx.fd.
Font transformation records written on xxx-rec.tex.
)
No pages of output.
Transcript written on xxx-drv.log.
This is pdfTeX, Version 3.141592653-2.6-1.40.26 (TeX Live 2024) (preloaded format=pdftex)
restricted \write18 enabled.
entering extended mode
(./xxx-map.tex
(/usr/local/texlive/2024/texmf-dist/tex/fontinst/base/finstmsc.sty
No file finstmsc.rc.
) (./xxx-rec.tex (./supp-xxx.etx)
Encoding vector written on supp-xxx.enc.
)
Map file fragments written on xxx.map.
Map file fragments written on xxx-pltotf.sh.

)
No pages of output.
Transcript written on xxx-map.log.
I had to round some heights by 0.0145006 units.
I had to round some depths by 0.0185003 units.
I had to round some heights by 0.0205002 units.
I had to round some depths by 0.0070000 units.
I had to round some heights by 0.0205002 units.
I had to round some depths by 0.0089998 units.

I should really test for something better than mere directory existence (especially since I believe clean leaves at least some directories), but I'm not actually sure what the correct behaviour is. Is there a way of checking if unpacking has been done?

Anyway, I'm not really sure what to do with this at this point. (I'm even less sure about the cross-platform bit.)