Huge rewrite, many fixes

[rm-dr]

Massive overhaul of bundle-building code, fixing #12, #13, and many other things. Pardon the messy git history, there were too many changes to fit into nicely-organized PRs.

What has been done:

• Builds are run from TeXlive tarballs (~3GB) instead of the whole repo (~40GB)
  • also allows for better reproducibility (it's easier to keep track of pinned releases)
• Build process is much simpler, configuration is easier (No tlmgr, just install-tl)
• Cleaner bundle specifications, fewer scripts
• No python. All build tools are written in bash and Rust.
• Improved reproducibility
  • scripts check hash of both input and output files
  • builds of the same bundle are bit-perfect
  • Hashes of every input file are baked into each bundle
• Detailed documentation
• A working TeXlive 2023 bundle, which resolves a few "missing file" issues
• Adding extra sources (not just texlive) is now trivial
• Patches can be provided as diffs, so we can more clearly see what changed.
• Bundles are now distributed as ttb ("tectonic bundle") files, a custom file format documented in scripts/builder/README.md. The ttb format provides a number of advantages over zip and itar bundles, including...
  • The same ttb may be used locally and on the web without modification, and require NO support files (unlike itar bundles, which need a seperate index)
  • ttb bundles are compressed, and are thus ~2x smaller than itar bundles
  • ttb bundles are easily extendable (new features may be added while maintaining backwards compatibility)
  • ttb bundles allow for proper file searching, which resolves a few issues we've had in the past. This will also make the "font-by-name" problem easier to resolve.
  • The changes here REQUIRE the changes in Bundle rework tectonic#1121.

What still needs to be done (probably in other PRs):

• Bundle tests
• Search order tweaks. The current order is a bit simpler than the one in kpathsea, but works with all the files I've tested. Do we need to add any more search paths?
  I think this will resolve itself with time---issues will come in as users compile their documents with the new bundle format.

[rm-dr]

I've just merged the proper-search branch into rework. This includes a LOT of changes, which are listed in this PR's description. The bundle hashes are also up-to-date, you should be able to build a ttb bundle with the exact same content as the one on my computer.

tectonic-typesetting/tectonic#1121 has also been updated, and the rework there is nearly complete. The new bundle format REQUIRES that PR to work.

The old rework branch is available under the old-rework tag, just in case.

[pkgw]

OK! I am finally, finally sitting down to take a look at this work. I like a lot of what I see and I am inclined to more or less YOLO and merge this PR. Thank you for putting the time in here, and my apologies for not being responsive to your effort.

Since it's still in draft mode, do you plan any more work on this PR? I would like to make some changes to improve the build instructions a bit and tidy up some of the Markdown coding (I'm old-fashioned, I guess, because I like consistent hard linebreaks and vertical spacing), but I'm not envisioning any major interventions.

For what it's worth, the setup for the "indexed tar" bundle to use two separate files was intentional; that way with on HTTP request you can pull down the entire index, as opposed to having to make one request to fetch a header, discover the index size, and then make another to get the actual index data. But that's a pretty small benefit, and it probably doesn't outweigh the annoyance of having to carry two files around.

[rm-dr]

tectonic-typesetting/tectonic#1121 isn't ready yet, but this is.
If we don't plan to release any more old-format bundles, this should be merged.

[pkgw]

OK, let's merge. I can work on documentation/etc. improvements in another PR. You should now have permissions to do the merge yourself, correct?

CC @CraftSpider for visibility.

[rm-dr]

You should now have permissions to do the merge yourself, correct?

Yes indeed.