If the cache for builtins can't be loaded, fine-grained will basically always crash trying to create builtins.objects. There are some crashes if it can't loading typing also, maybe other things.
The solution is probably to fall back to not using a cache if the most critical cache files are missing, but I get a handful of test failures where we produce additional errors when I do that, so it will require a bit more work.
#4909 (comment)