Idea: store unrendered READMEs on s3

[carols10cents]

This is another idea for speeding up README rerendering, similar to #1025.

Right now, part of the time that the render-readmes script takes is downloading the .crate tarballs from s3 and unpacking them to extract the README files. Every time we rerun the render-readmes script, it redoes all this work, even though the .crate files are immutable, and those crates that were missing READMES are still going to be missing READMEs even if we try again.

What if, instead, we store the raw markdown/text/asciidoc on s3, so that the render-readmes script doesn't have to do this extraction over and over? Instead of iterating through every version of every crate, the script could iterate through just the files it finds on s3.

For new crates, since crate publishing sends the README text separately for search purposes, we could even save the raw README to s3 directly and never have to unpack the .crate file for this purpose.

[jtescher]

@carols10cents this seems to have been implemented in #1745

[smarnach]

@jtescher No, this hasn't been implemented yet. We store the rendered readme files on S3, but not the unrendered ones.

We have a script to re-render all readme files for all versions of all crates ever uploaded. Sometimes changes to the readme rendering code make it desireable to run this script, but it takes a very long time to finish. This ticket is about speeding the script up.

[jtescher]

@smarnach ah I see that makes sense. Happy to help out here, sounds like 1) upload raw file to other s3 path prefix as well as the rendered file in the current way, 2) switch script to iterate through the raw files and rerender/upload to final path.

[smarnach]

I guess it would be a little more complex. For each new upload, we will have to extract the raw readme and upload it to S3, but we also need to go through all past uploads once, download the crates, extract the readme and upload it to S3. Once we have raw versions of all readmes on S3, we can swicht over the render-readmes script.

However, I don't think this ticket is a priority right now. I believe that running the render-readmes script is a rare occurrence these days. Moreover, the team does have limited time at the moment, so I would prefer to use the available review capacities for other tickets – this one requires rather complex changes for rather little immediate benefits.

[est31]

An alternative would be to not extract the entire .crate file in the render-readme script but only extract the two needed files: Cargo.toml and the readme file it references. The files don't even have to touch the hard disk, ever. The tar crate's API supports extracting single files.

[Turbo87]

@est31 I guess the problem is not the extraction, but having to download the full crate file when we only need a small subset from it.

over the past years we've rarely rerendered the readme files though, so I'm not sure how valuable this additional storage would be. we've backfilled other values from the Cargo.toml manifest a couple of times though, so one could argue we should store that in the extracted form as well...

ultimately this would essentially be a cache, but given how rarely we need it and that such backfilling operations are not particularly time-critical I don't think it would be worth the extra complexity.