docs.rs gives a 500 on /, but only in prod

[jyn514]

Backtrace

Sep 14 19:37:13 docsrs cratesfyi[1524]: thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', src/storage/s3.rs:124:48
Sep 14 19:37:13 docsrs cratesfyi[1524]: stack backtrace:
Sep 14 19:37:13 docsrs cratesfyi[1524]:    0: backtrace::backtrace::libunwind::trace
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
Sep 14 19:37:13 docsrs cratesfyi[1524]:    1: backtrace::backtrace::trace_unsynchronized
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
Sep 14 19:37:13 docsrs cratesfyi[1524]:    2: std::sys_common::backtrace::_print_fmt
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/sys_common/backtrace.rs:78
Sep 14 19:37:13 docsrs cratesfyi[1524]:    3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/sys_common/backtrace.rs:59
Sep 14 19:37:13 docsrs cratesfyi[1524]:    4: core::fmt::write
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libcore/fmt/mod.rs:1076
Sep 14 19:37:13 docsrs cratesfyi[1524]:    5: std::io::Write::write_fmt
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/io/mod.rs:1537
Sep 14 19:37:13 docsrs cratesfyi[1524]:    6: std::sys_common::backtrace::_print
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/sys_common/backtrace.rs:62
Sep 14 19:37:13 docsrs cratesfyi[1524]:    7: std::sys_common::backtrace::print
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/sys_common/backtrace.rs:49
Sep 14 19:37:13 docsrs cratesfyi[1524]:    8: std::panicking::default_hook::{{closure}}
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/panicking.rs:198
Sep 14 19:37:13 docsrs cratesfyi[1524]:    9: std::panicking::default_hook
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/panicking.rs:218
Sep 14 19:37:13 docsrs cratesfyi[1524]:   10: std::panicking::rust_panic_with_hook
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/panicking.rs:486
Sep 14 19:37:13 docsrs cratesfyi[1524]:   11: rust_begin_unwind
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libstd/panicking.rs:388
Sep 14 19:37:13 docsrs cratesfyi[1524]:   12: core::panicking::panic_fmt
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libcore/panicking.rs:101
Sep 14 19:37:13 docsrs cratesfyi[1524]:   13: core::panicking::panic
Sep 14 19:37:13 docsrs cratesfyi[1524]:              at src/libcore/panicking.rs:56
Sep 14 19:37:13 docsrs cratesfyi[1524]:   14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
Sep 14 19:37:13 docsrs cratesfyi[1524]:   15: std::thread::local::LocalKey<T>::with
Sep 14 19:37:13 docsrs cratesfyi[1524]:   16: tokio::runtime::enter::Enter::block_on
Sep 14 19:37:13 docsrs cratesfyi[1524]:   17: tokio::runtime::context::enter
Sep 14 19:37:13 docsrs cratesfyi[1524]:   18: tokio::runtime::handle::Handle::block_on
Sep 14 19:37:13 docsrs cratesfyi[1524]:   19: docs_rs::storage::Storage::get
Sep 14 19:37:13 docsrs cratesfyi[1524]:   20: <docs_rs::web::file::DatabaseFileHandler as iron::middleware::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   21: <alloc::boxed::Box<dyn iron::middleware::Handler> as iron::middleware::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   22: <docs_rs::web::routes::BlockBlacklistedPrefixes as iron::middleware::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   23: <alloc::boxed::Box<dyn iron::middleware::Handler> as iron::middleware::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   24: <docs_rs::web::CratesfyiHandler as iron::middleware::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   25: <iron::iron::RawHandler<H> as hyper::server::Handler>::handle
Sep 14 19:37:13 docsrs cratesfyi[1524]:   26: hyper::server::Worker<H>::handle_connection
Sep 14 19:37:13 docsrs cratesfyi[1524]:   27: hyper::server::listener::spawn_with::{{closure}}

The error comes from

docs.rs/src/storage/s3.rs

Line 124 in a2ab0d3

let date_updated = parse_timespec(&res.last_modified.unwrap())?;

which calls .unwrap() on the last_modified_date. But this makes no sense: that file hasn't been updated in weeks, and neither has the rest of the storage code.

The clue is here:

docs.rs/src/storage/s3.rs

Lines 311 to 313 in a2ab0d3

	// NOTE: trying to upload a file ending with `/` will behave differently in test and prod.
	// NOTE: On s3, it will succeed and create a file called `/`.
	// NOTE: On min.io, it will fail with 'Object name contains unsupported characters.'

What's happening is that locally min.io gives a 404 and it delegates to the router, no problem. But in prod, AWS gives a directory listing instead, which has no last_modified date.

The solution should be to skip the database for any URL ending in /; only index.html should be normalized and served.

This regression was introduced in https://github.com/rust-lang/docs.rs/pull/1043/files#diff-0aa36516a2a42b271d0c346c642fb5eaR181, which went through the storage backend for / before looking at the router (cc @robinhundt, but I'll take care of fixing it). This is making me rethink that change: we should only look at S3 after we checked it's not a recognized route.

[jyn514]

Fixed by #1052