To avoid hitting a Mutex for every event being recorded, and assuming that all the other streams have (much) lower frequency, we could keep the current format, while recording it differently:

• after the file header, and after any non-events page, start an events page (i.e. leave enough space for a page header)
• writing events happens ("atomically"), just like writing whole pages
  • requires the backing storage to be a mmap'd file, we don't want to actually hit a syscall here
    • we also need this for the ability to track what position we are at (using an AtomicUsize)
  • should work as we're (i.e. the atomic position is) effectively always inside an "open" events page
• writing a non-events page (as well as finishing the file) has to write the correct length in the "currently open" events page, for which we'd have to:
  • first grab a global (per backing store) lock, to avoid page-writing races with other non-events streams
  • claim the range for the new page to write using fetch_add on the same AtomicUsize position (that writing events uses), effectively "closing" the "currently open" events page (new events would effectively end up in a newer page)
  • use the "start of the events page" (from the global lock), and the "start of the new non-events page" (claimed above) to compute the correct length to use in the header of the just-closed events page, and to write that header
  • update the "start of the events page" (in the global lock) to just after the end of the newly claimed non-events page
  • release the lock and start writing the new page (in the claimed range)
    • nothing else will access that range so there's no need to keep the lock any longer

Sadly bypassing File entirely will probably be most of the work here, I don't fully understand why measureme never properly mmap'd a file-on-disk (the pre-paging mmaps were "anonymous", i.e. no different than ungrowable Vec<u8>s).

I'm also not sure how we'd handle not being able to grow the mmap'd region (not without re-adding locking to writing events), we need to be on a 64-bit host to even grab gratuitous amounts of virtual memory, without hampering the rest of the process, don't we?
Maybe for 32-bit hosts we can keep paging events and writing to a File?

cc @michaelwoerister @wesleywiser