Skip to content

Latest commit

 

History

History
92 lines (85 loc) · 5.42 KB

2022_10_10.md

File metadata and controls

92 lines (85 loc) · 5.42 KB
Raw

Oxide and Friends Twitter Space: October 10th, 2022

Holistic Boot

We've been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it's not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for October 10th, 2022.

In addition to Bryan Cantrill and Adam Leventhal, speakers on October 10th included Tom Lyon, Dave, Steve Klabnik, Joshua Clulow, Matt Campbell, Ian Rountree, and Todd. (Did we miss your name and/or get it wrong? Drop a PR!)

Some of the topics we hit on, in the order that we hit them:

  • Topic
  • [@M:SS](link into recording)
  • @0:25 Bryan's OSFC talk
  • Talk made it to the top of Hacker News
  • Firmware-first error handling
  • At Joyent, machines were dying with uncorrectable memory errors, and reporting 0 correctable errors, because the firmware-first model was hiding them
  • @7:54 Redfish boot order works like 60% of the time
  • Redfish thing to let you control boot order
  • People have internalized this is the way things have always been and must always be
  • Standards are incredibly wide, poorly specified
  • Tom's take: people don't stick to standards, and the problem with Redfish is that it's just a syntax
  • Everything being a subset or a superset of a standard is no standard at all
  • Conway's law
  • You can see from the API spec what the org chart is
  • @17:30 Getting away from the BMC
  • You want this to be well controlled, and a web interface probably isn't the right thing
  • Needing DRAM to start the BMC, which means DIMM training...
  • Elimination of proprietary BIOS that executes before the bootloader, unseen
  • Industry problem: There's an unfortunate co-dependency between microprocessor vendors and BIOS writers
  • This leads to a fragility, because you get it working, and don't touch it, and don't write anything down.
  • Low levels of firmware knowing things are very wrong, and soldiering on to make do with what it can find: 3 GHz CPU pinned at 800 MHz, half the memory bus failed, so just use the other half...
  • Consumer electronics heritage - Going to great lengths to ensure you don't get a support call
  • Vendors responding "this is only a you problem, we're not hearing this from anyone else"
  • @32:00
  • The curse of vertical integration
  • The fastest way to deflect blame is to find something different about what you were doing
  • Richmond 16 (how many bug numbers do you know)
  • software is running that has ruined the OS's memory
  • @37:15 SMM - System Management Mode
  • At any time, for any duration, for many reasons, the CPU can enter and do whatever it wants, and there's no way for the OS to know it happened
  • Originally for laptop suspend and resume without OS support
  • The amount of software in there is confounding (a mouse driver?!)
  • @52:17 Question from Todd - How much does your design constrain you from using different hardware?
  • Big lift to go to Intel x86 from AMD
  • Re: holistic boot, the host CPU is the most affected piece of hardware, because the integration is tight
  • Would still be a lot of work to go to a different host even if they didn't have a holistic boot, but they'd have very little insight into the underworkings
  • Todd's experience - a 17,000 GPU system in a DC for hosting thousands of nodes. Boot time is hours, bought 5 years out.
  • @59:10 Open source software is a big deal
  • Open source software is important. It's staggering to think of doing this without open source
  • Rust is culmination of many different layers of open source
  • Platform enablement needs to be open source, can't do with with the way Nvidia operates now
  • @1:04:05 This station is now under computer control
  • Obscure Star Trek III reference
  • @1:12:49 The hubris kernel as a development environment
  • can sleep, debuggable
  • typical kernel is very constrained
  • we enter in full 64-bit mode with virtual memory enabled
  • pico host boot loader
  • include_bytes makes testing easy
  • elfwrap
  • @1:27:50 Question - Have you considered running a server with L3 and not RAM
  • Memory comes to the OS trained, it just takes a minute
  • May be some workloads where it makes sense - Finance, but likely need more than a GB

If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We'd love to have you join us, as we always love to hear from new speakers!