Tracking Issue for Managing Low Disk Space Conditions

[smklein]

Spawned from RFD 312

• Tracking the right info in the DB
  • Store the "maximum allowed size" of a dataset
  • Add constraints where appropriate to limit this size (e.g., datasets < zpool size, regions < dataset size)
• Quotas & Reservations
  • (Nexus) Determine what Quotas & Reservations should be for all known datasets
  • (Nexus) Request that quotas/reservations are set in a request to the Sled Agent
  • (Sled Agent) Actually apply Quotas & Reservations
  • (Nexus) Request a "ballast" for reserved capacity explicitly
• Scaling
  • (Sled Agent) Add metric to periodically monitor space usage
  • (Nexus) Periodically query space usage of datasets, and take actions at certain thresholds...
    • ... When expected, perform horizontal scaling
    • ... Identify low-space conditions (long-term, we should be providing an alert to operators if we cannot remedy the situation ourselves).

[askfongjojo]

When considering caps and reservations, we probably need to consider DRAM and CPU as well so there is room for VM migration. If the sleds are all nearly full, it will be impossible to evacuate an entire sled.

[smklein]

When considering caps and reservations, we probably need to consider DRAM and CPU as well so there is room for VM migration. If the sleds are all nearly full, it will be impossible to evacuate an entire sled.

Agreed, #2299 is my first attempt to capture this information