Repository navigation
atelet: measure what a snapshot image costs, not how long it is - #931
Max Smythe (maxsmythe) merged 3 commits into
Conversation
|
The previous snapshot size metric did not consider the micro-VM case, so this update ensures both scenarios are supported. Please let me know if this approach is correct. Thanks. cc: Benjamin Elder (@BenTheElder) Max Smythe (@maxsmythe) Haven Xia (@HavenXia) |
Krisztian F (krisztianfekete)
left a comment
Krisztian F (krisztianfekete)
left a comment
There was a problem hiding this comment.
Thanks, added a few comments, but looks good overall!
We should probably also wire atelet.snapshot.size to PlatformMetricPrefixes e2e tests, it can also be a follow-up, though.
9996410 to
1dc089d
Compare
Added "atelet_snapshot_size" into e2e PlatformMetricPrefixes. |
|
Thanks Krisztian F (@krisztianfekete) for review. I have resolved comments. Please review again. |
Benjamin Elder (BenTheElder)
left a comment
Benjamin Elder (BenTheElder)
left a comment
There was a problem hiding this comment.
for upload we can just do the compressed size, for pause ... hmm, this is complicated. I don't think the current approach is quite correct either.
thanks for working on this
Bowei Du (bowei)
added
kind/bug
|
Still seeing this on a microVM metal run: every memory-ranges sample is 2147483648, while atelet's upload log shows populated_bytes ≈ 294 MiB for the same files. #1725 now collects memory-ranges in the benchmark telemetry, so this fix makes those sizes correct. +1 to Ben's SEEK_DATA/SEEK_HOLE approach. |
1dc089d to
8653be5
Compare
Micro-VM memory-ranges snapshot files are sparse files whose logical size equals the guest's configured RAM, while ateom skips fsync before handing checkpoint files to atelet so dirty pages remain under ext4 delayed allocation. Walk SEEK_DATA/SEEK_HOLE extents to record uncompressed populated bytes, excluding sparse holes while still counting unwritten delalloc extents in the page cache.
8653be5 to
21396b7
Compare
Record st_blocks*512 (allocatedBytes) in atelet.snapshot.size rather than apparent file size (st_size) so sparse micro-VM memory-ranges images report allocated bytes instead of the guest RAM ceiling. On Linux, ext4/XFS/btrfs include delayed-allocation blocks in st_blocks before writeback, so the existing Lstat/Stat FileInfo is sufficient without extra syscalls.
|
Note: moved back to st_blocks*512 since ext4 (ext4_file_getattr), XFS, and btrfs already include delayed-allocation blocks in st_blocks before writeback, avoiding the extra open and per-extent SEEK_DATA/SEEK_HOLE syscalls on the checkpoint path. |
7 participants
atelet.snapshot.sizereadinfo.Size()for every image it recorded. For the gVisor images that is right, but the micro-VM memory image is sparse: cloud-hypervisor createsmemory-rangesat the guest's full RAM size and writes only the pages the guest ever touched. Its length is therefore a constant — the VM's configured memory — so the metric reported the same number for a guest that had just booted and one that had filled its RAM. On a live counter actor it read 2147483648 for a file that occupies 163577856 bytes: the configured 2048 MiB, every time, on every actor.Record
st_blocks*512instead, which is what the filesystem actually gave the file (including delayed-allocation blocks before writeback on ext4/XFS/btrfs). Sparse or not, that is the answer to "what did this checkpoint cost", and it is the figure the snapshot's transfer time and storage bill follow.The count rounds up to a block, so the small JSON images beside the memory one now over-report by under 4 KiB. That is noise on a histogram whose first bucket boundary is 1 MB, and the gVisor images are dense, so for them the two readings agree to within that rounding.
docs/observability.mdanddocs/metrics/registry/metrics.yamldescribed this metric as the uncompressed size of a gVisor image, which it was back when gVisor was the only sandbox. They now say what is recorded, and why the two sandboxes read differently.