[Bug] Intermittent CI failure on AMD - test_snapshot_basic

[alindima]

Describe the bug

The AMD CI intermittently fails on this test: integration_tests/functional/test_snapshot_basic.py::test_5_full_snapshots
Usually, a retrigger is enough to make the test pass.

This doesn't reproduce on the other CI machines.

Example: https://buildkite.com/firecracker/firecracker-ci-amd/builds/1189#69242717-a6cd-44b2-b4d5-fb925b6144a7

To Reproduce

Cannot be reproduced deterministically (AFAIK).

Expected behaviour

The CI tests should pass.

[sandreim]

Reproduced locally, with more data.

16:56:07 snapshot_sequence: INFO Load snapshot #1, mem /srv/jailer/firecracker/dfc9bd19-a881-4d6e-9515-81b23ae9736d/root/snapshot/vm.mem
16:56:07 retry.api: WARNING [Errno 2] No such file or directory: '/srv/jailer/firecracker/d914647e-dbe2-4e5c-b768-1ba8a75524bc/root/run/firecracker.socket', retrying in 0.2 seconds...
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: retry: Resource temporarily unavailable
bash: fork: Interrupted system call


  FAILED    integration_tests/functional/test_snapshot_basic.py::test_5_full_snapshots

[sandreim]

This could be caused by some resource limit - ulimit -a

[dianpopa]

Fixed by #2503