Restore on X86

[blackgeorge-boom]

Currently, restore on x86 fails with:

sudo ~/TransProc/criu-3.15/criu/criu restore -vvv --shell-job

...
(00.015607)  45496: Found 12 VMAs in image
(00.015611)  45496: vma 0x4f0000 0x4f1000
(00.015612)  45496: vma 0x500000 0x506000
(00.015613)  45496: vma 0x600000 0x602000
(00.015614)  45496: vma 0x700000 0x702000
(00.015616)  45496: vma 0x800000 0x802000
(00.015617)  45496: vma 0x900000 0x901000
(00.015618)  45496: vma 0xa00000 0xa01000
(00.015620)  45496: vma 0xa01000 0xa03000
(00.015621)  45496: vma 0x7ffff7ffb000 0x7ffff7ffe000
(00.015622)  45496: vma 0x7ffff7ffe000 0x7ffff7fff000
(00.015624)  45496: vma 0xffffde6bf000 0xffffde6e0000
(00.015625)  45496: vma 0xffffffffff600000 0xffffffffff601000
(00.015777)  45496: Collect fdinfo pid=45496 fd=0 id=0x2
(00.015783)  45496: Collect fdinfo pid=45496 fd=1 id=0x4
(00.015784)  45496: Collect fdinfo pid=45496 fd=2 id=0x2
(00.016023)  45496: skqueue: Preparing SCMs
(00.016028)  45496: tty: Unpaired slave 0
(00.016031)  45496: tty: ctl tty leader 0x2
(00.016032)  45496: tty: Inherit terminal for id 0x2
(00.016034)  45496: tty: head driver pts id 0x2 index 0 (master 0 sid 37888 pgrp 45632 inherit 1)
(00.016036)  45496: tty: Found orphan slave fake leader (0x2)
(00.016038)  45496: unix: ghost: Resolving addresses
(00.016040)  45496: File descs:
(00.016041)  45496:  `- type 1 ID 0x1
(00.016042)  45496:  `- type 11 ID 0x2
(00.016043)  45496:    `- FD 0 pid 45496
(00.016045)  45496:    `- FD 2 pid 45496
(00.016046)  45496:  `- type 1 ID 0x3
(00.016047)  45496:  `- type 1 ID 0x4
(00.016048)  45496:    `- FD 1 pid 45496
(00.016049)  45496:  `- type 1 ID 0x5
(00.016050)  45496:  `- type 1 ID 0x6
(00.016349)  45496: Opened local page read 1 (parent 0)
(00.016353)  45496: Enqueue page-read
(00.016354)  45496: Enqueue page-read
(00.016355)  45496: Enqueue page-read
(00.016356)  45496: Enqueue page-read
(00.016357)  45496: Enqueue page-read
(00.016358)  45496: Error (criu/mem.c:1092): Trying to restore page for non-private VMA
(00.016359)  45496: Error (criu/mem.c:1221): Page entry address ffffde6dc000 outside of VMA ffffde6bf000-ffffde6e0000
(00.016393) Error (criu/cr-restore.c:2483): Restoring FAILED.

It seems like a similar issue to this: checkpoint-restore/criu#706, although there isn't a clear solution in the thread.

Is restore on x86 possible, from a recoded aarch64 image, or the differences in the vma addresses cause issues?

[blackgeorge-boom]

By hardcoding a different TASK_SIZE in criu-3.15/compel/arch/x86/src/lib/infect.c, e.g.,

#define TASK_SIZE      ((1UL << 49) - PAGE_SIZE)

I can surpass this error, but it stumbles later on a similar problem:

sudo /home/ubuntu/TransProc/criu-3.15/criu/criu restore -vvv --shell-job

...
(00.017310)  45496: Parsed 7ffccc788000-7ffccc7a9000 vma
(00.017314)  45496: Parsed 7ffccc7e6000-7ffccc7e9000 vma
(00.017318)  45496: Parsed 7ffccc7e6000-7ffccc7ea000 vma
(00.017322)  45496: Parsed ffffffffff600000-ffffffffff601000 vma
(00.017330)  45496: 1 threads require 100K of memory
(00.017335)  45496: Found bootstrap VMA hint at: 0x10000 (needs ~116K)
(00.017397)  45496:     call mremap(0x7fa27ffce000, 8192, 8192, MAYMOVE | FIXED, 0x25000)
(00.017415)  45496:     call mremap(0x7fa27ffd0000, 8192, 8192, MAYMOVE | FIXED, 0x27000)
(00.017438)  45496: x86: XFEATURE_OPMASK is not present in image, ignore
(00.017443)  45496: x86: XFEATURE_ZMM_Hi256 is not present in image, ignore
(00.017447)  45496: x86: XFEATURE_Hi16_ZMM is not present in image, ignore
(00.017451)  45496: x86: XFEATURE_PKRU is not present in image, ignore
(00.017454)  45496: x86: xsave runtime structure
(00.017457)  45496: x86: -----------------------
(00.017461)  45496: x86: cwd:0 swd:0 twd:0 fop:0 mxcsr:0x1f80 mxcsr_mask:0xffff
(00.017466)  45496: x86: magic1:0x46505853 extended_size:2700 xstate_bv:0x2f7 xstate_size:2696
(00.017470)  45496: x86: xstate_bv: 0x2f7
(00.017474)  45496: x86: -----------------------
(00.017477)  45496: Thread    0 stack  0x1a080 rt_sigframe  0x22080
(00.017509)  45496: Going to chroot into /proc/self/fd/8
(00.017517)  45496: Restoring umask to 2
(00.017545)  45496: task_args: 0x24000
task_args->pid: 45496
task_args->nr_threads: 1
task_args->clone_restore_fn: 0x11d30
task_args->thread_args: 0x24540
(00.017558) pie: 45496: Switched to the restorer 45496
(00.017654) pie: 45496: Error (criu/pie/restorer.c:1175): Unable to unmap (0x7fa27fcf6000-0xfffffffff000): -22
(00.017674) pie: 45496: Error (criu/pie/restorer.c:1972): Restorer fail 45496
(00.018085) Error (criu/cr-restore.c:1560): 45496 exited, status=1
(00.018114) Error (criu/cr-restore.c:2483): Restoring FAILED.

Have you faced this issue before? Is it solvable or can I take it up somehow?

Thanks a lot for your time.

[xjtuwxg]

Thanks for pointing it out! I remember the issue we had was if we start from the ARM side, the VMA layout will have issues to the x86 side.

So we just launch the program on x86 side and migrate to the ARM machine (we should be able to migrate back to x86).

By hardcoding a different TASK_SIZE in criu-3.15/compel/arch/x86/src/lib/infect.c, e.g.,

#define TASK_SIZE      ((1UL << 49) - PAGE_SIZE)

I can surpass this error, but it stumbles later on a similar problem:

Have you faced this issue before? Is it solvable or can I take it up somehow?

Thanks a lot for your time.

[blackgeorge-boom]

Thanks for pointing it out! I remember the issue we had was if we start from the ARM side, the VMA layout will have issues to the x86 side.

So we just launch the program on x86 side and migrate to the ARM machine (we should be able to migrate back to x86).

By hardcoding a different TASK_SIZE in criu-3.15/compel/arch/x86/src/lib/infect.c, e.g.,

#define TASK_SIZE      ((1UL << 49) - PAGE_SIZE)

I can surpass this error, but it stumbles later on a similar problem:

Have you faced this issue before? Is it solvable or can I take it up somehow?
Thanks a lot for your time.

Aaa I see, that makes sense!

I will try it and get back to you.

[blackgeorge-boom]

So, I launched the program on X86, migrated to ARM, and tried to migrate back to X86.

I'm getting this error:

...
(00.023310)  78553: Parsed 7f0614a4b000-7f0614a52000 vma
(00.023320)  78553: Parsed 7f0614a4b000-7f0614a75000 vma
(00.023330)  78553: Parsed 7f0614a4b000-7f0614a7d000 vma
(00.023415)  78553: Parsed 7f0614a4b000-7f0614a7e000 vma
(00.023431)  78553: Parsed 7f0614a4b000-7f0614a7f000 vma
(00.023442)  78553: Parsed 7f0614a4b000-7f0614a80000 vma
(00.023452)  78553: Parsed 7f0614a4b000-7f0614a81000 vma
(00.023463)  78553: Parsed 7ffd1f9dc000-7ffd1f9fd000 vma
(00.023473)  78553: Parsed 7ffd1fa41000-7ffd1fa44000 vma
(00.023483)  78553: Parsed 7ffd1fa41000-7ffd1fa45000 vma
(00.023494)  78553: Parsed ffffffffff600000-ffffffffff601000 vma
(00.023511)  78553: 1 threads require 100K of memory
(00.023601)  78553: Found bootstrap VMA hint at: 0x10000 (needs ~116K)
(00.023680)  78553:     call mremap(0x7f0614a4b000, 8192, 8192, MAYMOVE | FIXED, 0x25000)
(00.023789)  78553:     call mremap(0x7f0614a4d000, 8192, 8192, MAYMOVE | FIXED, 0x27000)
(00.023824)  78553: x86: XFEATURE_OPMASK is not present in image, ignore
(00.023837)  78553: x86: XFEATURE_ZMM_Hi256 is not present in image, ignore
(00.023920)  78553: x86: XFEATURE_Hi16_ZMM is not present in image, ignore
(00.023931)  78553: x86: XFEATURE_PKRU is not present in image, ignore
(00.023941)  78553: x86: xsave runtime structure
(00.023951)  78553: x86: -----------------------
(00.023961)  78553: x86: cwd:0 swd:0 twd:0 fop:0 mxcsr:0x1f80 mxcsr_mask:0xffff
(00.023971)  78553: x86: magic1:0x46505853 extended_size:2700 xstate_bv:0x2f7 xstate_size:2696
(00.023982)  78553: x86: xstate_bv: 0x2f7
(00.023992)  78553: x86: -----------------------
(00.024002)  78553: Thread    0 stack  0x1a080 rt_sigframe  0x22080
(00.024059)  78553: Going to chroot into /proc/self/fd/8
(00.024163)  78553: Restoring umask to 2
(00.024202)  78553: task_args: 0x24000
task_args->pid: 78553
task_args->nr_threads: 1
task_args->clone_restore_fn: 0x11d30
task_args->thread_args: 0x24540
(00.024314) pie: 78553: Switched to the restorer 78553
(00.024398) pie: 78553: Error (criu/pie/restorer.c:1175): Unable to unmap (0x7f0614773000-0xfffffffff000): -22
(00.025013) Error (criu/cr-restore.c:2483): Restoring FAILED.

which is similar to the output of the comment above where we hardcode a different TASK_SIZE.

Would you have any suggestions on this?

Thank you for your time.

[xjtuwxg]

Hmm, thanks for letting me know! Which application did you use for testing?

So, I launched the program on X86, migrated to ARM, and tried to migrate back to X86.

I'm getting this error:
... ...
(00.024314) pie: 78553: Switched to the restorer 78553
(00.024398) pie: 78553: Error (criu/pie/restorer.c:1175): Unable to unmap (0x7f0614773000-0xfffffffff000): -22
(00.025013) Error (criu/cr-restore.c:2483): Restoring FAILED.

which is similar to the output of the comment above where we hardcode a different `TASK_SIZE`.

Would you have any suggestions on this?

Thank you for your time.

[blackgeorge-boom]

Hmm, thanks for letting me know! Which application did you use for testing?

I'm using the simple example in test/loop (already provided in the repo as a binary).

[xjtuwxg]

Hmm, thanks for letting me know! Which application did you use for testing?

I'm using the simple example in test/loop (already provided in the repo as a binary).

Hmm, @abhishekbapat, do you have some time to have a quick check/answer?

[blackgeorge-boom]

Hmm, thanks for letting me know! Which application did you use for testing?

I'm using the simple example in test/loop (already provided in the repo as a binary).

Hmm, @abhishekbapat, do you have some time to have a quick check/answer?

It would be great if we got some advice on this!