Misaligned Pointer Dereference with every "hermit run" #41
Labels
bug
Something isn't working
Comments
|
Seeing the same issue.
|
|
Building with |
|
Digging a bit deeper, the error according to the logs is here: https://github.com/facebookexperimental/reverie/blob/main/safeptrace/src/memory.rs#L171 This is essentially a wrapper for
Now apparently, Rust didn't mind too much about this until 1.70, when this was transformed into an actual alignment failure and this is what we're seeing here. I confirmed this by building Hermit with nightly builds from the past months: |
|
I now have a working build with stable Rust, but it's getting weirder. I tried looking at Reverie because this is where the failure comes from. The Reverie main branch right now is at commit and indeed, when I run And after that Now I checked out Reverie locally on my machine and adjusted the various Cargo.toml files in Hermit to point to my local Reverie checkout: diff --git a/detcore-model/Cargo.toml b/detcore-model/Cargo.toml
index 97bdafc..44796c0 100644
--- a/detcore-model/Cargo.toml
+++ b/detcore-model/Cargo.toml
@@ -13,7 +13,8 @@ chrono = { version = "0.4", features = ["clock", "serde", "std"], default-featur
clap = { version = "3.2.25", features = ["derive", "env", "regex", "unicode", "wrap_help"] }
libc = "0.2.139"
nix = "0.25"
-reverie-syscalls = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie-syscalls = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-syscalls = { path = "../../reverie/reverie-syscalls" }
serde = { version = "1.0.176", features = ["derive", "rc"] }
shell-words = "1.1.0"
tracing = "0.1.35"
diff --git a/detcore/Cargo.toml b/detcore/Cargo.toml
index 02298c7..a8a3f5a 100644
--- a/detcore/Cargo.toml
+++ b/detcore/Cargo.toml
@@ -45,7 +45,8 @@ rand_distr = "0.4"
rand_pcg = { version = "0.3", features = ["serde1"] }
raw-cpuid = "10.6.0"
regex = "1.9.2"
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../reverie/reverie" }
serde = { version = "1.0.176", features = ["derive", "rc"] }
serde_json = { version = "1.0.100", features = ["float_roundtrip", "unbounded_depth"] }
tempfile = "3.5"
@@ -54,6 +55,7 @@ tracing = "0.1.35"
[dev-dependencies]
detcore-testutils = { version = "0.0.0", path = "tests/testutils" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../reverie/reverie-ptrace" }
test-allocator = { version = "0.0.0", path = "../common/test-allocator" }
test-case = "3.1"
diff --git a/detcore/tests/testutils/Cargo.toml b/detcore/tests/testutils/Cargo.toml
index c12419c..abf9d7f 100644
--- a/detcore/tests/testutils/Cargo.toml
+++ b/detcore/tests/testutils/Cargo.toml
@@ -9,8 +9,10 @@ edition = "2021"
detcore = { version = "0.0.0", path = "../.." }
lazy_static = "1.4"
pretty_assertions = { version = "1.2", features = ["alloc"], default-features = false }
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../../../reverie/reverie" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../../../reverie/reverie-ptrace" }
test-allocator = { version = "0.0.0", path = "../../../common/test-allocator" }
tokio = { version = "1.29.1", features = ["full", "test-util", "tracing"] }
tracing = "0.1.35"
diff --git a/hermit-cli/Cargo.toml b/hermit-cli/Cargo.toml
index d218795..227994e 100644
--- a/hermit-cli/Cargo.toml
+++ b/hermit-cli/Cargo.toml
@@ -25,8 +25,10 @@ pretty_assertions = { version = "1.2", features = ["alloc"], default-features =
rand = { version = "0.8", features = ["small_rng"] }
rand_pcg = { version = "0.3", features = ["serde1"] }
regex = "1.9.2"
-reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
-reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+#reverie = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie = { path = "../../reverie/reverie" }
+#reverie-ptrace = { version = "0.1.0", git = "https://github.com/facebookexperimental/reverie.git", branch = "main" }
+reverie-ptrace = { path = "../../reverie/reverie-ptrace" }
serde = { version = "1.0.176", features = ["derive", "rc"] }
serde_json = { version = "1.0.100", features = ["float_roundtrip", "unbounded_depth"] }
shell-words = "1.1.0"The local Reverie checkout is at exactly the same commit as the Github repo. Now, when I run I am utterly at loss what the difference here is. I'm compiling with the exact same version of Reverie as is in Github, just have it checked out locally. I inspected the Cargo.lock file and there is no difference between the two builds, except the fact that the failing build mentions the fact that the Reverie crates were pulled from Github instead of my local disk. I'm currently guessing that for some reason the unalgined memory address now is properly aligned, but honestly, I have no idea what to look at next. |
|
Now staring at the full backtrace. The issue happens early on when pre-initialising the tracee process and patching its VDSO apparently: |
|
Adding some debugging diff --git a/reverie-ptrace/src/vdso.rs b/reverie-ptrace/src/vdso.rs
index 55ad06d..794e14b 100644
--- a/reverie-ptrace/src/vdso.rs
+++ b/reverie-ptrace/src/vdso.rs
@@ -234,10 +234,13 @@ where
)
.await?;
+ println!("Iterating VDSO_PATCH_INFO @ {:x}", vdso.address.0);
for (name, (offset, size, bytes)) in VDSO_PATCH_INFO.iter() {
+ println!(" name {} off {:x} size {}", name, offset, size);
let start = vdso.address.0 + offset;
assert!(bytes.len() <= *size);
let rptr = AddrMut::from_raw(start as usize).unwrap();
+ println!(" rptr: {:?}", rptr);
memory.write_exact(rptr, bytes)?;
assert!(*size >= bytes.len());
if *size > bytes.len() {
diff --git a/safeptrace/src/memory.rs b/safeptrace/src/memory.rs
index 8fe5f70..8ec5350 100644
--- a/safeptrace/src/memory.rs
+++ b/safeptrace/src/memory.rs
@@ -163,6 +163,7 @@ impl MemoryAccess for Stopped {
}
fn write(&mut self, addr: AddrMut<u8>, buf: &[u8]) -> Result<usize, Errno> {
+ println!(" write(addr={:?} buf @ {:?})", addr.cast::<u64>(), buf.as_ptr());
let size = buf.len();
if size == 0 {
return Ok(0);this happens right the first time whereas with a local build: From that we at least know that the buffer triggering the assertion is the |
|
https://github.com/facebookexperimental/reverie/blob/main/reverie-ptrace/src/vdso.rs#L27 has the byte code that is injected into the VDSO. There is even a comment that this code must be 8 byte aligned. Apparently, it is not in all cases. I'll submit a Reverie PR. |
bjoernd
pushed a commit
to bjoernd/reverie
that referenced
this issue
Sep 8, 2023
The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case: facebookexperimental/hermit#41. Use an explicitly aligned data structure to enforce this. Signed-off-by: Bjoern Doebel <bjoern.doeel@gmail.com>
bjoernd
added a commit
to bjoernd/reverie
that referenced
this issue
Sep 11, 2023
The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case: facebookexperimental/hermit#41. Use an explicitly aligned data structure to enforce this. Signed-off-by: Bjoern Doebel <bjoern.doeel@gmail.com>
bjoernd
added a commit
to bjoernd/reverie
that referenced
this issue
Sep 12, 2023
The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case: facebookexperimental/hermit#41. Use an explicitly aligned data structure to enforce this. Signed-off-by: Bjoern Doebel <bjoern.doeel@gmail.com>
facebook-github-bot
pushed a commit
to facebookexperimental/reverie
that referenced
this issue
Sep 12, 2023
Summary: The code that is injected into the tracee processes' VDSO needs to be 8 byte aligned in order to satisfy conditions imposed by the ptrace interface on 64 bit architectures. There have been reproducible reports of that not always being the case: facebookexperimental/hermit#41. Use an explicitly aligned data structure to enforce this. Pull Request resolved: #22 Reviewed By: VladimirMakaev Differential Revision: D49195126 Pulled By: jasonwhite fbshipit-source-id: 8e21cd4306ee2bf75ec321b21e40137be5292f9f
|
I believe this issue is fixed by facebookexperimental/reverie#22 (thanks to @bjoernd). Closing it out. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A clear and concise description of what the bug is.
Any
hermit run <target>fails with a `misaligned pointer dereference: address must be a multiple of 0x8 but is ..."The exact address differs slightly, but it must always be a multiple of 0x8, and is not.
Indicate any of these common scenarios that apply:
To Reproduce
Minimal input to reproduce the behavior.
Expected behavior
A clear and concise description of what you expected to happen.
rand.py should execute with a deterministic string of numbers.
Environment
uname -a):Linux S1030755 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linuxlscpu):Intel(R) Xeon(R) W-2125 CPU @ 4.00GHz/etc/issue,/etc/redhat-release):Ubuntu 22.04.2 LTS \n \lAdditional context
Attach the logs to this issue as a text file generated by
hermit --log=trace --log-file=hermit.log run ./rand.py.hermit.log
The text was updated successfully, but these errors were encountered: