e1000e: probe of 0000:00:19:0 failed with error -3 on x230 (when gbe.bin is removed)

[tlaurion]

Can someone replicate

insmod /lib/modules/e1000e.ko
dmesg
e1000e: probe of 0000:00:19:0 failed with error -3

@osresearch @flammit ?

Tried upgrading to latest stable kernel, same thing. Coreboot patches failing with new musl-cross-make? Attempted to put muslc-cross-make to latest commit id without more luck.

Attempts here

[paulmenzel]

@tlaurion, please, you are missing quite a lot of needed information in your report. What Linux version (uname -a)? Both Linux kernels? Please attach the coreboot log and Linux logs.

When was the last time this worked? Does the same Linux kernel work with older coreboot? Does it work with upstream coreboot (and GRUB, SeaBIOS, or … as payload)?

The C library shouldn’t have anything to do with coreboot or the Linux kernel.

Do you have debug on the Linux kernel command line?

You can trace e1000_probe, in drivers/net/ethernet/intel/e1000e/netdev.c.

In your original report the return code was -9. Now it’s -3. Was that a typo or does it change every time?

[tlaurion]

@paulmenzel

In your original report the return code was -9. Now it’s -3. Was that a typo or does it change every time?
It was -3 and a corrected typo.

I arrive to the same error with built kernels. Always with coreboot 4.8.1:
linux-4.14.62
linux-4.19.63
linux-4.19.114
linux-5.4.30

I will make cleaner builds to receive proper help here once I figure out how to not strip modules from symbols so you guys can jump in easily and troubleshoot directly from qemu (and give insights #516 #701 ).

Will also add debug from coreboot kernel options for qemu-coreboot in the following builds and report in future comments.

[tlaurion]

@paulmenzel :

Downloadable qemu-coreboot debug artifacts

So I'm currently building qemu-coreboot debug builds for kernels:
4.14.62 (master used kernel for reference) : Gitlab CircleCi
4.19.63 : Gitlab CircleCi
5.4.30: Gitlab CircleCi

relevant modules changes to make those builds:

diff --git a/modules/linux b/modules/linux
index f881793..f90a1f1 100644
--- a/modules/linux
+++ b/modules/linux
@@ -1,6 +1,6 @@
 modules-y += linux
 
-linux_version := 4.14.62
+linux_version := 5.4.30
 linux_base_dir := linux-$(linux_version)
 
 # TODO: fixup the patch process
@@ -13,12 +13,15 @@ linux_kconfig := $(or $(CONFIG_LINUX_CONFIG),config/linux.config)
 linux_dir := $(linux_base_dir)/$(notdir $(basename $(linux_kconfig)))
 
 linux_tar := linux-$(linux_version).tar.xz
-linux_url := https://cdn.kernel.org/pub/linux/kernel/v4.x/$(linux_tar)
+#linux_url := https://cdn.kernel.org/pub/linux/kernel/v4.x/$(linux_tar)
+linux_url := https://cdn.kernel.org/pub/linux/kernel/v5.x/$(linux_tar)
 
 linux-4.9.38_hash := 76d789d87dd51d2fd58c095727171984fa4a992f5e25b9e3eb1e5fd5cd129074
 linux-4.9.80_hash := 9e2e83ccc0afc3f23340ed5e58a35d8c6300a7c58aa98ca913848de41226477b
 linux-4.14.62_hash := 51ca4d7e8ee156dc0f19bc7768915cfae41dbb0b4f251e4fa8b178c5674c22ab
-
+linux-4.19.63_hash := 75988760b931864b46292dcfad1dd54b3f4da10168041f48ca6d7f6dd4e5d25d
+linux-4.19.114_hash := 1e40a0dc6afc95a259f97b80d5f5ef8f89e2ee49e993ba6844e2bc55de361f0e
+linux-5.4.30_hash := 11dd78f701bce619d90d3b2ee597601716c48087e159c890c1decd7b90349def
 linux_hash := $(linux-$(linux_version)_hash)
 
 # Ensure that touching the config file will force a reconfig/rebuild
@@ -135,10 +151,17 @@ $(module_initrd_lib_dir)/$(notdir $1): $(build)/$(linux_dir)/$1
 	$(call do,INSTALL-MODULE,$1, \
 		$(CROSS)strip \
 			--preserve-dates \
-			--strip-debug \
 			-o "$$@" \
 			"$$<" \
 	)
+#$(module_initrd_lib_dir)/$(notdir $1): $(build)/$(linux_dir)/$1
+#	$(call do,INSTALL-MODULE,$1, \
+#		$(CROSS)strip \
+#			--preserve-dates \
+#			--strip-debug \
+#			-o "$$@" \
+#			"$$<" \
+#	)
 endef
 
 $(call map,linux_module,$(linux_modules-y))

Relevant qemu-coreboot board config change:

diff --git a/boards/qemu-coreboot/qemu-coreboot.config b/boards/qemu-coreboot/qemu-coreboot.config
index aeb57c7..a16256b 100644
--- a/boards/qemu-coreboot/qemu-coreboot.config
+++ b/boards/qemu-coreboot/qemu-coreboot.config
@@ -39,6 +39,8 @@ CONFIG_LINUX_AHCI=y
 CONFIG_LINUX_USB=y
 CONFIG_LINUX_E1000=y
 
+export CONFIG_BOOT_KERNEL_ADD="debug"
+
 #Uncomment only one BOOTSCRIPT:
 #Whiptail-based init (text-based or FBWhiptail)
 #export CONFIG_BOOTSCRIPT=/bin/gui-init

Else, let's remember that those linux and coreboot patches are applied prior of compilation.

To debug with QEMU

To debug, download artifact, extract, and run qemu against rom per board instruction:

qemu-system-x86_64 \
		--machine q35 \
		--serial /dev/tty \
		--bios /path/to/downloaded/artifact/coreboot.rom \
	; stty sane

From qemu Heads,go to recovery shell:

insmod /lib/modules/e1000.ko
dmesg

no eth0

Edit: Let's also remember that migrating from musl-cross to musl-cross-make seems responsible for :
#665
#571 (#571 (comment))
(Here is the version of cross-musl-make used under Heads. I tried to build with latest musl-cross-make, removing packed warning as error with same result.

As far as Heads code change, neither coreboot nor linux code changed but to fit musl-cross-make changes from this commit up to now. The building toolset changed from musl-cross to musl-croos-pin to musl-cross-make.

@paulmenzel @MrChromebox @Jan32 @kylerankin @osresearch @flammit : any help and insight welcome. This behavior is observable on qemu-coreboot and on x230 for what I can tell.

Edit: generic build recipes here:
CircleCI Gitlab

[orangecms]

Have you tried this?

https://github.com/orangecms/webboot/blob/boot-via-qemu/run-webboot.sh#L12

[tlaurion]

@orangecms: this is to replicate a problem that is occuring on real hardware: the network devices (covered by e1000e and e1000 drivers) are resulting in probing error. Not sure how this would resolve both qemu and hardware seen error from heads latest commit.

[tlaurion]

Complementary image:

[tlaurion]

Have you tried this?

https://github.com/orangecms/webboot/blob/boot-via-qemu/run-webboot.sh#L12

I infirm the problem presence under qemu image as per @orangecms advice:
-netdev user,id=u1 -device e1000,netdev=u1 permits the device to be properly detected under qemu by mapping it to host device. #516

[tlaurion]

Complementary image:

@osresearch @orangecms @paulmenzel :
Same kernel, same coreboot. Different results on hardware then on qemu.
The x230 is having that error where probing the ethernet card (while insmoding).
For info, the last time I was able to test network was on musl-cross (as opposed to musl-cross-make), where neither coreboot, patches applied nor kernel used changed.

Any insight welcome....
Artifacts for x230 resplications here

[tlaurion]

@paulmenzel @osresearch

Here are the logs from cbmem, lspci and dmesg from unmodified current master CI's x230 board artifact:

x230-master.cbmem_console.txt
x230-master.dmesg_after_e1000e_insmod.txt
x230-master.lspci_verbose.txt

[tlaurion]

@paulmenzel @osresearch

Here are the logs from cbmem, lspci and dmesg from modified master CIs (x230-debug) branch to have enough space to include qemu-coreboot's debug options inside of x230 board. Artifact

Same result: