Raspberry Pi | libc-bin segmentation fault when using USB-SD adapter

[lpseguin]

• DietPi version
  G_DIETPI_VERSION_CORE=8
  G_DIETPI_VERSION_SUB=24
  G_DIETPI_VERSION_RC=1
  G_GITBRANCH='master'
  G_GITOWNER='MichaIng'

• Distro version
  bookworm 0

• Kernel version
  Linux DietPi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux

• SBC model
  RPi 4 Model B (aarch64)

• Power supply used
  [5.1V 3.0A, Black, UL Listed]

• SD card used
  SanDisk ultra

• Bug report ID
  20d1f2ae-a76d-4003-888a-051aa39f901a

Steps to reproduce

1. Fresh install of Dietpi v8.25
2. Only config options changed are keyboard local to us, timezone to newyork and hostname changed to DietPi4

Expected behaviour

• No errors during first setup

Actual behaviour

• Getting libc-bin apt upgrade segmentation fault error everytime during first setup phase

Extra details

• Same error was occuring when trying to install DietPi v8.23
  Bug report of that one is : 5d5c4461-2ef7-4e4e-8a66-286367b0279c

[MichaIng]

Hmm, a segmentation fault in a native Debian binary could be a filesystem error. Hopefully not an issue with a package/update shipped by the RPi repo. Does the kernel show some errors?

dmesg -l 0,1,2,3

Reinstalling that package does not help, does it?

apt install --reinstall libc-bin

[lpseguin]

Hi!

Here's the output from the cmd:

 dmesg -l 0,1,2,3
[    0.839513] mmc1: Controller never released inhibit bit(s).

 apt install --reinstall libc-bin
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for libc-bin:arm64

[Joulinar]

Can you share following

Architecture | dpkg --print-architecture

[lpseguin]

Can you share following

Architecture | dpkg --print-architecture

arm64

[MichaIng]

Strange error. Not sure whether this means that some meta file (one of /var/lib/dpkg/info/libc-bin.*) is missing, or something else. Searching around, manually unpacking the package onto the system seems to be a solution for this kind of error, but lets try some more common APT issue solutions first:

/boot/dietpi/func/dietpi-set_software apt clean
apt update
apt install --reinstall libc-bin

[lpseguin]

Thanks for the help.
This is a brand new Pi kit I bought so I'm assuming the hardware is not faulty but not so sure anymore.
But my next step was to install the default Raspberry OS and see if similar errors persisted there.

the reinstall didn't work unfortunately:

(...)
All packages are up to date.
root@DietPi4:~# apt install --reinstall libc-bin
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for libc-bin:arm64

[MichaIng]

No need to test RPi OS. This is not some kind of systematic error, but a random one, like it can happen (in very rare cases) that a download you do with any OS is faulty, because some random bit was wrong, despite error-correction mechanisms. It could be also a bad block on the SD card, but then on RPi OS, it might affect a completely different file, remaining unrecognised or leading to a very different error.

The package is btw indeed shipped by the RPi repository. Originally (from your dietpi-update logs) it crashed with a segmentation fault:

Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up libc-bin (2.36-9+rpt2+deb12u3) ...
Segmentation fault
Segmentation fault
dpkg: error processing package libc-bin (--configure):
 installed libc-bin package post-installation script subprocess returned error exit status 139
Errors were encountered while processing:
 libc-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)

However, this package was last updated this Oktober, so it should be fine, otherwise we would have seen many more reports with this error already. So probably indeed a random download error.

So lets try to manually overwrite the content of this package on your system:

cd /tmp
apt download libc-bin
dpkg -x libc-bin_*.deb /
apt install --reinstall ./libc-bin_*.deb
rm ./libc-bin_*.deb

[lpseguin]

Know of a good way to test the entire SD card ?
I reformatted it maybe 5-6 times with 3 different DietPI OS versions and always failed on the same libc-bin error.
It seems more than likely at this point that this would be an hardware failure or as you said more reports would come in about this.

I still had the same error with the manual download you suggested:

root@DietPi4:/tmp# apt download libc-bin
Get:1 https://archive.raspberrypi.org/debian bookworm/main arm64 libc-bin arm64 2.36-9+rpt2+deb12u3 [531 kB]
Fetched 531 kB in 1s (598 kB/s)

Note, selecting 'libc-bin' instead of './libc-bin_2.36-9+rpt2+deb12u3_arm64.deb'
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for libc-bin:arm64

[MichaIng]

Another report of this with RPi OS on Pi-hole forum, the only other result of this exact error on DuckDuckGo: https://discourse.pi-hole.net/t/pi-hole-not-fully-installing-failing-at-libc-bin/66946

There a simple

dpkg --reconfigure -a

helped. This is the step that failed on the initial package upgrade (according to the part of the log I posted) and as well for the user on the Pi-hole forum. Not sure how it can prevent the package reinstall from even reaching the point where it would run this step, with this strange "No file name for" error, but worth to try.

Google throws out one second match, this time Ubuntu: ros2/ros2#1352
A very rare thing it seems, which can happen with APT packages. Interesting that you are able to replicate this even with different DietPi images.

Rufus can do bad block tests on SD cards. However, for this is needs to overwrite and read back every bit of the SD card. And sometimes it does not even find something on one run, while there are bad blocks for sure. Doing multiple runs not only takes much time but further decreases the life span of the card, so I do not recommend it. As long as you do not see I/O errors in dmesg -l 0,1,2,3, the SD card should be fine. The question remains where this error is coming from, but lets see whether the reconfiguration above helps, rendering the question obsolete for now.

[lpseguin]

Hi again and happy new year!

After coming back home the pi was throwing bin errors and wouldn't even finish booting up in a usable state.
I took the time to flash the default Raspberry 64 bit light OS and it booted fine.
But I got the exact same libc-bin error after doing apt update -> apt upgrade.

Reflashed DietPI and tried out the configure option and it's throwing out the same error as before:

root@DietPi:~# sudo dpkg --configure -a
Setting up libc-bin (2.36-9+rpt2+deb12u3) ...
Segmentation fault
Segmentation fault
dpkg: error processing package libc-bin (--configure):
 installed libc-bin package post-installation script subprocess returned error exit status 139
Errors were encountered while processing:
 libc-bin

Still getting the same Controller never released inhibit bits error message in dmesg.
Same error message popped up at bootup in the default Raspberry OS too.

[Joulinar]

does it mean you have the same error with plain RPI OS as you have on DietPi? If yes, we might need to report to RPI developer.

[lpseguin]

does it mean you have the same error with plain RPI OS as you have on DietPi? If yes, we might need to report to RPI developer.

Yes exact same Segmentation Faults and exit codes

[Joulinar]

@MichaIng any idea if this is something to report to RPI or Debian repository?

[68wooley]

Folks,

I don't have a solution for you, but I wanted to let you know I'm having the same issue with libc-bin trying to install XFS on a Pi 5 running Pi OS Lite (64Bit) (bookworm).

I'll let you know if I find a solution.

[68wooley]

Something worth trying:

I was running via a microSD to USB-A adaptor and booting from USB. I just re-imaged my card and booted from the internal microSD reader and I no longer have the problem.

If you're booting via USB, might be worth trying. Someone else posted the same issue/solution in the RPi forums:

https://forums.raspberrypi.com/viewtopic.php?t=358967#p2172823

[lpseguin]

Something worth trying:

I was running via a microSD to USB-A adaptor and booting from USB. I just re-imaged my card and booted from the internal microSD reader and I no longer have the problem.

If you're booting via USB, might be worth trying. Someone else posted the same issue/solution in the RPi forums:

https://forums.raspberrypi.com/viewtopic.php?t=358967#p2172823

Thanks a lot I was in the same situation and putting the MicroSD card in the internal Pi reader fixed the libc-bin error and the update is now working.

[Joulinar]

Does it just require a SD card present or does the system need to be booted from SD card as well?

[MichaIng]

Very strange. The error sounds more like if the package was compiled with wrong (too new for RPi 4, i.e. RPi 5 only) instruction set. Not good to know it's for whatever reason USB adapters, and indeed an issue with the RPi OS package itself.

[68wooley]

Does it just require a SD card present or does the system need to be booted from SD card as well?

Good question. I'll test this afternoon and let you know.

[68wooley]

OK - here's what I found:

It seems the issue is tied to the use of a microSD to USB-A adaptor:

• Flashing PiOS to an actual USB-A thumb-drive and booting from that had no issues.
• Flashing PiOS to a SSD in a USB-C enclosure and booting from that via a USB-C to USB-A adaptor had no issues.
• Booting from the microSD card in the USB-A adaptor continued to have problems, even with a second microSD card in the built-in reader.
• Flashing a different microSD card and booting from that via the USB-A adaptor resulted in the same problem.

If anyone wants to investigate further, they're welcome to go for it, but the 'solution' of using the internal reader is simple enough I can't see it being worth anyone's time unless whatever the problem is starts manifesting in other ways.

[MichaIng]

Did someone test to use switch to dwc2 driver or disable UAS (for the USB-SD adapter)?

[tanderson1992]

I'm following because I'm encountering the same issue. In my case it's because I messed up the SD reader somehow, either by pushing it in at an angle or perhaps backwards. I can't boot or read from the SD but can from the same card in a USB adapter. I have a new RPi4 on the way (ordered before I discovered the physical damage), but it would be nice to get this board working with the USB adapter. In January you asked if anyone had tested dwc2 or disabling UAS for the USB-SD adapter. If you can point me in the right direction to try that with Dietpi, I'm game. Since the box is solely for pihole, it's not doing me any good right now without being able to install pihole on this new SD card.

[MichaIng]

To disable UAS:

echo 'blacklist uas' > /etc/modprobe.d/disable-uas.conf

To switch to dwc2:

G_CONFIG_INJECT 'dtoverlay=dwc2' 'dtoverlay=dwc2' /boot/config.txt

If you have no other USB drive to boot from and hence no way to actually boot this RPi 4: Do you have another Linux system to access the ext4 filesystem and edit those files there?

Else, with a new image, you could use an automation script: Access the FAT setup partition and create a file named Automation_Custom_PreScript.sh with the content:

#!/bin/dash
echo 'blacklist uas' > /etc/modprobe.d/disable-uas.conf
reboot

respectively:

#!/bin/dash
echo 'dtoverlay=dwc2' >> /boot/config.txt
reboot

or with both commands. Assure to use Unix/LF file endings (instead of Windows/CR LF)! Proper editors should habe the option to change this. Then try to boot it. This script executes at early boot stage, which is hopefully reached. If not, then there is no other way than accessing the ext4 filesystem from another Linux system.

[tanderson1992]

I wish I'd been about 10 minutes more patient. I (obviously not knowing what I'm doing) manually inserted dtoverlay=dwc2 into /boot/config.txt then rebooted. On reboot the pi didn't start at all. I inserted the USB adapter into my other Linux box, and it is accessible there. I didn't know to make the changes in the FAT partition, so I ejected the device and was about to re-image it before I saw your post. Now I'll try to make those changes and see if it works.

[tanderson1992]

I didn't try disabling UAS because a number of posts I read suggested it might not be configured as a module and blacklisting it might not have an effect. In any event the SD card is now not bootable in the Pi although I can read it fine in Linux. I might give up on this broken board and setup the new one that's allegedly arriving today according to Amazon.

[MichaIng]

uas is a module on RPi:

modinfo uas

See the file path, else it would show "builtin".

Neither dwc2 nor disabling UAS should break USB access. The thing with the setup FAT partition was a little misleading, I was mixing it up, as such does not exist on our RPi images, which have a boot FAT partition natively 😄. So a final test would be to edit config.txt on the boot FAT partition and add dtoverlay=dwc2, as well as create /etc/modprobe.d/disable-uas.conf with mentioned blacklist on the ext4 partition, both before booting the image for the first time, to assure both is effective immediately.

But quite possible that the RPi/kernel simply does not support your USB-SD adapter, like OP and the RPi forum thread are indicating as well. Just worth an attempt to switch driver and UAS.

[tanderson1992]

Thanks. I can't get the RPi to boot from USB still, and now I have to go out so can't mess with it anymore tonight. I'll just put the working dietpi card in the new SD slot when the new board arrives. Thanks for the info and assistance.

[realDqm1]

Something worth trying:

I was running via a microSD to USB-A adaptor and booting from USB. I just re-imaged my card and booted from the internal microSD reader and I no longer have the problem.

If you're booting via USB, might be worth trying. Someone else posted the same issue/solution in the RPi forums:

https://forums.raspberrypi.com/viewtopic.php?t=358967#p2172823

Thank you for saving me my time. Yes, same Issue. Package libc-bin error on USB-A MicroSD Adapter. Both Raspberry OS 64 and DietPi latest iso same error. I was afraid that my hardware was broken. Now, just removed the SD Card from Adapter and booted from on-Board SD Card Reader. Everything works. Thanks a lot!!

[jeremysalwen]

Hit the same issue on my raspbian install after switching the card to an SD Card reader (which reduced GPS interference). I also ended up with an unbootable SD Card. Adding dtoverlay=dwc2 to config.txt and echo 'blacklist uas' > /etc/modprobe.d/disable-uas.conf did not fix the issue.

[raspiduino]

Hey! I just found out about this page, and now I know that I'm not alone. I have faced this issue for like 1 year. I boot Armbian (Debian Bookworm-based) using microsd card to USB adapter, and I always got libc segmentation fault when using apt. Downgrade to Debian Bullseye fixes the issue, but it causes great compatibility problem with my program, so I can't. I even make a gist about alternative way to use apt here.

Today I reinstalled Armbian 23.11.1 (still Bookworm based) and it works without any libc problems, until I do a apt upgrade and new libc come and the problem is introduced again. So I guess a fix is to downgrade glibc back to a earlier version.

EDIT: I think I found the solution. Just download the libc-bin version 2.36.9-deb12u6 here and then run dpkg -i on that deb file. This downgrade version 2.36.9-deb12u7 to 2.36.9-deb12u6.

Edit 2: I think one of these patches caused the problem, but I don't know which one did:

glibc (2.36-9+deb12u7) bookworm-security; urgency=medium

  * debian/patches/local-CVE-2024-33599-nscd.diff: Fix a stack-based buffer
    overflow in nscd netgroup cache (CVE-2024-33599).
  * debian/patches/local-CVE-2024-33600-nscd.diff: Fix a null pointer
    dereferences in nscd after failed netgroup cache insertion
    (CVE-2024-33600).
  * debian/patches/any/local-CVE-2024-33601-33602-nscd.diff: Fix a DoS in nscd
    in case of memory allocation failure (CVE-2024-33601) and a memory
    corruption in nscd when the underlying NSS callback function does not use
    the buffer space to store all strings (CVE-2024-33602).

(from https://metadata.ftp-master.debian.org/changelogs//main/g/glibc/glibc_2.36-9+deb12u7_changelog)

[MichaIng]

@raspiduino
Many thanks for you analysis. The thing is, OP had this error with 2.36-9+deb12u3 (actually a rebuild of RPi Ltd of that version, like the one which fixes things for you), hence form before the patch you linked, and older than the version that is working for you. Also, weird is that not everyone is affected by this issue. So I am thinking that it may be not actually related to the libc-bin version, but some error caused by upgrading it, which happens only under rare conditions.

So it is not an RPi but some ARM set-top box in your case? And the latest Debian build of libc-bin breaks it, while the RPi LTd build from archive.raspberrypi.org solves it? At least then we know that the RPi build is not exclusively causing the issue.

Does it work as well when you use the older build from Debian snapshots here? https://snapshot.debian.org/archive/debian/20240429T204458Z/pool/main/g/glibc/libc-bin_2.36-9%2Bdeb12u6_arm64.deb

[raspiduino]

but some error caused by upgrading it, which happens only under rare conditions.

I think so. I think a common thing between us is we all booting from SD-to-USB adapter.

So it is not an RPi but some ARM set-top box in your case?

Yes. It's a quadcore ARM Cortex-A53 set-top box. It boots Armbian from SD-to-USB adapter.

Does it work as well when you use the older build from Debian snapshots here? https://snapshot.debian.org/archive/debian/20240429T204458Z/pool/main/g/glibc/libc-bin_2.36-9%2Bdeb12u6_arm64.deb

Yes it works.

[MichaIng]

Guys, the latest version of the package contains quite a lot of fixes, including some crashes. Can you test again upgrading it?