ODROID-N2 doesn't boot from Kingston eMMC modules

[piercy]

Describe the issue you are experiencing

I recently purchased an odroid n2+ and 32gb emmc module. After flashing haos_odroid-n2-12.1.img.xz, first with a usb emmc device and then trying to do it via USB connection with the odroid n2+ device. The flash was successful in both cases, but once the device was powered up I couldnt access HAOS via IP or homeassistant.local. I knew the IP was correct as I could compare the mac address.

There was no blue light on the board, just a red one. After some trial and error I went back to HAOS 12.0 and it flashed and worked first time. I had a blue light and I could immediately access it through a browser.

What operating system image do you use?

odroid-n2 (Hardkernel ODROID-N2/N2+)

What version of Home Assistant Operating System is installed?

12.1

Did you upgrade the Operating System.

No

Steps to reproduce the issue

1. Flash haos_odroid-n2-12.1.img.xz to an odroid device
2. Power it on
3. observe no blue light and home assistant is inaccessible
   ...

Anything in the Supervisor logs that might be useful for us?

Couldn't get to logs as it wouldn't boot

Anything in the Host logs that might be useful for us?

Couldn't get to logs as it wouldn't boot

System information

No response

Additional information

No response

[scoobydrvr]

Just jumping in to say I have a very similar issue with my new n2+ as well...

#3227

[piercy]

I have been struggling with this again today, seems to not be based on the version. neither 12.0 or 12.1 work for me. However, if i flash ubuntu the device boots fine. So i dont think it can be a device or emmc issue.

No idea what to do any more but it seems like I cant use HAOS with my odroid n2+

[scoobydrvr]

Just FYI - I had to buy a USB to serial converter to get any indication of what's going on. I've been informed they changed the bootloader between 11.4 and 11.5. 11.4 boots fine but 11.5 gives me a message over serial saying that it's unable to find the eMMC card (see my thread above for specifics).

[piercy]

Just FYI - I had to buy a USB to serial converter to get any indication of what's going on. I've been informed they changed the bootloader between 11.4 and 11.5. 11.4 boots fine but 11.5 gives me a message over serial saying that it's unable to find the eMMC card (see my thread above for specifics).

Yeah, im following that now too, unfortunately 11.4 doesnt work for me either.

No idea what to do any more, on yours i noticed a successful boot seems to use mmc 1 but unsucessful is trying mmc 2. No idea why that would be but im at a loss on what else to do any more.

I basically just cant use HAOS at the moment, even though i had it running for 3 days on 12.0 before it died... i have no idea how to even begin trying to solve this

[piercy]

I tried 12.2.rc1 today, the issue still persists

[agners]

Let's track it in #3227 as this seems to be the issue with most information at this point.

[sairon]

Reopening this after discoveries from #3227 (comment). This issue is clearly a problem with the Kingston eMMC modules which never worked with the upstream U-Boot and the 24 MHz we carry to mitigate some other eMMC-related issues. Since we pulled back the 12.3 release for ODROID N2 which was the only one that fixed booting with the Kingston eMMC modules, it was actually never resolved.

[piercy]

Copying my commment from #3227 here:

@sairon See below. Since the update in april I have been working completely fine. Also, if you need to chat about it more, or need more information, im piercy (Car with pixar eyes for a profile photo) on the HA discord. Feel free to DM or tag me.

Kingston
2401076-001.A006-A  (it could be 00G-A at the end, hard to tell)
2338  4014187
EMMC32G-PJ30

4014187.09
Taiwan

[piercy]

Regarding version, I haven't changed my HAOS versions since installation. So I would assume its still on 12.3. I have been updating HA Core but I imagine that doesn't affect this.

[piercy]

So I didn't upgrade, in fact deliberately skipped the upgrade. And on a reboot it now wont boot again. Literally so sick of this, first trip into home assistant, ordroid and MMCs and I end with a PoS that barely works.

sorry for the rant but its so frustrating how anyone is supposed to know that their MMC is going to be an issue.

[piercy]

Just a further update. I have decided to no longer use HA. This was my first time using HA. This experience has been utterly awful and left an incredibly sour taste in my mouth. I am the unlucky person to buy a "supported" - or at least it wasn't unsupported - EMMC, and then have the developers unsupport it on me. I don't believe I have received much help regarding this issue, or getting it to work. The small amount of help I did receive only ended up in the EMMC being unsupported. Which is basically telling me to get stuffed and deal with it. I realise this is open source, and there's no reason for anyone to provide support, but there is also an entire company behind it but yet here I am struggling.

There only seems to be one company available for me to buy an EMMC from, and they cannot guarantee to provide me with a non-Kingston emmc. So i am basically screwed until someone with knowledge decides to pick this up and fix it.

I have given up on HAOS, and as a first experience into it, its been utterly terrible.

I thank anyone who has tried, and sorry this may come across as ungrateful but please understand i've spent 100s of pounds on supported hardware only to be told, yeah your not supported any more - Good luck on your own.

[piercy]

Link to another two people with the issue, as reported on issue #3227

#3227 (comment)

#3227 (comment)

Also, seems to be multiple people here with the same/similar issue:

#3351

[amiler]

Kingston 32GB device does not work on multiple N2+ systems. I have tried every single HAOS version from 11.3 up to the latest 13.1. None worked. Device tries to boot from network with "serverIP not set" console message. Just hangs. Other OS (Ubuntu) flashed on this 32GB eMMC does work without problems.
I have 64GB device (not Kingston) with same revision pcb that does boot with any HAOS version. Unfortunately, I cannot clone 64GB and use on 32GB. Is there anything that eMMC "driver" owner(s) is(are) going to do about this or we have to start getting non Kingston eMMC modules from now on?

Pics attached of working and non-working eMMC device.

[yfrans]

Same here.
New ODROID N2+, HAOS v13.1, no boot.
Tried HAOS v10.1 and it worked, upgraded through HA to 10.5 and it fail again.

Screenshot from non working v10.5:

[piercy]

In a strange, turn of events, I am currently running 12.4 on my 32gb kingston emmc. I am not restarting it, for fear that it wont turn back on.. however, it does seem very strange that I am now working fine on 12.4.

Posting this mainly to maybe help the developers here identify what may be going on. It doesn't seem as simple as it doesn't work with kingston emmc's, but something a little more intricate as it works randomly

[sairon]

Sorry for leaving you hanging here for such a long time, and sorry @piercy for the bad experience you had with Home Assistant. Unfortunately, this is something that struck us unprepared and it took a while to discover what was the cause of the sudden issues on N2. It also wasn't straightforward to obtain these eMMCs, as the distributors haven't had them or refused to check what model they have in stock, which prolonged the time before we could move forward.

The problem with N2 is that the eMMC driver in U-Boot is known to have issues for quite a long time. There is a hacky patch for quite a while in HAOS that helped to fix the intermittent issues on the previous hardware. It's not only problem of HAOS but of other distributions using mainstream U-Boot on meson-g12b (Amlogic S922X) in general - only just a few months ago Armbian included it too. The downside of the patch is that it breaks communication with new Kingstom eMMC modules, which fail (intermittently, as you noticed) to operate at the lower frequency (although they should work fine in this setting too). On the other hand, when we adjusted the patch to fix Kingston issues, it triggered regression on hardware that worked well up until that point, which is more serious than something not working "out of the box".

The nature of this hardware bug is most probably in the clock timing, and if you don't know what you're doing, you can only shoot in the dark and try fiddling with numbers, hoping it will change. To reliably analyze the issue, you need specialized hardware that can help you see what's going on on the eMMC lines and why it sometimes fails. Even though Nabu Casa is behind the Home Assitant, it doesn't mean we have ways to solve every issue that is out there, especially if it requires specialized knowledge and other resources. At this point, only telling you that there is a bug that we can't resolve and the hardware is unsupported (yet) is the only "support" we can provide you.

There are still workarounds though - you don't need to boot from an eMMC - you can use an SD card for the boot. You still might be able to use the Kingston eMMC as the data disk (which is where the most data is written), since the Linux eMMC driver doesn't seem to have the same issues as the U-Boot's does. You can also stay at HAOS 12.3, or use the U-Boot binary from this version to run any other HAOS version (I'll be glad to provide instructions how to do so) - you just need to make sure you don't upgrade the OS before the U-Boot bug is resolved - if you do, you'll have to replace the U-Boot binary again. Also, it looks like that the Kingston eMMCs only come in 32 GB size variants, that can serve as a buying guide to avoid these issues.

For some good news, I tried to find someone who could help me identify why the Kingston eMMC behaves differently, and potentially what's the difference in driving the eMMC from U-Boot and from the kernel. There is something in the works but it would not be sooner than by the end of the September though, so patience is the key now.

[mversluijs]

@sairon thnx for the update!
Have similar issues with a brand new 64GB eMMC module, probably Kingston as booting from 12.3 'fixed' it.
Looking forward to a fix.

[coconutssjaakie]

I've struggled with this issue on 32 and 64 GB Kingston eMMC. I am now running HA but I am afraid to reboot or upgrade the system. I bought myself a back power supply so my n2+ will stay alive when a power interrupt should occur.
The crazy thing is; I;ve tried to boot several versions of HA on a 64Gb eMMC. It kept me busy for a couple of days. No succes. When I returned the card to the reseller, they were able to boot it up within seconds. What the.....

[dselzle]

Fingers crossed that someone can find a solution for this, either here or upstream somewhere. I was going to order one of the new Odroid M2s until I noticed that the soldered eMMC it comes with appears to have the word "Kingston" on it. I'm going to hold off for now if booting off a microSD is the only easy option for it.

[sairon]

@dselzle The issue is only with the SoC found on ODROID N2, the Kingston eMMC works correctly on other SBCs - at least I have tested it on M1 without any issues. There is a problem though - there is no image for ODROID M2, it's out just a few weeks.

[dselzle]

@dselzle The issue is only with the SoC found on ODROID N2, the Kingston eMMC works correctly on other SBCs - at least I have tested it on M1 without any issues. There is a problem though - there is no image for ODROID M2, it's out just a few weeks.

Good to know, thanks!

[amiler]

I got a new eMMC module (64GB) to replace 32GB Kingston and, unfortunately, I got Kingston again. I have working system with non-Kingston eMMC that works in both N2+ systems. Comparing debug messages between the two (failing and passing boot with two different eMMC), it show that failing boot is trying to boot from MMC device at slot 2, continues to try to boot from network and fails. Both cards did not respond to "voltage select" but that did not stop non-Kingston eMMC from booting successfully. Failing boot shows an old U-Boot version because I was trying to flash an old version of HAOS (I believe it was 12.3 or older) while good boot is using the latest 13.1 version of HAOS. When HAOS is flashed onto SD card (any version), system boots without problems.
I guess this data is not new but wanted to share in case if it helps. Hopefully, this will be resolved soon since, when bying eMMC, you cannot tell if it is Kingston or not. Clearly, issue is with Kingston eMMCs. I have tried 32GB and 64GB purchased from two different sources and they both failed.
Picture of logs (left - failed, right - successful boot)

[betriebsanleitung]

No help for debugging. But just the information that it seems that Hardkernel emmc modules with Kingston get more common. Bought some in Germany from different suppliers to set up a new Home Assistant on ODROID-N2+. All with Kingston. And all show the same boot up error as described above (trying to boot from slot 2, not able to set voltage).

[betriebsanleitung]

Hi @sairon, is there any news about a fix for emmcs with Kingston chip?
I tried to find an emmc without Kingston chip, but all suppliers in Europe who could check their stock told me that all emmc modules are now with Kingston.

I bought the ODROID-N2+ because it is still recommended hardware on the Home Assistant website. And on the hassio setup page there's still no information that users with new hardware are currently unable to set up a new system as described on the website. So I guess more and more users will run into this problem.

Is there a timeline for when we can expect a fix? I know its not easy to fix, but any information or guess is better than nothing.
Unfortunately, I cannot return my newly purchased hardware because it took me too long to find the reason why my new system would not boot (and the reason in this thread). Now I'm wondering if it's faster to sell it on ebay and move on to other hardware than to wait for a fix. Or if a fix is planned within the next 1-3 weeks.

Thanks for the update!

[piercy]

I bought the ODROID-N2+ because it is still recommended hardware on the Home Assistant website.

This is a good point, HA need to stop recommending this if they aren't going to support it. The EMMC chips are getting more common. More and more people are going to get duped.

There's no mention of this incompatibility / lack of hardware support.

[piercy]

Also could we please remove the duplicate flag from this thread, this issue isn't a duplicate.

[sairon]

I've made some progress and things are looking good on my side, I will appreciate wider testing with the images available from this build (should finish in couple of hours top).

Eventually, the patch could be included in 14.0 and its pre-releases in the coming weeks. I'm hesitant to include it in the coming 13.2 as there's still a slight chance it could introduce regression on other eMMC modules.

[piercy]

Edit: neither 13.2, 12.4, nor the 12.3 dev build you provided a while back is running for me now. I don't know if its just luck. Going to keep trying with it see if I can work anything out. I was 100% running 12.4 since august..so seems to be temperamental

Edit 2: I tried again via micro-USB (rather than my MMC adapter), then flashing with balenaEtcher and no luck. So i've tried the 13.2 dev build, the 12.4 build that I have been using since august, and a 12.3 dev build you tried for us before. None of them work via micro USB or flashing with my mmc adapter. It's really odd that I managed to get 12.4 working since august, but now I cant get anything to work. I havent restarted since august, but i was definitely running 12.4.

Edit 3: I accidentally left it plugged in over night. It still didn't boot. However, I power cycled it and it started booting. It was on the 13.2 dev build. However, I didn't reflash it, just power cycled and it worked. Very odd.

@sairon Unfortunately that version doesn't boot for me. The N2+ shows a red light for pwr but the blue sys light that usually flashes, isn't flashing (which correct me if I am wrong, usually means it has an issue and isn't trying any more). I also tried to communicate with it via the HA web interface but it wasn't available (which I expected but thought I would try).

There is no output on the HDMI, so I am unable to get any additional information from it. Is there another way for me to get more information out of it?

Steps for what I did:

1. Downloaded haos_odroid-n2-13.2.dev1728582564.img.xz
2. Extracted it from the zip (it downloaded as a zip)
3. Plugged my emmc into a usb adapter
4. Flashed it to the emmc with balenaEtcher and let it verify, all good
5. Put the EMMC back into the N2+ and plugged the power in
6. Saw red pwr light, but no blue sys light. For info, he switch is in the right position (so on the MMC side)

[amiler]

Patch that @sairon put in devel branch does not work for me. Same message during boot

Card did not respond to voltage select! : -110
unable to select a mode : -5
MMC Device 2 not found
no mmc device at slot 2

Trying to source non-Kingston eMMC module.

[remconiesten]

Patch tested from the devel branch @sairon posted 3 days ago. Also tested 12.3 (didn't test this one before, but several other releases I did test as well, all with same result). It all ends in *** ERROR: `serverip' not set

This is on a 64GB Kingston eMMC, see photo. Ubuntu boots from this eMMC though.

[sairon]

Thank you all for testing and reporting back, I was off by the end of the week and couldn't reply sooner. While the fix worked in isolated U-Boot tests and quick HAOS boot test, for some reason it still fails with higher rate in the normal build/initialization procedure.

However, I power cycled it and it started booting. It was on the 13.2 dev build. However, I didn't reflash it, just power cycled and it worked. Very odd.

It's not that odd, given that if the boot fails, it doesn't reboot and try again, but tries to boot from the ethernet and then presents a U-Boot shell prompt (it's the default behavior of upstream U-Boot config and it's not documented for HAOS, so I'd like to change that as well).

Anyway, even though the bad news is that the fix doesn't work, there's still good news that I'm able to reproduce it and I still have couple of ideas what to try, just this time I'll do some more extensive/overnight testing to verify the issue is really gone. Thanks for the patience.

[piercy]

Thanks @sairon , if it helps I am piercy, on the HA discord. If you need more immediate feedback feel free to message me. I am happy to try random devbuilds and all sorts of stuff if it helps in the debug process. I realize it can be painful doing a build then waiting a couple of days for people to see if it works.

I do struggle to get debug information off of the n2+, i dont know if theres some kind of debug connector I can get to get more information from it? Either way, if it helps I am happy to spend some time just trying stuff until we find something that works

[sairon]

Thanks @piercy for the offer, you've been very helpful with the testing here as well, next time I'll just need to do test everything more thoroughly myself first. But surely I can ping you the next time I have something to test too.

About the debug information, there's a dedicated UART connector next to the microSD card - there's either a Hardkernel's proprietary USB-UART module, or you can use a generic 3.3V USB-TTL converter an connect it according to the pinout documented on the ODROID wiki. Disregard the instructions for UART_EE_A/UART_EE_B connection, U-Boot only uses the one on the CON5.

[scoobydrvr]

Quick note for everyone - I received an alert notifying me of an update to the supervisor to 2024.10.2; I've been skipping Core and Supervisor updates until the issue with the eMMC is resolved but this time I was presented with a message telling me auto updates are enabled and the version can't be skipped. After some digging around, I found the cli instructions to disable auto-updates:
ha supervisor options --auto-update=false

Maybe this is unnecessary but I'd rather not lose my working system to an unavoidable auto update.