kernel oops: undefined instruction

[emmanueltouzery]

This in on a self-compiled kernel, but on your source with just a minor patch which was already accepted on kernel.org.

the kernel i built is 94fbbc4, i just added this tiny patch from the upstream kernel:
torvalds/linux@9ab2393

i also use the latest firmware, 87a04c0. I'm using the 192Mb configuration.

I compiled the kernel on ubuntu 12.04.

Here is the kernel.img i'm booting on:
http://dl.dropbox.com/u/22600720/rpi/kernel.img

and the .config file I used:
http://dl.dropbox.com/u/22600720/rpi/.config

Compared to the default .config i have added I2C and pvrusb2 and such drivers. I'm plugging a Hauppage HVR-1900 on the pi for TV recording.

The problem occured after about 1h 10 minutes of recording from the device. Concretely it means streaming ~1Mb/s from the USB to the SD card. The CPU is relatively OK when doing that, like at 15%.

Because I have seen frame drops doing that, I am using two processes for the recording, one of which runs at realtime scheduling priority, using that flywheel program described there:
http://www.isely.net/pipermail/pvrusb2/2011-March/003037.html

So after about 1h I get that kernel error. I have had that same problem several times now, the previous time with the same kernel but the original firmware that came with the debian image, but then I didn't get a log in the kern.log, unlike this time.

and here is the full stack trace:

Jun 3 16:24:22 raspberrypi kernel: Internal error: Oops - undefined instruction: 0 [#1] PREEMPT
Jun 3 16:24:22 raspberrypi kernel: Modules linked in: ipv6 fuse tda10048 tda18271 tda8290 tuner cx25840 pvrusb2 dvb_core cx2341x tveeprom v4l2_common videodev i2c_core
Jun 3 16:24:22 raspberrypi kernel: CPU: 0 Not tainted (3.1.9 #2)
Jun 3 16:24:22 raspberrypi kernel: pc : [] lr : [] psr: 60000013
Jun 3 16:24:22 raspberrypi kernel: sp : c8893ea0 ip : 00000000 fp : c8893ecc
Jun 3 16:24:22 raspberrypi kernel: r10: c0388594 r9 : 00000000 r8 : 00000001
Jun 3 16:24:22 raspberrypi kernel: r7 : 00000000 r6 : 00000001 r5 : 00000010 r4 : c8893f78
Jun 3 16:24:22 raspberrypi kernel: r3 : 00000000 r2 : 00000001 r1 : 00000002 r0 : ffffffea
Jun 3 16:24:22 raspberrypi kernel: Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
Jun 3 16:24:22 raspberrypi kernel: Control: 00c5387d Table: 0bb64008 DAC: 00000015
Jun 3 16:24:22 raspberrypi kernel: Process top (pid: 1748, stack limit = 0xc8892268)
Jun 3 16:24:22 raspberrypi kernel: Stack: (0xc8893ea0 to 0xc8894000)
Jun 3 16:24:22 raspberrypi kernel: 3ea0: c009ffc0 c885fc40 c009ffc0 c885fc40 00000000 00000000 00000000 c8893f78
Jun 3 16:24:22 raspberrypi kernel: 3ec0: c8893f1c c8893ed0 c00e2cc8 c009ffcc 00000000 00000000 00000001 00000000
Jun 3 16:24:22 raspberrypi kernel: 3ee0: 00000004 00000004 c009ffc0 c03e0970 cb4085b8 c885fc40 c009ffc0 c8893f78
Jun 3 16:24:22 raspberrypi kernel: 3f00: 00000000 cb403d88 c8892000 c8893f78 c8893f2c c8893f20 c00e2f10 c00e2c6c
Jun 3 16:24:22 raspberrypi kernel: 3f20: c8893f4c c8893f30 c00dd464 c00e2ef0 c00dd43c c885fc40 cb403dec c009ffc0
Jun 3 16:24:22 raspberrypi kernel: 3f40: c8893f74 c8893f50 c00a03c0 c00dd448 00008000 c885fc40 00000000 00a02350
Jun 3 16:24:22 raspberrypi kernel: 3f60: c000e148 00000000 c8893fa4 c8893f78 c00a04b0 c00a0348 00a02350 00000000
Jun 3 16:24:22 raspberrypi kernel: 3f80: 00008000 ffffffea 00a02350 0000000b 00a02338 0000008d 00000000 c8893fa8
Jun 3 16:24:22 raspberrypi kernel: 3fa0: c000dfa0 c00a0454 00a02350 0000000b 00000007 00a02350 00008000 00a02350
Jun 3 16:24:22 raspberrypi kernel: 3fc0: 00a02350 0000000b 00a02338 0000008d 00000000 00000000 00000000 bee6270c
Jun 3 16:24:22 raspberrypi kernel: 3fe0: 402da5c8 bee626b8 4037cf80 4037d634 60000010 00000007 00000000 00000000
Jun 3 16:24:22 raspberrypi kernel: Backtrace:
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: r6:c009ffc0 r5:cb403dec r4:c885fc40 r3:c00dd43c
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: Function entered at [] from []
Jun 3 16:24:22 raspberrypi kernel: r7:0000008d r6:00a02338 r5:0000000b r4:00a02350
Jun 3 16:24:22 raspberrypi kernel: Code: e1a01fe1 e3a03000 e1530009 01520008 (0c06d0f6)
Jun 3 16:24:22 raspberrypi kernel: ---[ end trace 034411e662b4c9d9 ]---

I have also put the full kern.log there:
http://dl.dropbox.com/u/22600720/rpi/kern.log

[popcornmix]

Can you decode the addresses using your System.map, or rebuild with a config based on bcmrpi_defconfig (rather than bcmrpi_cutdown_defconfig) which will include a full backtrace.

[emmanueltouzery]

as a first step here is the system.map, i'll try to find out more on my own too:

http://dl.dropbox.com/u/22600720/rpi/System.map

[emmanueltouzery]

this is the output of decodecode on this stack:

Jun 3 16:24:22 raspberrypi kernel: Code: e1a01fe1 e3a03000 e1530009 01520008 (0c06d0f6)

All code

0: e1 1f loope 0x21
2: a0 e1 00 30 a0 mov 0xa03000e1,%al
7: e3 09 jecxz 0x12
9: 00 53 e1 add %dl,-0x1f(%ebx)
c: 08 00 or %al,(%eax)
e: 52 push %edx
f: 01 f6 add %esi,%esi
11: d0 06 rolb (%esi)
13: 0c .byte 0xc

Code starting with the faulting instruction

0: f6 d0 not %al
2: 06 push %es
3: 0c .byte 0xc

after that i'll try recompiling as you suggested (but maybe it won't crash the same way or maybe i don't get a oops stack this time), if there's anything else i can do besides that let me know.

[emmanueltouzery]

i'm in the process of building that "debug" kernel. should i then also copy the System.map in the /boot on the pi before attemping reproduction? or in any other folder?

[popcornmix]

System.map is not needed by kernel. Just allows you to look up addresses.
Hopefully the debug kernel won't need that as it should display function names in the backtrace.

[popcornmix]

System.map is not needed by kernel. Just allows you to look up addresses.
Hopefully the debug kernel won't need that as it should display function names in the backtrace.

[emmanueltouzery]

yes unfortunately it froze again (well at least it lost all network connections, had to reboot it) but this time no stack.
if it can help, the last lines in kern.log were:

Jun 3 21:02:32 raspberrypi kernel: [ 1645.022343] top used greatest stack depth: 4620 bytes left
Jun 3 21:25:16 raspberrypi kernel: [ 3009.083867] flush-179:0 used greatest stack depth: 4556 bytes left

i started another run now, but i think it was lucky to get a stack at all, so i might get it to freeze again (it seems a reliable freeze) but again maybe without a stack.

[emmanueltouzery]

another freeze, again no stack.

[emmanueltouzery]

it froze again, and again no stack. i really hope that first stack will be good enough in the end, because i start to doubt seriously that i'll ever get another stack. would there be another way to debug this that would guarantee i will get a stack, or something like that?

there is a good news though: i realised that I somehow garbled the original stack when pasting it here...all the entered at [] from [] addresses are in fact present in the stack I got.

you can see them in the kern.log i put on dropbox, or on this oops.txt:
http://dl.dropbox.com/u/22600720/rpi/oops.txt

i don't know how to further process that oops using the symbols file. if you tell me how or give me a link, i can try.

[emmanueltouzery]

i also tried to decode by hand the stack addresses to function names using the system.map, and it's what I got (since i did it by hand there could be mistakes but i think it's OK):

Backtrace:
Function entered at [t filldir] from [ -- T proc_readdir_de + 0x68]
Function entered at [T proc_readdir_de] from [ -- t proc_readdir + 0x2c]
Function entered at [T proc_readdir] from [ -- proc_root_readir + 0x28]
Function entered at [t proc_root_readdir] from [ -- vfs_readdir + 0x84]
r6:c009ffc0 r5:cb403dec r4:c885fc40 r3:c00dd43c
Function entered at [T vfs_readdir] from [ -- sys_getdents + 0x68]
Function entered at [T sys_getdents] from [t ret_fast_syscall]
r7:0000008d r6:00a02338 r5:0000000b r4:00a02350
Code: e1a01fe1 e3a03000 e1530009 01520008 (0c06d0f6)
---[ end trace 034411e662b4c9d9 ]---

[emmanueltouzery]

if it's actually reading /proc it seems it may be connected with the fact that on top of that flywheel (with chrt) and cat, i also had each time "top" running, which made it easier to notice if the pi froze or not. tomorrow evening i'll maybe try to run it without having top running.
but well no matter what it shouldn't freeze ;-)

[popcornmix]

A lots has changed since this was reported. Can you confirm if issue happens with latest kernel?

[ghollingworth]

No reply closing