rockchip isp1 driver

[wzyy2]

Rockchip ISP1 driver is a new isp driver for rockchip rk3288/rk3399 SoC, we might use it in future for all linux-kernel based OS, including Android, ChromeOS, Linux......
It's writen in standard v4l2, will be compatible with upstream sensor driver.
The development will be last until 2018.04, i hope we could have a working driver, camera hal3, gstreamer plugin at that time.

If you find something is missing in this driver, maybe you could help us add it. ; - p
e.g: MIPI TXRX, you can refer to https://github.com/rockchip-linux/kernel/tree/release-4.4/drivers/media/platform/rk-isp10

At present we have some working drivers for linux, but i recommend to not use them especially you don't have commercial support, since it's also buggy and no one can help you......

Status

I have push the new MIPI CSI driver to isp-early-access branch.
https://github.com/rockchip-linux/kernel/tree/isp-early-access

upstream status:
https://patchwork.linuxtv.org/project/linux-media/list/?submitter=7231

I will maintain this branch until the driver be merged into Rockchip 4.4 kernel/Upstream kernel/ChromeOS kernel.
Report bugs are welcome.

Missing feature in this new driver:

• No MIPI-TXRX/ DVP support
• No 3A support

TODO in kernel 4.4:

• Test isp0/isp1 in rk3399 and add dts node
• Test dual isp in rk3399

Test in:

• Tinker Board + imx219/ov5647
• Firefly + HDMI_IN

[nova-wenbo]

What is the configuration file for the kernel?

[wzyy2]

803f78e

[nova-wenbo]

thank you

[Ribonn]

Hi:

I have a question about the return value of the function rkisp1_irq_handler() in the file drivers/media/platform/rockchip-isp1/dev.c.

I think it should return IRQ_HANDLED other than IRQ_NONE, that is:
--- a/drivers/media/platform/rockchip-isp1/dev.c
+++ b/drivers/media/platform/rockchip-isp1/dev.c
@@ -554,7 +554,7 @@ static irqreturn_t rkisp1_irq_handler(int irq, void *cntxt)

    /* TODO: update crop & resize */
    clr_all_int(base);

•   return IRQ_NONE;
  

•   return IRQ_HANDLED;
  

}

When the function returns IRQ_NONE, the kernel will output these logs:
[ 661.702947] irq 49: nobody cared (try booting with the "irqpoll" option)
[ 661.709667] CPU: 0 PID: 919 Comm: rkcamsrc0:src Not tainted 4.4.83 #1
[ 661.716107] Hardware name: Rockchip (Device Tree)
[ 661.720847] [] (unwind_backtrace) from [] (show_stack+0x20/0x24)
[ 661.728607] [] (show_stack) from [] (dump_stack+0x84/0xa0)
[ 661.735845] [] (dump_stack) from [] (__report_bad_irq+0x34/0xcc)
[ 661.743602] [] (__report_bad_irq) from [] (note_interrupt+0x1f4/0x29c)
[ 661.751878] [] (note_interrupt) from [] (handle_irq_event_percpu+0x1d4/0x250)
[ 661.760759] [] (handle_irq_event_percpu) from [] (handle_irq_event+0x48/0x6c)
[ 661.769637] [] (handle_irq_event) from [] (handle_fasteoi_irq+0xb8/0x134)
[ 661.778168] [] (handle_fasteoi_irq) from [] (generic_handle_irq+0x28/0x38)
[ 661.786788] [] (generic_handle_irq) from [] (__handle_domain_irq+0x9c/0xc4)
[ 661.795492] [] (__handle_domain_irq) from [] (gic_handle_irq+0x60/0xa4)
[ 661.803849] [] (gic_handle_irq) from [] (__irq_usr+0x54/0x80)
[ 661.811329] Exception stack(0xec781fb0 to 0xec781ff8)
[ 661.816386] 1fa0: 00000008 c06864b8 b5bfe658 00000000
[ 661.824568] 1fc0: 00000000 b5bfe658 c06864b8 00000008 b5bfe7cc b5bfe658 00000008 00000280
[ 661.832747] 1fe0: b664315c b5bfe628 b662d567 b662cae0 800f0010 ffffffff
[ 661.839358] handlers:
[ 661.841645] [] rkisp1_irq_handler
[ 661.845844] Disabling IRQ #49
[ 772.802163] rkisp1: isp icr frame end err: 0x22
[ 1467.402200] rkisp1: isp icr v_statr err: 0x62
root@linaro-alip:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
16: 79634 36933 56554 67736 GIC 29 Edge arch_timer
17: 0 0 0 0 GIC 30 Edge arch_timer
20: 0 0 0 0 GIC 98 Level rk_timer
25: 0 0 0 0 GIC 34 Level ff250000.dma-controller
26: 0 0 0 0 GIC 35 Level ff250000.dma-controller
27: 201 0 0 0 GIC 32 Level ffb20000.dma-controller
28: 0 0 0 0 GIC 33 Level ffb20000.dma-controller
29: 0 0 0 0 GIC 64 Level dw-mci
30: 14530 0 0 0 GIC 65 Level dw-mci
31: 14446 0 0 0 GIC 67 Level dw-mci
32: 0 0 0 0 GIC 68 Level ff100000.saradc
34: 3060 0 0 0 GIC 92 Level ff650000.i2c
35: 943 0 0 0 GIC 94 Level ff140000.i2c
36: 0 0 0 0 GIC 96 Level ff160000.i2c
39: 1604 0 0 0 GIC 89 Level serial
41: 0 0 0 0 GIC 69 Level rockchip_thermal
42: 4686 0 0 0 GIC 59 Level eth0
43: 0 0 0 0 GIC 60 Level eth0
44: 12571514 0 0 0 GIC 57 Level ff540000.usb, dwc2_hsotg:usb1
45: 0 0 0 0 GIC 55 Level ff580000.usb, ff580000.usb, dwc2_hsotg:usb2
46: 0 0 0 0 GIC 93 Level ff660000.i2c
49: 100001 0 0 0 GIC 46 Level rkisp1
50: 0 0 0 0 GIC 50 Level ff920000.rga
51: 88825 0 0 0 GIC 47 Level ff930000.vop, ff930000.vop
52: 0 0 0 0 GIC 48 Level ff940000.vop
53: 1544 0 0 0 GIC 135 Level ff980000.hdmi
54: 0 0 0 0 GIC 41 Level ff9a0000.vpu-service
55: 0 0 0 0 GIC 42 Level ff9a0000.vpu-service
57: 0 0 0 0 GIC 44 Level ff9c0000.hevc-service
59: 7561 0 0 0 GIC 38 Level ffa30000.gpu
60: 0 0 0 0 GIC 39 Level ffa30000.gpu
61: 14065 0 0 0 GIC 40 Level ffa30000.gpu

[wzyy2]

Thanks.
It should return IRQ_HANDLED, I will add this change.

[axendev]

What changes should I make in DTS for rk3399 ?
I have tried do some changes to my DTS file (rk3399-firefly-linux.dts) according to this commit: a804db4 , but it rises compile-error "mipi_phy_rx0 not defined".

...
&mipi_phy_rx0 {      // There is no definition of lable 'mipi_phy_rx0'
	bus-width = <4>;
	status = "okay";
	port {
		isp_mipi_in: endpoint {
			remote-endpoint = <&camera_out>;
		};
	};
};

For rk3299 mipi_phy_rx0 is defined within 'cif_isp0: cif_isp@ff910000' node in rk3288.dtsi (https://github.com/rockchip-linux/kernel/blob/isp-early-access/arch/arm/boot/dts/rk3288.dtsi#L1126), but 'cif_isp0: cif_isp@ff910000' node in rk3399-linux.dtsi does not contains 'mipi_phy_rx0' (https://github.com/rockchip-linux/kernel/blob/isp-early-access/arch/arm64/boot/dts/rockchip/rk3399-linux.dtsi#L60). Should I modify 'cif_isp0: cif_isp@ff910000' node in rk3399-linux.dtsi like in rk3288.dtsi ?

[wzyy2]

https://chromium-review.googlesource.com/c/chromiumos/third_party/kernel/+/527854/149
Please refer to this for rk3399.

[axendev]

Thank you very much.
Can you tell me how I can test this driver in linux? Should I use 'v4l2-ctl' util for this purpose ?

[wzyy2]

https://github.com/rockchip-linux/gstreamer-rockchip-extra

[axendev]

Does rk3399 linux kernel support dual mipi port for ovXXX camera? #47

yes, it should work.
I have the same rk3399-firely board but with two imx219 cameras. I modified rockchip-isp1 sources (according to rk-isp10) in order to add mipi-tx1rx1 support. Now it works for each of the two cameras separately (using mipidphy_rx0+isp0 or mipidphy_tx1rx1+isp1). There are several problems with supporting two cameras simultaneously.
Firstly, both mipi-conectors on firefly`s board use the same i2c bus, unfortunately the imx219 cameras have identical i2c slave-address and have neither Chip-Select pins nor documented ability of changing slave-addr, so I use external i2c bus switchers IC (each has "enable" pin connected to RK3399 GPIO and controlled by imx219 i2c-driver) to provide switching between several cameras. You might need the same for ovXXX cameras.
The current task is to create the correct device-tree nodes for cameras and change the imx219.c i2c-driver to work with them. It is not acceptable to simply create two child-nodes (in 'i2c1' node) for each camera, because both cameras have the same slave address so both camera's nodes will be with identical reg property ("reg" property describes slave address in case of i2c device nodes) which will causes that only one camera will be recognized and probed.
If you need, I can provide my sources after I solve these problems.

[gangm]

@axendev , thanks for your recommend and guide, do you mean that I should modify the rockchip-isp1 soures to connect ov cameras? And I should also add the the ov camera's i2c code in the i2c node dir? If you have the worked code, can you provide your sources? Thanks very much! (my email: 565489195@qq.com / gmgtiger510@163.com)

[teseo-sw]

I use a single ov5647 sensor on rk3288. I managed to setup the kernel and the dts to the point I'm able to probe the sensor correctly:

root@ptam8:~# dmesg | grep ov5647
[    3.206161] i2c i2c-3: of_i2c: register /i2c@ff150000/ov5647@36
[    3.206324] i2c i2c-3: client [ov5647] registered with bus id 3-0036
[    3.305985] ov5647 3-0036: probe
[    3.308437] ov5647 3-0036: OmniVision ov5647 camera driver probed

However, when I try to capture frames the sensor seems to lose power and i2c r/w returns -6:

root@ptam8:~# v4l2-ctl --verbose -d /dev/video0 --stream-mmap=2 --stream-to=/tmp/stream.out --stream-count=60 --stream-poll
VIDIOC_QUERYCAP: ok
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QUERYBUF: ok
[  921.474910] rkisp1-isp-subdev: 107: name rkisp1-isp-subdev,on 1
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
[  921.486844] ov5647 3-0036: 107: name ov5647 3-0036,on 1
[  921.507757] ov5647 3-0036: Camera not available, check Power
[  921.514093] rockchip-sy-mipi-dphy: 107: name rockchip-sy-mipi-dphy,on 1
[  921.523356] ------------[ cut here ]------------
[  921.528481] WARNING: CPU: 0 PID: 523 at drivers/media/v4l2-core/videobuf2-core.c:1309 vb2_start_streaming+0xe8/0x164()
[  921.540305] Modules linked in:
[  921.543679] CPU: 0 PID: 523 Comm: v4l2-ctl Not tainted 4.4.93-pTAM8_v00.01 #39
[  921.551655] Hardware name: Rockchip (Device Tree)
[  921.556864] [<c0111038>] (unwind_backtrace) from [<c010c8bc>] (show_stack+0x20/0x24)
[  921.565423] [<c010c8bc>] (show_stack) from [<c041c470>] (dump_stack+0x8c/0xa0)
[  921.573405] [<c041c470>] (dump_stack) from [<c0120054>] (warn_slowpath_common+0x94/0xc4)
[  921.582345] [<c0120054>] (warn_slowpath_common) from [<c0120140>] (warn_slowpath_null+0x2c/0x34)
[  921.592052] [<c0120140>] (warn_slowpath_null) from [<c0687230>] (vb2_start_streaming+0xe8/0x164)
[  921.601753] [<c0687230>] (vb2_start_streaming) from [<c0688f40>] (vb2_core_streamon+0x128/0x170)
[  921.611460] [<c0688f40>] (vb2_core_streamon) from [<c068af4c>] (vb2_streamon+0x40/0x60)
[  921.620304] [<c068af4c>] (vb2_streamon) from [<c068afb8>] (vb2_ioctl_streamon+0x4c/0x50)
[  921.629247] [<c068afb8>] (vb2_ioctl_streamon) from [<c06715b4>] (v4l_streamon+0x2c/0x30)
[  921.638188] [<c06715b4>] (v4l_streamon) from [<c0675988>] (__video_do_ioctl+0x294/0x2f0)
[  921.647126] [<c0675988>] (__video_do_ioctl) from [<c0675250>] (video_usercopy+0x1d4/0x654)
[  921.656257] [<c0675250>] (video_usercopy) from [<c06756f0>] (video_ioctl2+0x20/0x24)
[  921.664811] [<c06756f0>] (video_ioctl2) from [<c066ffcc>] (v4l2_ioctl+0xb0/0xe8)
[  921.672986] [<c066ffcc>] (v4l2_ioctl) from [<c024fad8>] (do_vfs_ioctl+0x4ec/0x708)
[  921.681351] [<c024fad8>] (do_vfs_ioctl) from [<c024fd70>] (SyS_ioctl+0x7c/0x8c)
[  921.689426] [<c024fd70>] (SyS_ioctl) from [<c0107bc0>] (ret_fast_syscall+0x0/0x3c)
[  921.697807] ---[ end trace e7d096a1c9e37644 ]---
VIDIOC_STREAMON: failed: No such device or address

[wzyy2]

v4l2-ctl can't be used now since this driver use media controller api.
You have to use rkcamsrc gstreamer plugin to use this driver in linux.

[teseo-sw]

No luck with rkcamsrc either:

root@ptam8:~# gst-launch-1.0 rkcamsrc device=/dev/video0 io-mode=4 disable-3A=true 
Setting pipeline to PAUSED ...
rkcamsrc: Using ISP self path......
Pipeline is live and does[   39.989457] rkisp1-isp-subdev: 99: name rkisp1-isp-subdev,on 1
 not need PREROLL ...
Setting pipeline to PLAYING ...
New cloc[   40.000838] ov5647 3-0036: 99: name ov5647 3-0036,on 1
k: GstSystemClock
[   40.024121] ov5647 3-0036: Camera not available, check Power
[   40.030481] rockchip-sy-mipi-dphy: 99: name rockchip-sy-mipi-dphy,on 1
[   40.039652] ------------[ cut here ]------------
[   40.044782] WARNING: CPU: 1 PID: 795 at drivers/media/v4l2-core/videobuf2-core.c:1309 vb2_start_streaming+0xbc/0x138()
[   40.056606] Modules linked in:
[   40.059981] CPU: 1 PID: 795 Comm: rkcamsrc0:src Tainted: G        W       4.4.93-pTAM8_v00.01 #55
[   40.069781] Hardware name: Rockchip (Device Tree)
[   40.074998] [<c010fdf8>] (unwind_backtrace) from [<c010b9c4>] (show_stack+0x20/0x24)
[   40.083561] [<c010b9c4>] (show_stack) from [<c040488c>] (dump_stack+0x84/0xa0)
[   40.091546] [<c040488c>] (dump_stack) from [<c011dba8>] (warn_slowpath_common+0x98/0xc4)
[   40.100486] [<c011dba8>] (warn_slowpath_common) from [<c011dc90>] (warn_slowpath_null+0x2c/0x34)
[   40.110194] [<c011dc90>] (warn_slowpath_null) from [<c0827e40>] (vb2_start_streaming+0xbc/0x138)
[   40.119904] [<c0827e40>] (vb2_start_streaming) from [<c0829054>] (vb2_core_streamon+0xe0/0x13c)
[   40.129517] [<c0829054>] (vb2_core_streamon) from [<c082b410>] (vb2_streamon+0x48/0x58)
[   40.138360] [<c082b410>] (vb2_streamon) from [<c082b46c>] (vb2_ioctl_streamon+0x4c/0x50)
[   40.147305] [<c082b46c>] (vb2_ioctl_streamon) from [<c0811c80>] (v4l_streamon+0x2c/0x30)
[   40.156248] [<c0811c80>] (v4l_streamon) from [<c0817490>] (__video_do_ioctl+0x220/0x2ac)
[   40.165187] [<c0817490>] (__video_do_ioctl) from [<c0816f88>] (video_usercopy+0x2f0/0x5b4)
[   40.174318] [<c0816f88>] (video_usercopy) from [<c0817268>] (video_ioctl2+0x1c/0x24)
[   40.182873] [<c0817268>] (video_ioctl2) from [<c08107d4>] (v4l2_ioctl+0x70/0xac)
[   40.191037] [<c08107d4>] (v4l2_ioctl) from [<c0241444>] (do_vfs_ioctl+0x98/0x68c)
[   40.199306] [<c0241444>] (do_vfs_ioctl) from [<c0241a94>] (SyS_ioctl+0x5c/0x80)
[   40.207382] [<c0241a94>] (SyS_ioctl) from [<c01071c0>] (ret_fast_syscall+0x0/0x3c)
[   40.215778] ---[ end trace 8f8b1f5b2904f232 ]---
ERROR: from element /GstPipeline:pipeline0/GstRKCamSrc:rkcamsrc0: Could not read from resource.
Additional debug info:
../../../git/gst/rkv4l2/v4l2/gstv4l2bufferpool.c(1064): gst_v4l2_buffer_pool_poll (): /GstPipeline:pipeline0/GstRKCamSrc:rkcamsrc0:
poll error 1: No such device or address (6)
Execution ended after 0:00:00.233954583
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

dts: https://gist.github.com/teseo-sw/ff8641adf0733078b8053448b9303ccb