Can't run docker on 6.17 (wrong kernel config)

[Dmitrij-Solovjev]

First of all:
Thanks to you, enthusiasts are no longer tied to outdated banana pi images. I am extremely grateful for the work you have done.
I use this CI release:
https://github.com/frank-w/BPI-Router-Linux/releases/tag/CI-BUILD-6.17-main-6.17.0-2025-11-04_2136

Of cource, I copy BPI-ROOT to / and for example now I have next modules:

$ lsmod 
Module                  Size  Used by
mt7996e               147456  0
mt76_connac_lib        53248  1 mt7996e
mt76                   98304  2 mt7996e,mt76_connac_lib
mac80211              860160  3 mt76,mt7996e,mt76_connac_lib
libarc4                12288  1 mac80211
cfg80211              835584  4 mt76,mt7996e,mac80211,mt76_connac_lib
nft_masq               12288  1
nft_chain_nat          12288  1
nf_nat                 45056  2 nft_masq,nft_chain_nat
nf_conntrack          102400  2 nf_nat,nft_masq
nf_defrag_ipv6         20480  1 nf_conntrack
nf_defrag_ipv4         12288  1 nf_conntrack
nf_tables             229376  4 nft_masq,nft_chain_nat
fuse                  163840  1
nfnetlink              16384  3 nf_tables

But I can't load module

$ sudo modprobe br_netfilter
modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/6.17.0-bpi-r4-main

[frank-w]

seems BRIDGE_NETFILTER is missing. are you able to compile kernel yourself with this option added?

[Dmitrij-Solovjev]

Tomorrow I'll try to build an image for BPI-R4 with the flag enabled and test it.
Please tell me which file I need to modify.

~/Downloads/BPI-Router-Linux-CI-BUILD-6.17-main-6.17.0-2025-11-04_2136 (1)/BPI-Router-Linux-CI-BUILD-6.17-main-6.17.0-2025-11-04_2136/kernel/configs $ ls
debug.config  hardening.config  kvm_guest.config  nopm.config  rust.config  tiny-base.config  tiny.config  x86_debug.config  xen.config

[frank-w]

change build.conf to your board (remove the #) and then

./build.sh defconfig

To change the config in arch/arm64/configs

[Dmitrij-Solovjev]

I modify build.config for my bpi-r4 device.
I also used ./build.sh defconf to add CONFIG_BRIDGE_NETFILTER=y, but after booting, I got this output.

My uEnv.txt:

bootmenu_default=3
console=ttyS0,115200 earlycon=uart8250,mmio32,0x11000000
device=nvme
partition=1
nvme_boot=0:1
nvme_root=/dev/nvme0n1p2
uImage=uImage_nodt_mod
dtb=bpi-r4-mod.dtb
bootfile=bpi-r4-mod.itb

~ $ sudo modprobe br_netfilter
modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/6.17.0-bpi-r4-main

[frank-w]

are you sure your new kernel is taken?

have you checked if the option is set with ./build.sh config?

there are many errors in the uEnv.txt...partition is wrong (1 is not x:y) for loading uEnv.txt, but ignored for nvme (nvme_boot overwrites this). neither uImage nor dtb nor bootfile is used for kernel-boot.

the property is named "fit" to target to the itb file which has to be on the device/partition defined there (except in nvme partition there is additional uEnv.txt). Setting device to nvme by default may break things (sdcard/emmc boot)...it is set by the bootnvme function called from bootmenu with some additional initcode. please only change bootmenu_default and maybe set nvme_boot and nvme_root and your new fit (better set this in nvme boot partition to not interfere the sdmmc/emmc fit).

could you please first try to load your kernel from sdcard? the chainloading to nvme is a bit tricky

[Dmitrij-Solovjev]

tree.log

here is tree from SD folder (compile output dir) and tail of config file

I don't create build log, but when compile i search "br_netfilter" as .o file and don't see it as module

about file "fit"
Before these modifications, I used fit parametr to load .itb, but when my file changed the name to -mod.itb, my system stopped booting. In the logs, I came across bootfile=bpi-r4.itb and guessed that apparently uboot set this parameter by default, so I corrected fit=bpi-r4-mod.itb to bootfile=bpi-r4-mod.itb

[frank-w]

bootfile property is default for tftp

on your sdcard i see no uenv.txt and also not your new kernel as separate file, so i guess (when you boot from sdcard via hw jumpers) the default bpi-r4.itb is booted and not your modified version.

can you make backup of the existing bpi-r4.itb and copy your new compiled itb (after checking "./build.sh config", search with "/" and enter your config option and look if it set to "y") to the sdcard and boot from there?

after system is booted check with "uname -a" if buildtime is matching nearly your current time to verify that you have not booted the older kernel.

[Dmitrij-Solovjev]

I don't know how to load the kernel from the SD card yet. Please provide a working uEnv.txt configuration file for loading the kernel from the SD card and setting the root to nvme.

Probably what I should have done at the very beginning:
system configuration: I use an SD card to store eEnv.txt (and possibly .itb) on it, and I load the kernel from the first partition of nvme, which uses fat32, and the root of the system is mounted on the second partition of the disk.

Device        Bootable          start         end                ....                                         type
/dev/nvme0n1p1  *                       2048        1023999        1021952        499M              W95 FAT32 (LBA)
/dev/nvme0n1p2                          1024000   2000409263 1999385264  953,4G            Linux

dima@kube-master ~ $ lsblk 
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0    31:0    0     2M  0 disk 
mtdblock1    31:1    0   126M  0 disk 
mmcblk0     179:0    0   7,4G  0 disk 
├─mmcblk0p1 179:1    0     4M  0 part 
├─mmcblk0p2 179:2    0   512K  0 part 
├─mmcblk0p3 179:3    0     2M  0 part 
├─mmcblk0p4 179:4    0     2M  0 part 
├─mmcblk0p5 179:5    0   100M  0 part /mnt
└─mmcblk0p6 179:6    0   6,4G  0 part 
nvme0n1     259:0    0 953,9G  0 disk 
├─nvme0n1p1 259:1    0   499M  0 part /boot
└─nvme0n1p2 259:2    0 953,4G  0 part /
dima@kube-master ~ $ ls -la /boot
итого 70500
drwxr-xr-x  2 root root    16384 янв  1  1970 .
drwxr-xr-x 17 root root     4096 сен  3 21:37 ..
-rwxr-xr-x  1 root root    33164 ноя  5 03:53 bpi-r4.dtb
-rwxr-xr-x  1 root root 10195276 ноя  5 03:53 bpi-r4.itb
-rwxr-xr-x  1 root root    33164 ноя 22 19:54 bpi-r4-mod.dtb
-rwxr-xr-x  1 root root  9916196 ноя 22 22:19 bpi-r4-mod.itb
-rwxr-xr-x  1 root root 27650624 ноя  5 03:53 uImage_nodt
-rwxr-xr-x  1 root root 24308288 ноя 22 22:08 uImage_nodt_mod
dima@kube-master ~ $ cat /mnt/uEnv.txt 
bootmenu_default=3
console=ttyS0,115200 earlycon=uart8250,mmio32,0x11000000
device=nvme
partition=1
nvme_boot=0:1
nvme_root=/dev/nvme0n1p2
uImage=uImage_nodt
dtb=bpi-r4.dtb
fit=bpi-r4.itb

[frank-w]

Ah you do not have a rootfs on sdcard....how did you created a rootfs on nvme?

I expected you flashed full image with rootfs from my gdrive and not only the base image from uboot repo.

You have to create a vfat partition (p1) so on nvme and one for your rootfs (p2).

On sdcard you set "bootmenu_default=3". If your partitions are not different you do not need to set

nvme_boot=0:1
nvme_root=/dev/nvme0n1p2

Else change these.

So on nvme p1 you also create a uEnv.txt where you set fit to the itb which is also on nvme p1.

[Dmitrij-Solovjev]

I have updated my post above, apologies for the delay.
I am using this unusual boot configuration because, as far as I understand, it is not possible to boot directly from nvme. Therefore, I am using the image which I installed on an SD card, where I later added uEnv.txt and the .itb file (I'm not sure if it's needed on a bootable microSD card, maybe it's unnecessary and it will boot without it). On nvme, I get the kernel, device tree, and itb on the first partition, and the usual rootfs on the second.

Once I figure it out, I'll need to write detailed instructions for uboot, the Linux kernel, and the installation process in general.
Or at least links to the manuals.

[frank-w]

As i said above,the img from uboot repo releases is only a base image without kernel and rootfs. I have full images for sdcard on my gdrive.

And yes directly booting from nvme is not possible,you need the bootchain till uboot on mmc or spi and then you can load the other parts from nvme.

Just try the steps i wrote above

[Dmitrij-Solovjev]

dima@kube-master ~ $ uname -a
Linux kube-master 6.17.0-bpi-r4 #1 SMP Sat Nov 22 19:05:22 MSK 2025 aarch64 GNU/Linux
dima@kube-master ~ $ zcat /proc/config.gz | grep CONFIG_BR
# CONFIG_BRIDGE_NETFILTER is not set
# CONFIG_BRIDGE_NF_EBTABLES is not set
CONFIG_BRIDGE=y
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_BRIDGE_VLAN_FILTERING=y
# CONFIG_BRIDGE_MRP is not set
# CONFIG_BRIDGE_CFM is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
CONFIG_BRANCH_PROFILE_NONE=y

This is strange because I specified “y”, and the time stamp shows that the kernel is “new”. In any case, I solved the problem by simply changing the configuration to CONFIG_BRIDGE_NETFILTER=m, and the module file appeared in ./BPI-ROOT/lib/modules/6.17.0-bpi-r4/kernel/net/bridge/br_netfilter.ko. And I successfully loaded it.
Thank you very much for your help and time.

I suggest adding this configuration as the default value, as is done for desktops and servers.

[Dmitrij-Solovjev]

0. I try install docker:

 * Messages for package app-containers/docker-28.0.4:

 * Unable to find kernel sources at /usr/src/linux
 * Unable to calculate Linux Kernel version for build, attempting to use running version
 *   CONFIG_IP_NF_FILTER:	 is not set when it should be.
 *   CONFIG_IP_NF_RAW:	 is not set when it should be.
 *   CONFIG_IP_NF_TARGET_MASQUERADE:	 is not set when it should be.
 *   CONFIG_IP_NF_NAT:	 is not set when it should be.
 *   CONFIG_BLK_DEV_THROTTLING:	 is not set when it should be.
 *   CONFIG_CGROUP_HUGETLB:	 is not set when it should be.
 *   CONFIG_NET_CLS_CGROUP:	 is not set when it should be.
 *   CONFIG_IP_NF_TARGET_REDIRECT:	 is not set when it should be.
 *   CONFIG_IP_VS_NFCT:	 is not set when it should be.
 *   CONFIG_IP_VS_PROTO_TCP:	 is not set when it should be.
 *   CONFIG_IP_VS_PROTO_UDP:	 is not set when it should be.
 *   CONFIG_IP_VS_RR:	 is not set when it should be.
 *   CONFIG_VXLAN:	 is not set when it should be.
 *   CONFIG_XFRM_USER:	 is not set when it should be.
 *   CONFIG_INET_ESP:	 is not set when it should be.
 *   CONFIG_NF_NAT_TFTP:	 is not set when it should be.
 *   CONFIG_NF_CONNTRACK_TFTP:	 is not set when it should be.
 * Please check to make sure these options are set correctly.
 * Failure to do so may cause unexpected problems.

1. I add config by command

./build.sh defconfig

# for kubernetes
CONFIG_BRIDGE_NETFILTER=m

#not set
CONFIG_IP_NF_RAW=m
CONFIG_CGROUP_HUGETLB=y
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_NET_CLS_CGROUP=m
CONFIG_IP_VS_RR=m
CONFIG_VXLAN=m
CONFIG_XFRM_USER=m
CONFIG_INET_ESP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_CONNTRACK_TFTP=m
# config_not_module
CONFIG_BLK_DEV_THROTTLING=y
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y

# set by default
#CONFIG_IP_NF_FILTER=m
#CONFIG_IP_NF_TARGET_MASQUERADE=m
#CONFIG_IP_NF_NAT=m

2. I am importing config:
   My config file:

cat ./build.conf 
uploaduser=$USER
uploadserver=r3
uploaddir=/var/lib/tftp
#uploaduser=root
#uploadserver=192.168.0.11
#uploaddir=/boot/bananapi/bpi-r2/linux

builddir=../build
ramdisksize=8G
#numproc=8

#board=bpi-r2
#board=bpi-r64
#board=bpi-r2pro
#board=bpi-r3
#board=bpi-r3mini
board=bpi-r4
#board=bpi-r4pro
#board=bpi-r4lite

#r64 with rtl8367
#boardversion=v0.1
#r2pro with rtl8367
#boardversion=v00

#mainline uboot for r64 (old ATF) needs 64bit uImage
#uimagearch=arm64

#grep whitelist filter for adding modules to initramfs
ownmodules='mt76\|bluetooth'

command:

./build.sh importconfig
import a defconfig file
import mt7988a_bpi-r4_defconfig (arch/arm64/configs/mt7988a_bpi-r4_defconfig)
make[1]: Entering directory '/home/dima/Загрузки/build'
  GEN     Makefile
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf
#
# configuration written to .config
#
make[1]: Leaving directory '/home/dima/Загрузки/build'

3. After that, I try to find the flags I'm interested in, but I can't:

grep -E "CONFIG_IP_NF_FILTER|\
CONFIG_IP_NF_RAW|\
CONFIG_IP_NF_TARGET_MASQUERADE|\
CONFIG_IP_NF_NAT|\
CONFIG_BLK_DEV_THROTTLING|\
CONFIG_CGROUP_HUGETLB|\
CONFIG_NET_CLS_CGROUP|\
CONFIG_IP_NF_TARGET_REDIRECT|\
CONFIG_IP_VS_NFCT|\
CONFIG_IP_VS_PROTO_TCP|\
CONFIG_IP_VS_PROTO_UDP|\
CONFIG_IP_VS_RR|\
CONFIG_VXLAN|\
CONFIG_XFRM_USER|\
CONFIG_INET_ESP|\
CONFIG_NF_NAT_TFTP|\
CONFIG_NF_CONNTRACK_TFTP" ../build/.config

CONFIG_BLK_DEV_THROTTLING=y
CONFIG_XFRM_USER=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
# CONFIG_INET_ESPINTCP is not set
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_NFCT=y
CONFIG_NET_CLS_CGROUP=m
CONFIG_VXLAN=m

4. I tried manually editing the .config, but it seems the ./build.sh script overwrites it.

P.s.: It's interesting that some modules installed by default are also not included in the final config.

[frank-w]

when manually changing .config you have to run make oldconfig, but also with same builddir...i do not use this. I always add config-options to defconfig or set them in menuconfig (./build.sh config).

in menuconfig you can search with / for the symbol and then you see depencies...e.g.

  │ Symbol: IP_NF_RAW [=n]                                                                                                                                                                                          │  
  │ Type  : tristate                                                                                                                                                                                                │  
  │ Defined at net/ipv4/netfilter/Kconfig:301                                                                                                                                                                       │  
  │   Prompt: raw table support (required for NOTRACK/TRACE)                                                                                                                                                        │  
  │   Depends on: NET [=y] && INET [=y] && NETFILTER [=y] && IP_NF_IPTABLES [=m] && IP_NF_IPTABLES_LEGACY [=n]                                                                                                      │  
  │   Location:                                                                                                                                                                                                     │  
  │     -> Networking support (NET [=y])                                                                                                                                                                            │  
  │       -> Networking options                                                                                                                                                                                     │  
  │         -> Network packet filtering framework (Netfilter) (NETFILTER [=y])                                                                                                                                      │  
  │           -> IP: Netfilter Configuration                                                                                                                                                                        │  
  │ (1)         -> IP tables support (required for filtering/masq/NAT) (IP_NF_IPTABLES [=m])                                                                                                                        │  
  │               -> raw table support (required for NOTRACK/TRACE) (IP_NF_RAW [=n])          

here you see that IP_NF_RAW has depency on IP_NF_IPTABLES_LEGACY which is currently not set. so you have to set this option before the one you want. And this option has depency on NETFILTER_XTABLES_LEGACY (which is bool, not tristate so can only be "y", not "m")....

CONFIG_NETFILTER_XTABLES_LEGACY=y
CONFIG_IP_NF_IPTABLES_LEGACY=m
CONFIG_IP_NF_RAW=m

after changing to this, the CONFIG_IP_NF_RAW is set to m

some options are selected automaticly when other options are set or complete "default y" (always enabled except they are set to "n" like wlan_vendors), thats why they do not appear in defconfig, but in final config.

so the way to go is take your option-list and enter menuconfig (./build.sh config) after importconfig. now search the options and put depencies before the option that was previously not set. Look for tristate (can be module) or bool (n/y only).