ASU is down 12.09.2025

[lorand-horvath]

@aparcar Hi Paul, just wanted to let you know there's something wrong with the custom build server, it only returns init when requesting a build with the firmware-selector, there are many reports on the forum as well. If you can spare the time, would you please take a look? Thanks a lot!

[suppenkasper0815]

I can confirm this

`Status
Overview
Routing
Firewall
System Log
Processes
Channel Analysis
Realtime Graphs
WireGuard
System
System
Administration
Software
Startup
Scheduled Tasks
Attended Sysupgrade
LED Configuration
Backup / Flash Firmware
Reboot
Services
AdBlock Fast
Network
Interfaces
Wireless
Routing
DHCP and DNS
Diagnostics
Firewall
VPN
OpenVPN
Log out
Paused
Overview
Configuration
Attended Sysupgrade
The attended sysupgrade service allows to upgrade vanilla and custom firmware images easily.

This is done by building a new firmware on demand via an online service.

Currently running: 23.05.5 - r24106-10cc5fcd00

Search for firmware upgrade

Powered by LuCI openwrt-23.05 branch (git-25.108.47089-90daf5b) / OpenWrt 23.05.5 (r24106-10cc5fcd00)
Error building the firmware image
Server response: init

Please report the error message and request
Request Data:

{
"url": "https://sysupgrade.openwrt.org",
"branch": "23.05",
"revision": "r24106-10cc5fcd00",
"efi": null,
"advanced_mode": "0",
"request_hash": "",
"sha256_unsigned": "",
"client": "luci/git-25.108.47089-90daf5b",
"packages": {
"luci-proto-wireguard": "git-25.108.47089-90daf5b",
"iwinfo": "2023-07-01-ca79f641-1",
"liblucihttp": "2023-03-15-9b5b683f-1",
"cgi-io": "2022-08-10-901b0f04-21",
"libc": "1.2.4-4",
"avahi-dbus-daemon": "0.8-8",
"opkg": "2022-02-24-d038e5b6-2",
"luci-app-opkg": "git-25.108.47089-90daf5b",
"ubus": "2023-06-05-f787c97b-1",
"libuci": "2023-08-10-5781664d-1",
"iperf3-ssl": "3.17.1-4",
"rpcd": "2023-07-01-c07ab2f9-1",
"busybox": "1.36.1-1",
"libiwinfo": "2023-07-01-ca79f641-1",
"kmod-nft-fib": "5.15.167-1",
"wpad-mesh-mbedtls": "2023-09-08-e5ccbfc6-8",
"kmod-nfnetlink": "5.15.167-1",
"kmod-crypto-hash": "5.15.167-1",
"kmod-nf-reject6": "5.15.167-1",
"luci-mod-system": "git-25.108.47089-90daf5b",
"kmod-nf-flow": "5.15.167-1",
"libnl-tiny": "2023-07-27-bc92a280-1",
"kmod-lib-crc-ccitt": "5.15.167-1",
"ucode-mod-uloop": "2024-07-11-1a8a0bcf-3",
"px5g-mbedtls": "10",
"getrandom": "2022-08-13-4c7b720b-2",
"ucode-mod-ubus": "2024-07-11-1a8a0bcf-3",
"luci-theme-bootstrap": "git-25.108.47089-90daf5b",
"kmod-pppoe": "5.15.167-1",
"kmod-pppox": "5.15.167-1",
"kmod-nf-reject": "5.15.167-1",
"procd-ujail": "2023-06-25-2db83655-2",
"base-files": "1562-r24106-10cc5fcd00",
"kmod-nf-nat": "5.15.167-1",
"kmod-crypto-crc32c": "5.15.167-1",
"ucode-mod-uci": "2024-07-11-1a8a0bcf-3",
"kmod-mt7615e": "5.15.167+2024-04-03-1e336a85-1",
"libustream-mbedtls": "2023-02-25-498f6e26-1",
"netifd": "2024-01-04-c18cc79d-2",
"firewall4": "2023-09-01-598d9fbb-1",
"dnsmasq": "2.90-2",
"procd": "2023-06-25-2db83655-2",
"ubusd": "2023-06-05-f787c97b-1",
"qrencode": "4.1.1-1",
"kmod-crypto-acompress": "5.15.167-1",
"rpcd-mod-ucode": "2023-07-01-c07ab2f9-1",
"ucode-mod-math": "2024-07-11-1a8a0bcf-3",
"kmod-lib-crc32c": "5.15.167-1",
"luci-mod-status": "git-25.108.47089-90daf5b",
"libmbedtls": "2.28.9-1",
"kmod-nft-nat": "5.15.167-1",
"luci-app-firewall": "git-25.108.47089-90daf5b",
"libubus": "2023-06-05-f787c97b-1",
"libmnl": "1.0.5-1",
"jansson": "2.14-3",
"tcpdump": "4.99.4-1",
"ubi-utils": "2.1.5-1",
"odhcp6c": "2023-05-12-bcd28363-20",
"fstools": "2023-02-28-bfe882d5-1",
"uclient-fetch": "2023-04-13-007d9454-1",
"uci": "2023-08-10-5781664d-1",
"ucode-mod-fs": "2024-07-11-1a8a0bcf-3",
"luci-ssl": "git-25.108.47089-90daf5b",
"dropbear": "2022.82-6",
"libnftnl": "1.2.6-1",
"rpcd-mod-file": "2023-07-01-c07ab2f9-1",
"mtd": "26",
"odhcpd-ipv6only": "2023-10-24-d8118f6e-1",
"procd-seccomp": "2023-06-25-2db83655-2",
"ucode-mod-nl80211": "2024-07-11-1a8a0bcf-3",
"libjson-script": "2023-05-23-75a3b870-1",
"rssileds": "4",
"libiwinfo-data": "2023-07-01-ca79f641-1",
"ucode": "2024-07-11-1a8a0bcf-3",
"rpcd-mod-luci": "20240305-1",
"kmod-nf-log": "5.15.167-1",
"urandom-seed": "3",
"luci-proto-ppp": "git-25.108.47089-90daf5b",
"luci-mod-admin-full": "git-25.108.47089-90daf5b",
"ppp": "2.4.9.git-2021-01-04-4",
"luci-base": "git-25.108.47089-90daf5b",
"socat": "1.7.4.4-1",
"kmod-leds-gpio": "5.15.167-1",
"kmod-gpio-button-hotplug": "5.15.167-3",
"logd": "2022-08-13-4c7b720b-2",
"kmod-nf-log6": "5.15.167-1",
"python3": "3.11.7-1",
"wireguard-tools": "1.0.20210914-2",
"luci-proto-ipv6": "git-25.108.47089-90daf5b",
"openwrt-keyring": "2022-03-25-62471e69-2",
"luci-app-attendedsysupgrade": "git-25.108.47089-90daf5b",
"jshn": "2023-05-23-75a3b870-1",
"nftables-json": "1.0.8-1",
"kmod-ppp": "5.15.167-1",
"kmod-nft-offload": "5.15.167-1",
"openvpn-easy-rsa": "3.0.8-4",
"igmpproxy": "0.4-1",
"libjson-c": "0.16-3",
"luci-proto-relay": "git-25.158.57731-16c81f5",
"uhttpd": "2023-06-25-34a8a74d-2",
"kmod-nf-conntrack": "5.15.167-1",
"wg-installer-server": "28",
"usign": "2020-05-23-f1f65026-1",
"libblobmsg-json": "2023-05-23-75a3b870-1",
"kmod-mt7915-firmware": "5.15.167+2024-04-03-1e336a85-1",
"htop": "3.3.0-1",
"ucode-mod-rtnl": "2024-07-11-1a8a0bcf-3",
"ca-bundle": "20230311-1",
"libuclient": "2023-04-13-007d9454-1",
"ucode-mod-html": "1",
"luci": "git-25.108.47089-90daf5b",
"kmod-nf-conntrack6": "5.15.167-1",
"luci-light": "git-25.108.47089-90daf5b",
"kmod-lib-lzo": "5.15.167-1",
"luci-app-adblock-fast": "1.1.2",
"ubox": "2022-08-13-4c7b720b-2",
"kernel": "5.15.167-1-144de9e5c1a8813b724b14faa054d9f0",
"rpcd-mod-iwinfo": "2023-07-01-c07ab2f9-1",
"luci-mod-network": "git-25.108.47089-90daf5b",
"kmod-nft-core": "5.15.167-1",
"libucode": "2024-07-11-1a8a0bcf-3",
"uhttpd-mod-ubus": "2023-06-25-34a8a74d-2",
"fwtool": "2019-11-12-8f7fe925-1",
"jsonfilter": "2024-01-23-594cfa86-1",
"omcproxy": "2021-11-04-bfba2aa7-9",
"liblucihttp-ucode": "2023-03-15-9b5b683f-1",
"libubox": "2023-05-23-75a3b870-1",
"hostapd-common": "2023-09-08-e5ccbfc6-8",
"kmod-ipt-offload": "5.15.167-1",
"openvpn-mbedtls": "2.5.8-5",
"urngd": "2023-11-01-44365eb1-1",
"kmod-slhc": "5.15.167-1",
"iftop": "2018-10-03-77901c8c-2",
"rpcd-mod-rrdns": "20170710",
"ppp-mod-pppoe": "2.4.9.git-2021-01-04-4",
"luci-app-openvpn": "git-25.108.47089-90daf5b"
},
"profile": "dlink,dap-x1860-a1",
"target": "ramips/mt7621",
"version": "23.05.5",
"diff_packages": true,
"filesystem": "squashfs"
}
Close`

[efahl]

Looks like disk full:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/rq/worker.py", line 1688, in perform_job
    return_value = job.perform()
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1371, in perform
    self._result = self._execute()
                   ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1405, in _execute
    result = self.func(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/asu/build.py", line 180, in build
    container.start()
  File "/usr/local/lib/python3.12/site-packages/podman/domain/containers.py", line 420, in start
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/podman/api/client.py", line 82, in raise_for_status
    raise APIError(cause, response=self._response, explanation=message)
podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (crun: create keyring ..: Disk quota exceeded: OCI runtime error)

[efahl]

Digging deeper, it's not disk space, it's keyring space. I can see on my local server some of the important stats, but I have not yet reproduced the issue. Various github issues and blogs talk about increasing either the maxbytes or maxkeys. containers/podman#23784 mentions keyring=false in the podman config, but notes it doesn't do anything.

$ cat /proc/keys | wc -l
10

$ sysctl kernel.keys
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.persistent_keyring_expiry = 259200
kernel.keys.root_maxbytes = 25000000
kernel.keys.root_maxkeys = 1000000

I can't seem to make the number of keys rise above 10 (well below the 200 max); I've shot off a bunch of build requests on my server and it doesn't budge.

[efahl]

Aha, I figured out how to control this. While running this in one window:

watch 'cat /proc/keys | wc -l'

I could launch off a dozen or so owut build requests from one of my routers. This caused the number of keys to jump up from 10 to 13-14 for the duration of the builds, then return to 10.

When I killed all the podman services on the machine (podman-compose down), then the number drops to a baseline of 7, jumps back to 10 on podman-compose up.

Creating a new ~/.config/containers/containers.conf results in 8 keys at startup, and no bump in usage during the builds.

[containers]
keyring = false

@aparcar I'm not sure of the implications of this beyond reducing the keyring usage, so I'll defer as to whether it's useful or a real solution or not...

[lorand-horvath]

@aparcar Thanks for resolving the issue!