pyPowerwall Proxy Stopped (getting more frequent) #506

Green-Pool · 2024-07-25T23:50:58Z

Green-Pool
Jul 25, 2024

I am still having trouble figuring out why my dashboard stops data collection. (This is a follow on question from #498)

Just to recap the installation. It is running on a virtual Linux Mint install. Co-hosted on that VM is my own LAMP install that is querying the Fronius inverters and doing separate weather data collection. I don't think the two installs are clashing, but I cannot prove that conclusively.

This is the tail end of the logs from a recent reinstall. The warnings about not being able to collect vitals is expected.

07/18/2024 11:48:21 AM [proxy] [INFO] pyPowerwall [0.10.6] Proxy Server [t63] - HTTP Port 8675
07/18/2024 11:48:21 AM [proxy] [INFO] pyPowerwall Proxy Started
07/18/2024 11:48:22 AM [proxy] [INFO] pyPowerwall Proxy Server - Local Mode
07/18/2024 11:48:22 AM [proxy] [INFO] Connected to Energy Gateway 192.168.2.7 (Home battery)
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.2.7/api/devices/vitals
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.2.7/api/devices/vitals
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.2.7/api/devices/vitals
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.2.7/api/devices/vitals
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] 404 Powerwall API not found at https://192.168.2.7/api/devices/vitals
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 242000 detected - Does not support vitals API - disabling.
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 242000 detected - Does not support vitals API - disabling.
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 242000 detected - Does not support vitals API - disabling.
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 242000 detected - Does not support vitals API - disabling.
07/18/2024 11:48:45 AM [pypowerwall.local.pypowerwall_local] [ERROR] Firmware 242000 detected - Does not support vitals API - disabling.

Powerwall seems to work fine for a period, then stops collecting data, shortly after midnight (each time). There is no other clues that I can see, except the very next line of the log file says ...

07/23/2024 12:37:44 AM [proxy] [INFO] pyPowerwall Proxy Stopped

Running a curl command gives the following output.

curl -i http://localhost:8675/stats
curl: (7) Failed to connect to localhost port 8675 after 0 ms: Connection refused

curl -i http://localhost:8675/soe
curl: (7) Failed to connect to localhost port 8675 after 0 ms: Connection refused

Running the verify script produces this result

Verify Powerwall-Dashboard 4.4.6 on Linux -

Checking pypowerwall

Config File pypowerwall.env: GOOD

Container (pypowerwall): ERROR: Stopped

Version: Unknown

Powerwall State: ERROR: Not Connected - Firmware: Unknown

Powerwall Gateway TEDAPI: Not Available (192.168.91.1)

Cloud Mode: NO

Checking telegraf

Config File telegraf.conf: GOOD

Local Config File telegraf.local: GOOD

Container (telegraf): GOOD

Version: Telegraf 1.28.2 (git: HEAD@8d9cf395)

Checking influxdb

Config File influxdb.conf: GOOD

Environment File influxdb.env: GOOD

Container (influxdb): GOOD

Service (port 8086): GOOD

Filesystem (./influxdb): GOOD

Version: InfluxDB shell version: 1.8.10

Checking grafana

Config File grafana.env: GOOD

Container (grafana): GOOD

Service (port 9000): GOOD

Filesystem (./grafana): GOOD

Version: Grafana CLI version 9.1.2

Checking weather411

Container (weather411): ERROR: Stopped

Version: Unknown

One or more tests failed.

Whatever the problem is, it seems to be taking down the weather container as well.

Re-run the upgrade script and everything starts working again. The verify script produces the following result.

Verify Powerwall-Dashboard 4.4.6 on Linux -

This script will attempt to verify all the services needed to run
Powerwall-Dashboard. Use this output when you open an issue for help:
https://github.com/jasonacox/Powerwall-Dashboard/issues/new

Checking pypowerwall

Config File pypowerwall.env: GOOD

Container (pypowerwall): GOOD

Service (port 8675): GOOD

Version: 0.10.9 Proxy t63

Powerwall State: CONNECTED - Firmware: 24.20.0

Site Name: Home battery

Powerwall Gateway TEDAPI: Not Available (192.168.91.1)

Cloud Mode: NO

Checking telegraf

Config File telegraf.conf: GOOD

Local Config File telegraf.local: GOOD

Container (telegraf): GOOD

Version: Telegraf 1.28.2 (git: HEAD@8d9cf395)

Checking influxdb

Config File influxdb.conf: GOOD

Environment File influxdb.env: GOOD

Container (influxdb): GOOD

Service (port 8086): GOOD

Filesystem (./influxdb): GOOD

Version: InfluxDB shell version: 1.8.10

Checking grafana

Config File grafana.env: GOOD

Container (grafana): GOOD

Service (port 9000): GOOD

Filesystem (./grafana): GOOD

Version: Grafana CLI version 9.1.2

Checking weather411

Container (weather411): GOOD

Service (port 8676): GOOD

Weather: {"temperature": 16.64}

Version: 0.2.3

All tests succeeded.

There is one other bit of info. I suspect it is completely unrelated and a red herring for this particular problem. My Security Onion installation (different VM on the same server) is occasionally generating Suricata alerts against outbound traffic from the powerwall VM server to the client PC running the web page. Security Onion is sitting on the LAN with a mirrored traffic feed. Here is an extract of the alert.

"signature_id":2101201,"rev":11,"signature":"GPL WEB_SERVER 403 Forbidden"
"files":[{"filename":"/api/datasources/proxy/1/query"

This is not happening all of the time. It comes in little batches of about 10 seconds duration. Then just stops. I do think it is a red hearing, but, shrug ...

Thoughts as to the next place to look for logs/clues?

Answered by jasonacox

Jul 27, 2024

With the docker and containerd errors in the logs, it does seem like something is unhealthy about the docker setup. You could paste all of those logs in an LLM like ChatGPT to see what it says.

As for re-installing or creating a new VM, my recommendation would be to:

Create new VM
Copy over (e.g. tar) the entire Powerwall-Dashboard folder (stop dashbaord ./compose-dash.sh stop)
Ensure all files are owned by the user (otherwise chown -R myuser Powerwall-Dashboard/*)
Run setup.sh - The inflxudb container should mount and use he old database you copied over.

View full answer

jasonacox · 2024-07-26T01:36:12Z

jasonacox
Jul 26, 2024
Maintainer

07/23/2024 12:37:44 AM [proxy] [INFO] pyPowerwall Proxy Stopped

This is key. This is caused by pypowerwall getting a KeyboardInterurpt (^C) or SystemExit interrupt. There could be other failure reasons, but in any case I would ask:

Why is the container not restarting? The docker compose configuration is set for restart: unless-stopped - The only way it would not restart is if something on your system is telling it to stop (e.g docker stop pypowerwall). I don't know enough about mint to understand your setup, but I would start by investigating why docker is not restarting when the application stops.
Why is pypowerwall stopping? Do you see anything else in the logs? I don't think so based on your note about the above being the "very next line". If there are no error traces, that indicates that it is indeed getting a signal to stop, either from some kernel level signal or the docker system.

If I were you, I would set up another installation of the Powerwall Dashboard on a non-Mint system to see if this is specific to that VM setup. I would look at the Mint system logs to see if it shows anything.

Also, confirm that the docker compose shows the correct restart configuration.

9 replies

Green-Pool Jul 26, 2024
Author

Humm, OK. Never even suspected that it was something stopping pypowerwall. The web interface is still working and still draws graphs up to the point it stops.

Thanks for the tip. I will go looking in that direction and report back.

jasonacox Jul 26, 2024
Maintainer

I know it can be frustrating! I hope you can find the cause.

The web interface is Grafana which is a different container. The pypowerwall container is an API proxy that the telegraf container uses to store the metrics in influxdb. Grafana reads influxdb to display the graphs.

As a note, to start the container stack you can use ./compose-dash.sh up -d instead of re-runing upgrade.

You also mention weather411 stops. That is odd too. docker logs weather411 may give you a clue as to why, but I suspect the same thing that is stopping pypowerwall is stopping it. Other things to check:

Security tools stopping / killing these apps?
VM having memory issues and killing processes?

Green-Pool Jul 26, 2024
Author

Just by way of background. The VM hosting the docker for pypowerwall has 12 cores available to it and 10 GB of memory. As mentioned before, it also host a LAMP server that queries the Fronius inverters and process weather data.

Most of the time, it chuggs away with less than 5% on the CPU, except for a burst of data processing every 5 minutes where the CPU climbs up to 50%. Memory in use is about 2GB, climbs to 5GB when data processing , and drops back to 2GB.

Thinking about this a bit more ... I have migrated this particular VM three times now, across three different hardware platforms. It certainly isn't fresh. But I do cringe at the thought of exporting the pypowerwall data and importing it back into a fresh install.

I found docker in the Linux Mint logs easily enough. But, I have little idea what I am looking at.

The log extract starting at Jul 23 00:37:41 systemd: Stopping Docker Application Container Engine... (colour coded red) is associated with the time that pypowerwall stopped in the early hours of Tuesday morning. It restarted after I ran the upgrade script.

The part of the log coloured green is when docker stopped again yesterday (almost the same time 00:32).

There are two more log extracts below. This time, just looking at all the events in the syslog when the pypowerwall stopped responding

It may be coincidence, but docker stopping appears to coincide with the Linux Mint automatic update manager running.

Extract from the logs filtered on the keyword "docker".

Covers both events were docker stopped and didn't restart. (Time stamp is descending.)

Thoughts?

Jul 25 19:55:34 dockerd: time="2024-07-25T19:55:34.487242154+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 25 19:55:27 systemd: var-lib-docker-overlay2-f75789661d0f35950cb6a80f54ca6f761f2b25e0a36927910bfcdd1b54981836-merged.mount: Deactivated successfully.
Jul 25 19:55:17 dockerd: time="2024-07-25T19:55:17.275597350+10:00" level=warning msg="Failed to retrieve containerd version" error="Canceled: context canceled"
Jul 25 19:55:07 systemd: var-lib-docker-overlay2-6623eb48bd8021bccb1e6ea8d2839fd496541a29dee04cefb84953eb488e5d81-merged.mount: Deactivated successfully.
Jul 25 19:55:06 dockerd: time="2024-07-25T19:55:06.765103371+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52 daemonShuttingDown=false error="restart canceled" execDuration=11.992529352s exitStatus="{0 2024-07-25 09:55:06.705737651 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 25 19:55:06 systemd: docker-ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52.scope: Consumed 1.464s CPU time.
Jul 25 19:55:01 dockerd: time="2024-07-25T19:55:01.545241945+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb daemonShuttingDown=false error="restart canceled" execDuration=19h22m16.273041222s exitStatus="{0 2024-07-25 09:55:01.443243109 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 25 19:55:01 systemd: docker-c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.scope: Consumed 6min 7.896s CPU time.
Jul 25 19:55:00 dockerd: time="2024-07-25T19:55:00.724493995+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3 daemonShuttingDown=false error="restart canceled" execDuration=5.954803339s exitStatus="{0 2024-07-25 09:55:00.567232829 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 25 19:55:00 systemd: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Consumed 1.248s CPU time.
Jul 25 19:54:54 dockerd: time="2024-07-25T19:54:54.024528644+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 25 19:09:47 systemd: run-docker-runtime\x2drunc-moby-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067-runc.rS5oBr.mount: Deactivated successfully.
Jul 25 19:09:46 dockerd: time="2024-07-25T19:09:46.739156440+10:00" level=warning msg="failed to close stdin: NotFound: process does not exist 15679dc4e0c680d42e202d4ecb56d94bbce8008f72d0d0973654c9b923a5c3b5: not found"
Jul 25 19:09:46 systemd: run-docker-runtime\x2drunc-moby-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e-runc.Gwl1CG.mount: Deactivated successfully.
Jul 25 19:05:34 dbus-daemon: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.295' (uid=1000 pid=1918335 comm="xed /home/weatherwall/Desktop/Powerwall docker err" label="unconfined")
Jul 25 12:41:18 docker-registry: time="2024-07-25T12:41:18.196050572+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
+ Jul 25 00:32:47 systemd: Started Docker Application Container Engine.
+ Jul 25 00:32:47 dockerd: time="2024-07-25T00:32:47.074508696+10:00" level=info msg="API listen on /run/docker.sock"
+ Jul 25 00:32:46 systemd: var-lib-docker-overlay2-opaque\x2dbug\x2dcheck943418215-merged.mount: Deactivated successfully.
+ Jul 25 00:32:46 dockerd: time="2024-07-25T00:32:46.183320629+10:00" level=info msg="Loading containers: done."
+ Jul 25 00:32:45 systemd: run-docker-runtime\x2drunc-moby-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e-runc.s5KauY.mount: Deactivated successfully.
+ Jul 25 00:32:45 dockerd: time="2024-07-25T00:32:45.262993436+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
+ Jul 25 00:32:45 systemd: run-docker-netns-2ed445e5cc85.mount: Deactivated successfully.
+ Jul 25 00:32:45 dockerd: time="2024-07-25T00:32:45.000354014+10:00" level=warning msg="error locating sandbox id 531a9d4cc9994d4b1d1115949acf10e82eb6680ded70eeb4f4fda81d26b7b1ec: sandbox 531a9d4cc9994d4b1d1115949acf10e82eb6680ded70eeb4f4fda81d26b7b1ec not found"
+ Jul 25 00:32:44 systemd: run-docker-netns-ae83477f1a74.mount: Deactivated successfully.
+ Jul 25 00:32:44 dockerd: time="2024-07-25T00:32:44.020873635+10:00" level=info msg="Removing stale sandbox aba8b6b71cfbec1e12175e6de0cb888574b27867c613729d6723a537d1d0b5c1 (c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e)"
+ Jul 25 00:32:44 systemd: run-docker-netns-aba8b6b71cfb.mount: Deactivated successfully.
+ Jul 25 00:32:42 dockerd: time="2024-07-25T00:32:42.869648185+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52 daemonShuttingDown=false error="restart canceled" execDuration=40h13m28.197713147s exitStatus="{0 2024-07-24 14:32:42.781355043 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
+ Jul 25 00:32:42 systemd: docker-ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52.scope: Consumed 28.398s CPU time.
+ Jul 25 00:32:38 dockerd: time="2024-07-25T00:32:38.426419744+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3 daemonShuttingDown=false error="restart canceled" execDuration=40h13m23.7279368s exitStatus="{0 2024-07-24 14:32:38.255668662 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
+ Jul 25 00:32:38 systemd: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Consumed 19min 8.528s CPU time.
+ Jul 25 00:32:38 systemd: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Consumed 19min 8.528s CPU time.
+ Jul 25 00:32:38 systemd: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Deactivated successfully.
+ Jul 25 00:32:36 systemd: var-lib-docker-overlay2-56166d0d12fece68ade75b0c73a9baa76a3abc113f92c4dc5f361ed39b007b88-merged.mount: Deactivated successfully.
+ Jul 25 00:32:36 systemd: var-lib-docker-overlay2-e42b8ab59062f99e82bc54bf9bf69b37295065e5f55b3921ae787402d4bf3512-merged.mount: Deactivated successfully.
+ Jul 25 00:32:36 systemd: var-lib-docker-overlay2-2bad425d953bf971f6ec7b2c81a0ac0804fc13f2db9d80e4c8ea836be82170c3-merged.mount: Deactivated successfully.
+ Jul 25 00:32:36 dockerd: time="2024-07-25T00:32:36.473437663+10:00" level=info msg="Loading containers: start."
+ Jul 25 00:32:36 dockerd: time="2024-07-25T00:32:36.473437663+10:00" level=info msg="Loading containers: start."
+ Jul 25 00:32:30 dockerd: time="2024-07-25T00:32:30.469351611+10:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
+ Jul 25 00:32:29 dockerd: time="2024-07-25T00:32:29.060075342+10:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
+ Jul 25 00:32:29 dockerd: time="2024-07-25T00:32:29.029080349+10:00" level=info msg="Starting up"
+ Jul 25 00:32:28 systemd: Starting Docker Application Container Engine...
+ Jul 25 00:32:28 systemd: Starting Docker Application Container Engine...
+ Jul 25 00:32:28 systemd: docker.service: Found left-over process 1321179 (docker-proxy) in control group while starting unit. Ignoring.
+ Jul 25 00:32:28 systemd: docker.service: Found left-over process 1321156 (docker-proxy) in control group while starting unit. Ignoring.
+ Jul 25 00:32:28 systemd: docker.service: Found left-over process 1321125 (docker-proxy) in control group while starting unit. Ignoring.
+ Jul 25 00:32:28 systemd: docker.service: Found left-over process 1321111 (docker-proxy) in control group while starting unit. Ignoring.
+ Jul 25 00:32:28 systemd: Listening on Docker Socket for the API.
+ Jul 25 00:32:28 systemd: Starting Docker Socket for the API...
+ Jul 25 00:32:28 systemd: Stopping Docker Socket for the API...
+ Jul 25 00:32:28 systemd: Closed Docker Socket for the API.
+ Jul 25 00:32:28 systemd: docker.socket: Deactivated successfully.
+ Jul 25 00:32:28 systemd: docker.service: Consumed 5min 1.816s CPU time.
+ Jul 25 00:32:28 systemd: Stopped Docker Application Container Engine.
+ Jul 25 00:32:28 systemd: docker.service: Unit process 1321179 (docker-proxy) remains running after unit stopped.
+ Jul 25 00:32:28 systemd: docker.service: Unit process 1321156 (docker-proxy) remains running after unit stopped.
+ Jul 25 00:32:28 systemd: docker.service: Unit process 1321125 (docker-proxy) remains running after unit stopped.
+ Jul 25 00:32:28 systemd: docker.service: Unit process 1321111 (docker-proxy) remains running after unit stopped.
+ Jul 25 00:32:28 systemd: docker.service: Failed with result 'timeout'.
+ Jul 25 00:32:28 systemd: docker.service: Main process exited, code=killed, status=9/KILL
+ Jul 25 00:32:28 systemd: docker.service: Killing process 1261710 (dockerd) with signal SIGKILL.
+ Jul 25 00:32:28 systemd: docker.service: State 'stop-sigterm' timed out. Killing.
+ Jul 25 00:32:27 systemd: run-docker-netns-6706134a2515.mount: Deactivated successfully.
+ Jul 25 00:32:23 dockerd: time="2024-07-25T00:32:23.832366913+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e daemonShuttingDown=true error="restart canceled" execDuration=47h54m11.200216109s exitStatus="{0 2024-07-24 14:32:21.251899731 +0000 UTC}" hasBeenManuallyStopped=false restartCount=0
+ Jul 25 00:32:22 systemd: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Consumed 5min 59.850s CPU time.
+ Jul 25 00:32:22 systemd: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Consumed 5min 59.850s CPU time.
+ Jul 25 00:32:22 systemd: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Deactivated successfully.
+ Jul 25 00:32:21 systemd: docker-c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.scope: Consumed 18min 21.889s CPU time.
+ Jul 25 00:32:21 systemd: docker-c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.scope: Deactivated successfully.
+ Jul 25 00:32:20 systemd: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Consumed 1h 9min 57.607s CPU time.
+ Jul 25 00:32:20 systemd: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Deactivated successfully.
+ Jul 25 00:32:17 dockerd: time="2024-07-25T00:32:17.930784603+10:00" level=info msg="Processing signal 'terminated'"
+ Jul 25 00:32:17 systemd: Stopping Docker Application Container Engine...
Jul 24 12:41:18 docker-registry: time="2024-07-24T12:41:18.092060952+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 24 12:41:18 docker-registry: time="2024-07-24T12:41:18.092060952+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 24 12:41:18 docker-registry: time="2024-07-24T12:41:18.092005008+10:00" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
Jul 24 12:41:18 docker-registry: time="2024-07-24T12:41:18.073382497+10:00" level=info msg="PurgeUploads starting: olderThan=2024-07-17 12:41:17.996502798 +1000 AEST m=-83023.722479226, actuallyDelete=true"
Jul 24 10:48:37 dockerd: time="2024-07-24T10:48:37.344582901+10:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:127.0.0.1:50747" dns-server="udp:127.0.0.53:53" error="read udp 127.0.0.1:50747->127.0.0.53:53: i/o timeout" question=";grafana.com.\tIN\t A"
Jul 23 12:41:17 docker-registry: time="2024-07-23T12:41:17.976871121+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 23 12:41:17 docker-registry: time="2024-07-23T12:41:17.976871121+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 23 12:41:17 docker-registry: time="2024-07-23T12:41:17.976566656+10:00" level=info msg="Purge uploads finished.  Num deleted=0, num errors=1"
Jul 23 12:41:17 docker-registry: time="2024-07-23T12:41:17.945931652+10:00" level=info msg="PurgeUploads starting: olderThan=2024-07-16 12:41:17.871764579 +1000 AEST m=-169423.847217444, actuallyDelete=true"
Jul 23 08:19:15 systemd: run-docker-runtime\x2drunc-moby-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3-runc.NSyzZf.mount: Deactivated successfully.
Jul 23 08:19:15 dockerd: time="2024-07-23T08:19:15.231808950+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 23 08:19:14 systemd: run-docker-runtime\x2drunc-moby-ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52-runc.kxiIiy.mount: Deactivated successfully.
Jul 23 08:19:14 dockerd: time="2024-07-23T08:19:14.556130597+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 23 08:19:14 systemd: var-lib-docker-overlay2-6623eb48bd8021bccb1e6ea8d2839fd496541a29dee04cefb84953eb488e5d81\x2dinit-merged.mount: Deactivated successfully.
Jul 23 08:19:13 dockerd: time="2024-07-23T08:19:13.228501704+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696 daemonShuttingDown=false error="restart canceled" execDuration=16.40288508s exitStatus="{0 2024-07-22 22:19:13.148076818 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 23 08:19:13 systemd: docker-c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696.scope: Consumed 1.563s CPU time.
Jul 23 08:19:08 dockerd: time="2024-07-23T08:19:08.607042395+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=9f871e2449a4ba9171e71f64a76c12686a17a98d0fce247e03dc2d7f3b144a8d daemonShuttingDown=false error="restart canceled" execDuration=8.412387132s exitStatus="{0 2024-07-22 22:19:08.521068792 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 23 08:19:08 systemd: docker-9f871e2449a4ba9171e71f64a76c12686a17a98d0fce247e03dc2d7f3b144a8d.scope: Deactivated successfully.
Jul 23 08:19:07 dockerd: time="2024-07-23T08:19:07.939783060+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=d74a6520273af4e16aa7ef3fb26f7785379c202355e862ec30440ae42bd39f13 daemonShuttingDown=false error="restart canceled" execDuration=11.159251845s exitStatus="{0 2024-07-22 22:19:07.817843905 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 23 08:19:07 systemd: docker-d74a6520273af4e16aa7ef3fb26f7785379c202355e862ec30440ae42bd39f13.scope: Deactivated successfully.
Jul 23 08:18:59 dockerd: time="2024-07-23T08:18:59.560729130+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 23 08:18:57 systemd: run-docker-runtime\x2drunc-moby-d74a6520273af4e16aa7ef3fb26f7785379c202355e862ec30440ae42bd39f13-runc.HzKxUZ.mount: Deactivated successfully.
Jul 23 08:18:55 dockerd: time="2024-07-23T08:18:55.998787107+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 23 08:18:54 systemd: var-lib-docker-overlay2-073b6d59c5d1ae98214eb5e2ace94ac6914cbf07195f806664f5339e652f86e9-merged.mount: Deactivated successfully.
Jul 23 08:18:53 dockerd: time="2024-07-23T08:18:53.878004854+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156 daemonShuttingDown=false error="restart canceled" execDuration=7h40m41.265433748s exitStatus="{0 2024-07-22 22:18:52.81297554 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
Jul 23 08:18:52 systemd: docker-870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156.scope: Consumed 2min 32.051s CPU time.
Jul 23 08:14:21 dockerd: time="2024-07-23T08:14:21.492590902+10:00" level=error msg="Error setting up exec command in container pypowerwall: container ee2c02ee50f1dcd388a878607051bf314182e98aed461ba9b9022d29583ff61d is not running"
- Jul 23 00:38:19 systemd: Started Docker Application Container Engine.
- Jul 23 00:38:19 dockerd: time="2024-07-23T00:38:19.789176353+10:00" level=info msg="API listen on /run/docker.sock"
- Jul 23 00:38:11 systemd: run-docker-netns-f1e93de42a2b.mount: Deactivated successfully.
- Jul 23 00:38:10 dockerd: time="2024-07-23T00:38:10.867818131+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 3110afa20a18a1bb70c6baec228770caac1bb241cbbd951caae4f92d9a240382], retrying...."
- Jul 23 00:38:10 systemd: run-docker-netns-bc0b4ecff182.mount: Deactivated successfully.
- Jul 23 00:38:10 dockerd: time="2024-07-23T00:38:10.363797406+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 ff6f27a7f06d46e91e5f0574a4c444ea3aac5204850fd96a9baf540db397bf1b], retrying...."
- Jul 23 00:38:10 systemd: run-docker-netns-8998a8fef10e.mount: Deactivated successfully.
- Jul 23 00:38:09 dockerd: time="2024-07-23T00:38:09.821254985+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 ebebf359f4ec8bd02c4457b606241cda2053c77e994d8eb2d1e3396269f1360d], retrying...."
- Jul 23 00:38:09 systemd: run-docker-netns-4833ed4962f2.mount: Deactivated successfully.
- Jul 23 00:38:09 dockerd: time="2024-07-23T00:38:09.310857730+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 983a8c6b08d6a7c6e44f38e6a52fd81906122225072e97dd8d320408d060098b], retrying...."
- Jul 23 00:38:09 systemd: run-docker-netns-374b85024c72.mount: Deactivated successfully.
- Jul 23 00:38:07 dockerd: time="2024-07-23T00:38:07.626907464+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696 daemonShuttingDown=false error="restart canceled" execDuration=108h50m17.705618979s exitStatus="{0 2024-07-22 14:38:07.484905437 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
- Jul 23 00:38:07 systemd: docker-c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696.scope: Consumed 1min 17.026s CPU time.
- Jul 23 00:38:01 dockerd: time="2024-07-23T00:38:01.153394418+10:00" level=warning msg="ShouldRestart failed, container will not be restarted" container=ee2c02ee50f1dcd388a878607051bf314182e98aed461ba9b9022d29583ff61d daemonShuttingDown=false error="restart canceled" execDuration=108h50m11.400441193s exitStatus="{0 2024-07-22 14:38:00.95120732 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
- Jul 23 00:38:00 systemd: docker-ee2c02ee50f1dcd388a878607051bf314182e98aed461ba9b9022d29583ff61d.scope: Consumed 50min 12.849s CPU time.
- Jul 23 00:37:58 dockerd: time="2024-07-23T00:37:58.459284519+10:00" level=info msg="Loading containers: start."
- Jul 23 00:37:52 systemd: Starting Docker Application Container Engine...
- Jul 23 00:37:49 dockerd: time="2024-07-23T00:37:49.049089411+10:00" level=info msg="ignoring event" container=c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
- Jul 23 00:37:49 systemd: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Consumed 13min 8.358s CPU time.
- Jul 23 00:37:41 dockerd: time="2024-07-23T00:37:41.409440356+10:00" level=info msg="Processing signal 'terminated'"
- Jul 23 00:37:41 systemd: Stopping Docker Application Container Engine...
Jul 22 12:41:17 docker-registry: time="2024-07-22T12:41:17.867116448+10:00" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 18 11:48:08 dockerd: time="2024-07-18T11:48:08.368432699+10:00" level=info msg="API listen on /run/docker.sock"
Jul 18 11:48:08 systemd: Started Docker Application Container Engine.
Jul 18 11:48:07 dockerd: time="2024-07-18T11:48:07.299326363+10:00" level=info msg="Daemon has completed initialization"
Jul 18 11:47:53 systemd: var-lib-docker-overlay2-opaque\x2dbug\x2dcheck2510689170-merged.mount: Deactivated successfully.
Jul 18 11:47:52 dockerd: time="2024-07-18T11:47:52.309200717+10:00" level=info msg="Loading containers: done."
Jul 18 11:47:49 systemd: run-docker-runtime\x2drunc-moby-870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156-runc.sXaOYM.mount: Deactivated successfully.
Jul 18 11:47:49 dockerd: time="2024-07-18T11:47:49.411099654+10:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers"
Jul 18 11:47:47 systemd: run-docker-netns-22e831872dba.mount: Deactivated successfully.
Jul 18 11:47:47 dockerd: time="2024-07-18T11:47:47.043479755+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 141d89a8fe121390709322bcf5292f7a4ae385cc64df372867883ac4e5faf3e9], retrying...."
Jul 18 11:47:46 systemd: run-docker-netns-0b02fe6c84c4.mount: Deactivated successfully.
Jul 18 11:47:46 dockerd: time="2024-07-18T11:47:46.271549542+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 4f3308d447b8e79c4575a534333bde90785a399293e7f752ed3044f98508f155], retrying...."
Jul 18 11:47:45 systemd: run-docker-netns-fe83db4b1b67.mount: Deactivated successfully.
Jul 18 11:47:44 dockerd: time="2024-07-18T11:47:44.952220393+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 5c6762647a54c0c4ca4c3e1c18a155f05f4d603ed60c7f012f696a86e7292b2f], retrying...."
Jul 18 11:47:44 systemd: run-docker-netns-ef8a889674ae.mount: Deactivated successfully.
Jul 18 11:47:43 dockerd: time="2024-07-18T11:47:43.517636671+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 c8031b19dfd6c937319088a489dced3f1119ca3e92cfa6a1fd9089375bf824ce], retrying...."
Jul 18 11:47:42 systemd: run-docker-netns-4e4742115ddf.mount: Deactivated successfully.
Jul 18 11:47:42 dockerd: time="2024-07-18T11:47:42.563235103+10:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint dffad016560945c744d3706a4fe66eb48469c132212960203a43fc1ea9a99566 629531e7cf3e07c8b1f125ffa6347dae09322452a8cebf081a05ad3c59f472af], retrying...."
Jul 18 11:47:40 NetworkManager: <info>  [1721267260.0847] device (docker0): Activation: successful, device activated.
Jul 18 11:47:39 avahi-daemon: Registering new address record for 172.17.0.1 on docker0.IPv4.
Jul 18 11:47:39 NetworkManager: <info>  [1721267259.4840] manager: (docker0): new Bridge device (/org/freedesktop/NetworkManager/Devices/3)
Jul 18 11:47:35 systemd: var-lib-docker-overlay2-14897669c9a89f1f650be1846530e29da3db2c38b163a65065b5d2e6897b8301-merged.mount: Deactivated successfully.
Jul 18 11:47:32 dockerd: time="2024-07-18T11:47:32.615775795+10:00" level=info msg="Loading containers: start."
Jul 18 11:46:28 kernel: audit: type=1400 audit(1721267188.692:39): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1753 comm="apparmor_parser"
Jul 18 11:46:28 apparmor_parser: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1753 comm="apparmor_parser"
Jul 18 11:46:25 dockerd: time="2024-07-18T11:46:22.144309315+10:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jul 18 11:45:41 systemd: Starting Docker Application Container Engine...
Jul 18 11:45:15 docker-registry: time="2024-07-18T11:45:15.550729296+10:00" level=info msg="listening on [::]:5000" go.version=go1.17.3 instance.id=65d09564-e197-4373-a184-89e71a0778ad service=registry version=2.8.0+ds1
Jul 18 11:44:46 systemd: Started the Docker toolset to pack, ship, store, and deliver content.

This is an extract from syslog for events on Jul 25.

Note time stamps out of the syslog are ascending. (Reverse to docker logs above.)

Jul 25 00:31:04 WWVB systemd[1]: Finished Update the local ESM caches.
Jul 25 00:31:06 WWVB systemd[1]: apt-news.service: Deactivated successfully.
Jul 25 00:31:06 WWVB systemd[1]: Finished Update APT News.
Jul 25 00:32:13 WWVB systemd[1]: Reloading.
Jul 25 00:32:15 WWVB systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
+ Jul 25 00:32:17 WWVB systemd[1]: Reloading.
+ Jul 25 00:32:17 WWVB systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
+ Jul 25 00:32:17 WWVB systemd[1]: Stopping Docker Application Container Engine...
Jul 25 00:32:20 WWVB systemd[1]: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Deactivated successfully.
Jul 25 00:32:20 WWVB systemd[1]: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Consumed 1h 9min 57.607s CPU time.
Jul 25 00:32:21 WWVB systemd[1]: docker-c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.scope: Deactivated successfully.
Jul 25 00:32:21 WWVB systemd[1]: docker-c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.scope: Consumed 18min 21.889s CPU time.
Jul 25 00:32:22 WWVB systemd[1]: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Deactivated successfully.
Jul 25 00:32:22 WWVB systemd[1]: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Consumed 5min 59.850s CPU time.
Jul 25 00:32:27 WWVB systemd[1]: run-docker-netns-6706134a2515.mount: Deactivated successfully.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Killing process 1261710 (dockerd) with signal SIGKILL.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Failed with result 'timeout'.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Unit process 1321111 (docker-proxy) remains running after unit stopped.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Unit process 1321125 (docker-proxy) remains running after unit stopped.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Unit process 1321156 (docker-proxy) remains running after unit stopped.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Unit process 1321179 (docker-proxy) remains running after unit stopped.
Jul 25 00:32:28 WWVB systemd[1]: Stopped Docker Application Container Engine.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Consumed 5min 1.816s CPU time.
Jul 25 00:32:28 WWVB systemd[1]: docker.socket: Deactivated successfully.
Jul 25 00:32:28 WWVB systemd[1]: Closed Docker Socket for the API.
Jul 25 00:32:28 WWVB systemd[1]: Stopping Docker Socket for the API...
Jul 25 00:32:28 WWVB systemd[1]: Starting Docker Socket for the API...
Jul 25 00:32:28 WWVB systemd[1]: Listening on Docker Socket for the API.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Found left-over process 1321111 (docker-proxy) in control group while starting unit. Ignoring.
Jul 25 00:32:28 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Found left-over process 1321125 (docker-proxy) in control group while starting unit. Ignoring.
Jul 25 00:32:28 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Found left-over process 1321156 (docker-proxy) in control group while starting unit. Ignoring.
Jul 25 00:32:28 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 25 00:32:28 WWVB systemd[1]: docker.service: Found left-over process 1321179 (docker-proxy) in control group while starting unit. Ignoring.
Jul 25 00:32:28 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 25 00:32:28 WWVB systemd[1]: Starting Docker Application Container Engine...
Jul 25 00:32:29 WWVB dockerd[1790796]: time="2024-07-25T00:32:29.060075342+10:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jul 25 00:32:29 WWVB systemd-udevd[1790793]: Using default interface naming scheme 'v249'.
Jul 25 00:32:36 WWVB systemd[1]: var-lib-docker-overlay2-2bad425d953bf971f6ec7b2c81a0ac0804fc13f2db9d80e4c8ea836be82170c3-merged.mount: Deactivated successfully.
Jul 25 00:32:36 WWVB systemd[1]: var-lib-docker-overlay2-e42b8ab59062f99e82bc54bf9bf69b37295065e5f55b3921ae787402d4bf3512-merged.mount: Deactivated successfully.
Jul 25 00:32:36 WWVB systemd[1]: var-lib-docker-overlay2-56166d0d12fece68ade75b0c73a9baa76a3abc113f92c4dc5f361ed39b007b88-merged.mount: Deactivated successfully.
Jul 25 00:32:38 WWVB systemd[1]: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Deactivated successfully.
Jul 25 00:32:38 WWVB systemd[1]: docker-ead765108c2ea915966f13999fe21d787d0625420e2e1070766e0a5f717080b3.scope: Consumed 19min 8.528s CPU time.
Jul 25 00:32:38 WWVB systemd[1]: var-lib-docker-overlay2-8409cd1b100a9f71ffcd00f01ace4c01524d3eaa3c868c297d92ee737164cf45-merged.mount: Deactivated successfully.
Jul 25 00:32:42 WWVB systemd[1]: docker-ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52.scope: Deactivated successfully.
Jul 25 00:32:42 WWVB systemd[1]: docker-ed4d189ab842ebfff049c9b4e903b256f19aaefe775a5daafc55aa3bf74b2e52.scope: Consumed 28.398s CPU time.
Jul 25 00:32:42 WWVB systemd[1]: var-lib-docker-overlay2-6623eb48bd8021bccb1e6ea8d2839fd496541a29dee04cefb84953eb488e5d81-merged.mount: Deactivated successfully.
Jul 25 00:32:44 WWVB systemd[1]: run-docker-netns-aba8b6b71cfb.mount: Deactivated successfully.
Jul 25 00:32:44 WWVB systemd[1]: run-docker-netns-aba8b6b71cfb.mount: Deactivated successfully.
Jul 25 00:32:44 WWVB systemd[1]: run-docker-netns-ae83477f1a74.mount: Deactivated successfully.
Jul 25 00:32:45 WWVB systemd-udevd[1791091]: Using default interface naming scheme 'v249'.
Jul 25 00:32:45 WWVB systemd-udevd[1791092]: Using default interface naming scheme 'v249'.
Jul 25 00:32:45 WWVB systemd[1]: run-docker-netns-ae83477f1a74.mount: Deactivated successfully.
Jul 25 00:32:45 WWVB systemd[1]: run-docker-netns-531a9d4cc999.mount: Deactivated successfully.
Jul 25 00:32:45 WWVB systemd[1]: run-docker-netns-2ed445e5cc85.mount: Deactivated successfully.
Jul 25 00:32:45 WWVB systemd-udevd[1791196]: Using default interface naming scheme 'v249'.
Jul 25 00:32:45 WWVB systemd-udevd[1791198]: Using default interface naming scheme 'v249'.
Jul 25 00:32:45 WWVB systemd-udevd[1791201]: Using default interface naming scheme 'v249'.
Jul 25 00:32:45 WWVB systemd[1]: run-docker-runtime\x2drunc-moby-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e-runc.s5KauY.mount: Deactivated successfully.
Jul 25 00:32:45 WWVB systemd[1]: Started libcontainer container c2aafd7a0803216750826db7fb7c249e60c8b570a944a8797e7696ebd83a85bb.
Jul 25 00:32:45 WWVB systemd[1]: Started libcontainer container c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.
Jul 25 00:32:45 WWVB systemd[1]: Started libcontainer container c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.
Jul 25 00:32:46 WWVB systemd[1]: var-lib-docker-overlay2-opaque\x2dbug\x2dcheck943418215-merged.mount: Deactivated successfully.
Jul 25 00:32:47 WWVB systemd[1]: Started Docker Application Container Engine.
Jul 25 00:32:53 WWVB systemd[1]: mintupdate-automation-upgrade.service: Deactivated successfully.
+ Jul 25 00:32:53 WWVB systemd[1]: Finished Update Manager automatic upgrades.
+ Jul 25 00:32:53 WWVB systemd[1]: mintupdate-automation-upgrade.service: Consumed 34.795s CPU time.
Jul 25 00:39:01 WWVB CRON[1792373]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 25 00:39:17 WWVB systemd[1]: Starting Clean php session files...
Jul 25 00:39:17 WWVB systemd[1]: phpsessionclean.service: Deactivated successfully.
Jul 25 00:39:17 WWVB systemd[1]: Finished Clean php session files.
Jul 25 01:06:25 WWVB systemd[1]: Starting Daily man-db regeneration...

This is the syslogs from Jul 23.

Note time stamps in this log are ascending.

Jul 23 00:35:52 WWVB systemd[1]: Finished Update APT News.
Jul 23 00:37:34 WWVB systemd[1]: Reloading.
Jul 23 00:37:37 WWVB systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
Jul 23 00:37:39 WWVB systemd[1]: Starting Refresh fwupd metadata and update motd...
Jul 23 00:37:39 WWVB dbus-daemon[821]: [system] Activating via systemd: service name='org.freedesktop.fwupd' unit='fwupd.service' requested by ':1.150' (uid=127 pid=1261566 comm="/usr/bin/fwupdmgr refresh " label="unconfined")
Jul 23 00:37:39 WWVB systemd[1]: Starting Firmware update daemon...
Jul 23 00:37:40 WWVB systemd[1]: Started Firmware update daemon.
Jul 23 00:37:40 WWVB systemd[1]: fwupd-refresh.service: Deactivated successfully.
Jul 23 00:37:40 WWVB systemd[1]: Finished Refresh fwupd metadata and update motd.
Jul 23 00:37:40 WWVB systemd[1]: Reloading.
Jul 23 00:37:40 WWVB systemd[1]: Configuration file /run/systemd/system/netplan-ovs-cleanup.service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.
- Jul 23 00:37:41 WWVB systemd[1]: Starting Daily man-db regeneration...
- Jul 23 00:37:41 WWVB systemd[1]: Stopping Docker Application Container Engine...
Jul 23 00:37:45 WWVB systemd[1]: docker-870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156.scope: Deactivated successfully.
Jul 23 00:37:45 WWVB systemd[1]: docker-870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156.scope: Consumed 49min 32.700s CPU time.
Jul 23 00:37:46 WWVB systemd[1]: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Deactivated successfully.
Jul 23 00:37:46 WWVB systemd[1]: docker-c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.scope: Consumed 2h 25min 23.212s CPU time.
Jul 23 00:37:49 WWVB systemd[1]: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Deactivated successfully.
Jul 23 00:37:49 WWVB systemd[1]: docker-c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.scope: Consumed 13min 8.358s CPU time.
Jul 23 00:37:49 WWVB systemd[1]: man-db.service: Deactivated successfully.
Jul 23 00:37:49 WWVB systemd[1]: Finished Daily man-db regeneration.
Jul 23 00:37:49 WWVB systemd[1]: man-db.service: Consumed 1.238s CPU time.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: State 'stop-sigterm' timed out. Killing.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Killing process 1618 (dockerd) with signal SIGKILL.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Main process exited, code=killed, status=9/KILL
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Failed with result 'timeout'.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3342 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3349 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3380 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3386 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3401 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3407 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3444 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Unit process 3451 (docker-proxy) remains running after unit stopped.
Jul 23 00:37:51 WWVB systemd[1]: Stopped Docker Application Container Engine.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Consumed 10min 5.856s CPU time.
Jul 23 00:37:51 WWVB systemd[1]: docker.socket: Deactivated successfully.
Jul 23 00:37:51 WWVB systemd[1]: Closed Docker Socket for the API.
Jul 23 00:37:51 WWVB systemd[1]: Stopping Docker Socket for the API...
Jul 23 00:37:51 WWVB systemd[1]: Starting Docker Socket for the API...
Jul 23 00:37:51 WWVB systemd[1]: Listening on Docker Socket for the API.
Jul 23 00:37:51 WWVB systemd[1]: docker.service: Found left-over process 3342 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:51 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3349 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3380 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3386 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3401 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3407 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3444 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: docker.service: Found left-over process 3451 (docker-proxy) in control group while starting unit. Ignoring.
Jul 23 00:37:52 WWVB systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jul 23 00:37:52 WWVB systemd[1]: Starting Docker Application Container Engine...
Jul 23 00:37:52 WWVB containerd[1158]: time="2024-07-23T00:37:52.318069696+10:00" level=warning msg="cleanup warnings time=\"2024-07-23T00:37:52+10:00\" level=warning msg=\"failed to remove runc container\" error=\"runc did not terminate successfully: exit status 1: open /run/containerd/io.containerd.runtime.v2.task/moby/c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e/log.json: no such file or directory\\n\\nNAME:\\n   runc - Open Container Initiative runtime\\n\\nrunc is a command line client for running applications packaged according to\\nthe Open Container Initiative (OCI) format and is a compliant implementation of the\\nOpen Container Initiative specification.\\n\\nrunc integrates well with existing process supervisors to provide a production\\ncontainer runtime environment for applications. It can be used with your\\nexisting process monitoring tools and the container will be spawned as a\\ndirect child of the process supervisor.\\n\\nContainers are configured using bundles. A bundle for a container is a directory\\nthat includes a specification file named \\\"config.json\\\" and a root filesystem.\\nThe root filesystem contains the contents of the container.\\n\\nTo start a new instance of a container:\\n\\n    # runc run [ -b bundle ] <container-id>\\n\\nWhere \\\"<container-id>\\\" is your name for the instance of the container that you\\nare starting. The name you provide for the container instance must be unique on\\nyour host. Providing the bundle directory using \\\"-b\\\" is optional. The default\\nvalue for \\\"bundle\\\" is the current directory.\\n\\nUSAGE:\\n   runc [global options] command [command options] [arguments...]\\n\\nVERSION:\\n   1.7.19\\ncommit: v1.1.13-0-g58aa920\\nspec: 1.0.2-dev\\ngo: go1.21.12\\nlibseccomp: 2.5.3\\n\\nCOMMANDS:\\n   checkpoint  checkpoint a running container\\n   create      create a container\\n   delete      delete any resources held by the container often used with detached container\\n   events      display container events such as OOM notifications, cpu, memory, and IO usage statistics\\n   exec        execute new process inside the container\\n   kill        kill sends the specified signal (default: SIGTERM) to the container's init process\\n   list        lists containers started by runc with the given root\\n   pause       pause suspends all processes inside the container\\n   ps          ps displays the processes running inside a container\\n   restore     restore a container from a previous checkpoint\\n   resume      resumes all processes that have been previously paused\\n   run         create and run a container\\n   spec        create a new specification file\\n   start       executes the user defined process in a created container\\n   state       output the state of a container\\n   update      update container resource constraints\\n   features    show the enabled features\\n   help, h     Shows a list of commands or help for one command\\n\\nGLOBAL OPTIONS:\\n   --debug             enable debug logging\\n   --log value         set the log file to write runc logs to (default is '/dev/stderr')\\n   --log-format value  set the log format ('text' (default), or 'json') (default: \\\"text\\\")\\n   --root value        root directory for storage of container state (this should be located in tmpfs) (default: \\\"/run/runc\\\")\\n   --criu value        path to the criu binary used for checkpoint and restore (default: \\\"criu\\\")\\n   --systemd-cgroup    enable systemd cgroup support, expects cgroupsPath to be of form \\\"slice:prefix:name\\\" for e.g. \\\"system.slice:runc:434234\\\"\\n   --rootless value    ignore cgroup permission errors ('true', 'false', or 'auto') (default: \\\"auto\\\")\\n   --help, -h          show help\\n   --version, -v       print the version\\n{\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"open /run/containerd/io.containerd.runtime.v2.task/moby/c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e/log.json: no such file or directory\\\",\\\"time\\\":\\\"2024-07-23T00:37:52+10:00\\\"}\\n\" runtime=io.containerd.runc.v2\ntime=\"2024-07-23T00:37:52+10:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=moby
Jul 23 00:37:52 WWVB containerd[1158]: time="2024-07-23T00:37:52.330850925+10:00" level=warning msg="cleanup warnings time=\"2024-07-23T00:37:52+10:00\" level=warning msg=\"failed to remove runc container\" error=\"runc did not terminate successfully: exit status 1: open /run/containerd/io.containerd.runtime.v2.task/moby/870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156/log.json: no such file or directory\\n\\nNAME:\\n   runc - Open Container Initiative runtime\\n\\nrunc is a command line client for running applications packaged according to\\nthe Open Container Initiative (OCI) format and is a compliant implementation of the\\nOpen Container Initiative specification.\\n\\nrunc integrates well with existing process supervisors to provide a production\\ncontainer runtime environment for applications. It can be used with your\\nexisting process monitoring tools and the container will be spawned as a\\ndirect child of the process supervisor.\\n\\nContainers are configured using bundles. A bundle for a container is a directory\\nthat includes a specification file named \\\"config.json\\\" and a root filesystem.\\nThe root filesystem contains the contents of the container.\\n\\nTo start a new instance of a container:\\n\\n    # runc run [ -b bundle ] <container-id>\\n\\nWhere \\\"<container-id>\\\" is your name for the instance of the container that you\\nare starting. The name you provide for the container instance must be unique on\\nyour host. Providing the bundle directory using \\\"-b\\\" is optional. The default\\nvalue for \\\"bundle\\\" is the current directory.\\n\\nUSAGE:\\n   runc [global options] command [command options] [arguments...]\\n\\nVERSION:\\n   1.7.19\\ncommit: v1.1.13-0-g58aa920\\nspec: 1.0.2-dev\\ngo: go1.21.12\\nlibseccomp: 2.5.3\\n\\nCOMMANDS:\\n   checkpoint  checkpoint a running container\\n   create      create a container\\n   delete      delete any resources held by the container often used with detached container\\n   events      display container events such as OOM notifications, cpu, memory, and IO usage statistics\\n   exec        execute new process inside the container\\n   kill        kill sends the specified signal (default: SIGTERM) to the container's init process\\n   list        lists containers started by runc with the given root\\n   pause       pause suspends all processes inside the container\\n   ps          ps displays the processes running inside a container\\n   restore     restore a container from a previous checkpoint\\n   resume      resumes all processes that have been previously paused\\n   run         create and run a container\\n   spec        create a new specification file\\n   start       executes the user defined process in a created container\\n   state       output the state of a container\\n   update      update container resource constraints\\n   features    show the enabled features\\n   help, h     Shows a list of commands or help for one command\\n\\nGLOBAL OPTIONS:\\n   --debug             enable debug logging\\n   --log value         set the log file to write runc logs to (default is '/dev/stderr')\\n   --log-format value  set the log format ('text' (default), or 'json') (default: \\\"text\\\")\\n   --root value        root directory for storage of container state (this should be located in tmpfs) (default: \\\"/run/runc\\\")\\n   --criu value        path to the criu binary used for checkpoint and restore (default: \\\"criu\\\")\\n   --systemd-cgroup    enable systemd cgroup support, expects cgroupsPath to be of form \\\"slice:prefix:name\\\" for e.g. \\\"system.slice:runc:434234\\\"\\n   --rootless value    ignore cgroup permission errors ('true', 'false', or 'auto') (default: \\\"auto\\\")\\n   --help, -h          show help\\n   --version, -v       print the version\\n{\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"open /run/containerd/io.containerd.runtime.v2.task/moby/870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156/log.json: no such file or directory\\\",\\\"time\\\":\\\"2024-07-23T00:37:52+10:00\\\"}\\n\" runtime=io.containerd.runc.v2\ntime=\"2024-07-23T00:37:52+10:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/moby/870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156/init.pid: no such file or directory\" runtime=io.containerd.runc.v2\n" namespace=moby
Jul 23 00:37:52 WWVB dockerd[1261710]: time="2024-07-23T00:37:52.417961292+10:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Jul 23 00:37:58 WWVB systemd[1]: var-lib-docker-overlay2-56166d0d12fece68ade75b0c73a9baa76a3abc113f92c4dc5f361ed39b007b88-merged.mount: Deactivated successfully.
Jul 23 00:37:58 WWVB systemd[1]: var-lib-docker-overlay2-073b6d59c5d1ae98214eb5e2ace94ac6914cbf07195f806664f5339e652f86e9-merged.mount: Deactivated successfully.
Jul 23 00:37:59 WWVB systemd[1]: var-lib-docker-overlay2-e42b8ab59062f99e82bc54bf9bf69b37295065e5f55b3921ae787402d4bf3512-merged.mount: Deactivated successfully.
Jul 23 00:37:59 WWVB systemd[1]: var-lib-docker-overlay2-56166d0d12fece68ade75b0c73a9baa76a3abc113f92c4dc5f361ed39b007b88-merged.mount: Deactivated successfully.
Jul 23 00:38:00 WWVB systemd[1]: docker-ee2c02ee50f1dcd388a878607051bf314182e98aed461ba9b9022d29583ff61d.scope: Deactivated successfully.
Jul 23 00:38:00 WWVB systemd[1]: docker-ee2c02ee50f1dcd388a878607051bf314182e98aed461ba9b9022d29583ff61d.scope: Consumed 50min 12.849s CPU time.
Jul 23 00:38:01 WWVB systemd[1]: var-lib-docker-overlay2-6dac44c5983a43bb837ab2792c18c54002701fc3d3b8c93254cfd8a7b052e96c-merged.mount: Deactivated successfully.
Jul 23 00:38:07 WWVB systemd[1]: docker-c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696.scope: Deactivated successfully.
Jul 23 00:38:07 WWVB systemd[1]: docker-c9cfb00ce9399859b68a6e8f4f429e34c31bc8d30f347f371281bd6629991696.scope: Consumed 1min 17.026s CPU time.
Jul 23 00:38:07 WWVB systemd[1]: var-lib-docker-overlay2-14897669c9a89f1f650be1846530e29da3db2c38b163a65065b5d2e6897b8301-merged.mount: Deactivated successfully.
Jul 23 00:38:09 WWVB systemd[1]: run-docker-netns-374b85024c72.mount: Deactivated successfully.
Jul 23 00:38:09 WWVB systemd[1]: run-docker-netns-374b85024c72.mount: Deactivated successfully.
Jul 23 00:38:09 WWVB systemd[1]: run-docker-netns-4833ed4962f2.mount: Deactivated successfully.
Jul 23 00:38:10 WWVB systemd[1]: run-docker-netns-4833ed4962f2.mount: Deactivated successfully.
Jul 23 00:38:10 WWVB systemd[1]: run-docker-netns-8998a8fef10e.mount: Deactivated successfully.
Jul 23 00:38:10 WWVB systemd[1]: run-docker-netns-8998a8fef10e.mount: Deactivated successfully.
Jul 23 00:38:10 WWVB systemd[1]: run-docker-netns-bc0b4ecff182.mount: Deactivated successfully.
Jul 23 00:38:11 WWVB systemd[1]: run-docker-netns-f1e93de42a2b.mount: Deactivated successfully.
Jul 23 00:38:12 WWVB systemd-udevd[1262051]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd-udevd[1262033]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd-udevd[1262052]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd-udevd[1262042]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd-udevd[1262030]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd-udevd[1262043]: Using default interface naming scheme 'v249'.
Jul 23 00:38:12 WWVB systemd[1]: Started libcontainer container c5ded8a5ae48b8bd6f8d0712c40d436d2a4efc1e383973fcf6c47f36c6bde067.
Jul 23 00:38:12 WWVB systemd[1]: Started libcontainer container c5969bef582bf557bcb01f01ce072356e7f19f82959d187a35b6d5e97f83068e.
Jul 23 00:38:13 WWVB systemd[1]: Started libcontainer container 870c2307d3ff03f2890200369c9aaf95c53522b39ec2d9a541dfc0a4752a8156.
- Jul 23 00:38:19 WWVB systemd[1]: Started Docker Application Container Engine.
Jul 23 00:38:27 WWVB systemd[1]: mintupdate-automation-upgrade.service: Deactivated successfully.
Jul 23 00:38:27 WWVB systemd[1]: Finished Update Manager automatic upgrades.
Jul 23 00:38:27 WWVB systemd[1]: mintupdate-automation-upgrade.service: Consumed 50.000s CPU time.
Jul 23 00:39:01 WWVB systemd[1]: Starting Clean php session files...
Jul 23 00:39:01 WWVB systemd[1]: phpsessionclean.service: Deactivated successfully.

jasonacox Jul 27, 2024
Maintainer

With the docker and containerd errors in the logs, it does seem like something is unhealthy about the docker setup. You could paste all of those logs in an LLM like ChatGPT to see what it says.

As for re-installing or creating a new VM, my recommendation would be to:

Create new VM
Copy over (e.g. tar) the entire Powerwall-Dashboard folder (stop dashbaord ./compose-dash.sh stop)
Ensure all files are owned by the user (otherwise chown -R myuser Powerwall-Dashboard/*)
Run setup.sh - The inflxudb container should mount and use he old database you copied over.

Answer selected by Green-Pool

Green-Pool Jul 27, 2024
Author

Thanks for taking the time to look through my log files.

Yeah, I agree with you. Time to make the migration across to a new (and fresh) VM. I think I will also try a different flavour of Linux as well.

It might take a bit of time, but I will report back at some point with how I go with a new VM.

Green-Pool Jul 27, 2024
Author

Just a second update ...

I built a new VM using Lubuntu. Seems nice and lightweight. As it was the first time I have used Lubuntu, it took me a few attempts to get this VM the way I wanted it.

Installed docker. Copied the docker folders as per your instructions (thanks!). Powerwall docker came up OK on the new VM. Total time from shutdown on the old VM to it gathering statistics from the new VM was about 25 minutes.

Left a backup on the old VM, but was confident enough to delete the docker install on the old VM.

Thanks again Jason for your help.

Now, we wait to see if it misbehaves again ...

jasonacox Jul 27, 2024
Maintainer

Nice! Fingers crossed! 🤞 😉

Green-Pool Aug 15, 2024
Author

Just to close this discussion out ... it has been three weeks so far and all has gone well. No crashes.

Looking back through the logs a bit more, and I am convinced that the Docker random crashes was somehow linked to Linux Mint auto-update. It didn't happen every night. But if the processes somehow lined up, the Mint update didn't stop (or restart) Docker cleanly.

In defence of both Mint and Docker ... this was a rather old VM. Vague memory was it started at Mint 19.1 (Ubuntu 18.04). Over the last few years I have been updating it with new releases. The VM was also hosting a LAMP installation that has done sterling services as well. That is the trouble with a well running VM, there is a strong impulse to leave well enough alone.

Anyway. I separated the LAMP out into its own Debian VM. Presently pyPowerwall is running in a separate Lubuntu VM (just to see what Lubuntu is like).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyPowerwall Proxy Stopped (getting more frequent) #506

{{title}}

Verify Powerwall-Dashboard 4.4.6 on Linux -

Checking pypowerwall

Checking telegraf

Checking influxdb

Checking grafana

Checking weather411

Verify Powerwall-Dashboard 4.4.6 on Linux -

Checking pypowerwall

Checking telegraf

Checking influxdb

Checking grafana

Checking weather411

Replies: 1 comment 9 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

pyPowerwall Proxy Stopped (getting more frequent) #506

Green-Pool Jul 25, 2024

Verify Powerwall-Dashboard 4.4.6 on Linux -

Checking pypowerwall

Checking telegraf

Checking influxdb

Checking grafana

Checking weather411

Verify Powerwall-Dashboard 4.4.6 on Linux -

Checking pypowerwall

Checking telegraf

Checking influxdb

Checking grafana

Checking weather411

Replies: 1 comment · 9 replies

jasonacox Jul 26, 2024 Maintainer

Green-Pool Jul 26, 2024 Author

jasonacox Jul 26, 2024 Maintainer

Green-Pool Jul 26, 2024 Author

Extract from the logs filtered on the keyword "docker".

This is an extract from syslog for events on Jul 25.

This is the syslogs from Jul 23.

jasonacox Jul 27, 2024 Maintainer

Green-Pool Jul 27, 2024 Author

Green-Pool Jul 27, 2024 Author

jasonacox Jul 27, 2024 Maintainer

Green-Pool Aug 15, 2024 Author

Green-Pool
Jul 25, 2024

Replies: 1 comment 9 replies

jasonacox
Jul 26, 2024
Maintainer

Green-Pool Jul 26, 2024
Author

jasonacox Jul 26, 2024
Maintainer

Green-Pool Jul 26, 2024
Author

jasonacox Jul 27, 2024
Maintainer

Green-Pool Jul 27, 2024
Author

Green-Pool Jul 27, 2024
Author

jasonacox Jul 27, 2024
Maintainer

Green-Pool Aug 15, 2024
Author