container is removed if a new image is found but download has failed

[eskiiom]

Hello everyone

I saw many closed issues on the same topic, mostly related to synology, which is not my case.
Also, most of these issues where closed with PR implemented, so I don't know if this is a know issue at the moment or not.
I am using the latest watchtower version

Describe the bug

It happened a couple of times already, on various images, so I believe the issue comes from my local network or ISP, but what happened is the following :

Found new owncloud/server:latest image (eeee478cb601) Found new homeassistant/home-assistant:latest image (203c7de4397c) Found new reallibrephotos/librephotos-frontend:dev image (f809549e4bc2) Stopping /librephotos-docker_frontend_1 (c33e994f9829) with SIGTERM Stopping /Homeassistant (8559d67e5658) with SIGTERM Stopping /Owncloud (3fbaf324eeb6) with SIGTERM Creating /Owncloud Error response from daemon: No such image: owncloud/server:latest Creating /Homeassistant Error response from daemon: No such image: homeassistant/home-assistant:latest Creating /librephotos-docker_frontend_1 Error response from daemon: No such image: reallibrephotos/librephotos-frontend:dev

To Reproduce

Try to update an image but pull the network plug when image is being downloaded.
New image found, but download fails, affected container is then deleted.

Expected behavior

I would like anything else but my containers removed.
So different options would be for example to recreate the container from the new image if and only is the download is successful
Even better would be to retry/resume the download when it has failed.
I would rather have an up and running container with a 1+ day old image, instead of recent download attempt which ends in container to disappear. A notification would be sufficient.

Screenshots

Environment

• Platform x86

Logs from running watchtower with the --debug option

time="2022-09-02T17:16:08Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=4 notify=no
time="2022-09-03T17:14:18Z" level=info msg="Found new reallibrephotos/librephotos:dev image (80bf42190551)"
time="2022-09-03T17:14:22Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (4b56ff07d7a4)"
time="2022-09-03T17:14:48Z" level=info msg="Stopping /librephotos-docker_frontend_1 (e188acd9f55e) with SIGTERM"
time="2022-09-03T17:14:51Z" level=info msg="Stopping /librephotos-docker_backend_1 (74d3a06436d7) with SIGTERM"
time="2022-09-03T17:15:04Z" level=info msg="Creating /librephotos-docker_backend_1"
time="2022-09-03T17:15:09Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-03T17:15:12Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=2 notify=no
time="2022-09-04T17:13:59Z" level=info msg="Found new reallibrephotos/librephotos:dev image (4cfdbcdb74a0)"
time="2022-09-04T17:14:10Z" level=info msg="Found new lscr.io/linuxserver/heimdall:latest image (dbc0d3e258ce)"
time="2022-09-04T17:14:29Z" level=info msg="Found new ghcr.io/ajnart/homarr:latest image (214d5b2dcfab)"
time="2022-09-04T17:14:38Z" level=info msg="Stopping /homarr (6945f3a52660) with SIGTERM"
time="2022-09-04T17:14:56Z" level=info msg="Stopping /Heimdall (5b8e5c99ff0c) with SIGTERM"
time="2022-09-04T17:15:03Z" level=info msg="Stopping /librephotos-docker_backend_1 (f8048957e738) with SIGTERM"
time="2022-09-04T17:15:17Z" level=info msg="Creating /librephotos-docker_backend_1"
time="2022-09-04T17:15:20Z" level=info msg="Creating /Heimdall"
time="2022-09-04T17:15:25Z" level=info msg="Creating /homarr"
time="2022-09-04T17:15:29Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=3 notify=no
time="2022-09-05T17:13:26Z" level=info msg="Found new louislam/uptime-kuma:latest image (f58667a58b2c)"
time="2022-09-05T17:13:32Z" level=info msg="Stopping /UptimeKuma (f1c94ae570c6) with SIGTERM"
time="2022-09-05T17:14:06Z" level=info msg="Creating /UptimeKuma"
time="2022-09-05T17:14:11Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=1 notify=no
time="2022-09-06T17:13:04Z" level=info msg="Found new linuxserver/piwigo:latest image (57a32bf8ac9b)"
time="2022-09-06T17:13:07Z" level=info msg="Found new linuxserver/piwigo:latest image (57a32bf8ac9b)"
time="2022-09-06T17:13:34Z" level=info msg="Found new ghcr.io/linuxserver/syncthing:latest image (1305b9b6c852)"
time="2022-09-06T17:13:52Z" level=info msg="Found new portainer/portainer-ce:latest image (47535c4d5fa6)"
time="2022-09-06T17:13:59Z" level=info msg="Stopping /Portainer (d393f4e70d3e) with SIGTERM"
time="2022-09-06T17:14:12Z" level=info msg="Stopping /syncthing (23dfaa9152c7) with SIGTERM"
time="2022-09-06T17:14:26Z" level=info msg="Stopping /Piwigo_Thierry_Ginette (8f86fc4c6ad2) with SIGTERM"
time="2022-09-06T17:14:34Z" level=info msg="Stopping /Piwigo (ec6160978083) with SIGTERM"
time="2022-09-06T17:14:41Z" level=info msg="Creating /Piwigo"
time="2022-09-06T17:14:43Z" level=info msg="Creating /Piwigo_Thierry_Ginette"
time="2022-09-06T17:14:46Z" level=info msg="Creating /syncthing"
time="2022-09-06T17:14:51Z" level=info msg="Creating /Portainer"
time="2022-09-06T17:14:55Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=4 notify=no
time="2022-09-07T17:13:58Z" level=info msg="Found new reallibrephotos/librephotos:dev image (77f2547c7489)"
time="2022-09-07T17:14:11Z" level=info msg="Unable to update container \"/librephotos-docker_frontend_1\": Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.0.100:53: read udp 192.168.0.111:47198->192.168.0.100:53: i/o timeout. Proceeding to next."
time="2022-09-07T17:14:50Z" level=info msg="Found new owncloud/server:latest image (799402fee56f)"
time="2022-09-07T17:15:00Z" level=info msg="Stopping /Owncloud (89a3b103d6f5) with SIGTERM"
time="2022-09-07T17:15:58Z" level=info msg="Stopping /librephotos-docker_backend_1 (40b0a2b71a5e) with SIGTERM"
time="2022-09-07T17:16:12Z" level=info msg="Creating /librephotos-docker_backend_1"
time="2022-09-07T17:16:17Z" level=info msg="Creating /Owncloud"
time="2022-09-07T17:16:22Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=2 notify=no
time="2022-09-08T17:12:54Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (edba0ae3c55e)"
time="2022-09-08T17:13:09Z" level=info msg="Found new lscr.io/linuxserver/bookstack:latest image (0b2a870d325c)"
time="2022-09-08T17:15:44Z" level=info msg="Found new homeassistant/home-assistant:latest image (c6b58ff9510a)"
time="2022-09-08T17:15:59Z" level=info msg="Stopping /Homeassistant (ad0959b8188d) with SIGTERM"
time="2022-09-08T17:16:13Z" level=info msg="Stopping /Bookstack (1da4ea71a1c2) with SIGTERM"
time="2022-09-08T17:16:23Z" level=info msg="Stopping /librephotos-docker_frontend_1 (dcf7a6cff23e) with SIGTERM"
time="2022-09-08T17:16:26Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-08T17:16:29Z" level=info msg="Creating /Bookstack"
time="2022-09-08T17:16:31Z" level=info msg="Creating /Homeassistant"
time="2022-09-08T17:16:35Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=3 notify=no
time="2022-09-09T17:14:40Z" level=info msg="Found new homeassistant/home-assistant:latest image (1cbd36153aac)"
time="2022-09-09T17:14:47Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (82763297843c)"
time="2022-09-09T17:15:17Z" level=info msg="Stopping /librephotos-docker_frontend_1 (e17292286291) with SIGTERM"
time="2022-09-09T17:15:18Z" level=info msg="Stopping /Homeassistant (fc2f607bc7d3) with SIGTERM"
time="2022-09-09T17:15:31Z" level=info msg="Creating /Homeassistant"
time="2022-09-09T17:15:33Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-09T17:15:37Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=2 notify=no
time="2022-09-10T17:13:10Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=0 notify=no
time="2022-09-11T17:12:46Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (cdad1b2a3a74)"
time="2022-09-11T17:13:15Z" level=info msg="Stopping /librephotos-docker_frontend_1 (7983f440b197) with SIGTERM"
time="2022-09-11T17:13:17Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-11T17:13:20Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=1 notify=no
time="2022-09-12T17:12:46Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (1d1670999154)"
time="2022-09-12T17:14:17Z" level=info msg="Found new homeassistant/home-assistant:latest image (dc6d95cb9337)"
time="2022-09-12T17:14:46Z" level=info msg="Stopping /Homeassistant (3c489595388f) with SIGTERM"
time="2022-09-12T17:14:59Z" level=info msg="Stopping /librephotos-docker_frontend_1 (9a549977c8c3) with SIGTERM"
time="2022-09-12T17:15:01Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-12T17:15:05Z" level=info msg="Creating /Homeassistant"
time="2022-09-12T17:15:09Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=2 notify=no
time="2022-09-13T17:13:15Z" level=info msg="Found new owncloud/server:latest image (d75c4cc96ce3)"
time="2022-09-13T17:13:37Z" level=warning msg="Could not do a head request for \"redis:6\", falling back to regular pull." container=/librephotos-docker_redis_1 image="redis:6"
time="2022-09-13T17:13:38Z" level=warning msg="Reason: Head \"https://index.docker.io/v2/library/redis/manifests/6\": net/http: TLS handshake timeout" container=/librephotos-docker_redis_1 image="redis:6"
time="2022-09-13T17:14:05Z" level=info msg="Found new redis:6 image (94cfe49e6758)"
time="2022-09-13T17:14:22Z" level=info msg="Found new postgres:13 image (e30fb19f297b)"
time="2022-09-13T17:14:24Z" level=info msg="Found new postgres:13 image (e30fb19f297b)"
time="2022-09-13T17:14:36Z" level=info msg="Stopping /librephotos-docker_db_1 (112b63b8c827) with SIGTERM"
time="2022-09-13T17:14:53Z" level=info msg="Stopping /joplinpostgres (203e0704de84) with SIGTERM"
time="2022-09-13T17:15:08Z" level=info msg="Stopping /librephotos-docker_redis_1 (1749b81fe8fd) with SIGTERM"
time="2022-09-13T17:15:10Z" level=info msg="Stopping /Owncloud (3ee80c72de7b) with SIGTERM"
time="2022-09-13T17:15:22Z" level=info msg="Creating /Owncloud"
time="2022-09-13T17:15:25Z" level=info msg="Creating /librephotos-docker_redis_1"
time="2022-09-13T17:15:29Z" level=info msg="Creating /joplinpostgres"
time="2022-09-13T17:15:32Z" level=info msg="Creating /librephotos-docker_db_1"
time="2022-09-13T17:15:36Z" level=info msg="Session done" Failed=0 Scanned=33 Updated=4 notify=no
time="2022-09-14T17:14:13Z" level=info msg="Found new owncloud/server:latest image (eeee478cb601)"
time="2022-09-14T17:16:35Z" level=info msg="Found new homeassistant/home-assistant:latest image (203c7de4397c)"
time="2022-09-14T17:16:51Z" level=info msg="Found new reallibrephotos/librephotos-frontend:dev image (f809549e4bc2)"
time="2022-09-14T17:17:24Z" level=info msg="Stopping /librephotos-docker_frontend_1 (c33e994f9829) with SIGTERM"
time="2022-09-14T17:17:26Z" level=info msg="Stopping /Homeassistant (8559d67e5658) with SIGTERM"
time="2022-09-14T17:17:38Z" level=info msg="Stopping /Owncloud (3fbaf324eeb6) with SIGTERM"
time="2022-09-14T17:17:49Z" level=info msg="Creating /Owncloud"
time="2022-09-14T17:17:49Z" level=error msg="Error response from daemon: No such image: owncloud/server:latest"
time="2022-09-14T17:17:49Z" level=info msg="Creating /Homeassistant"
time="2022-09-14T17:17:49Z" level=error msg="Error response from daemon: No such image: homeassistant/home-assistant:latest"
time="2022-09-14T17:17:49Z" level=info msg="Creating /librephotos-docker_frontend_1"
time="2022-09-14T17:17:49Z" level=error msg="Error response from daemon: No such image: reallibrephotos/librephotos-frontend:dev"
time="2022-09-14T17:17:49Z" level=info msg="Session done" Failed=3 Scanned=33 Updated=0 notify=no

Additional context
Sorry if I missed an important information on this issue, I looked but could not find

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[eskiiom]

oops it did it again ...
as a result, the container was removed again.

I understand the issue may mainly be on my network or on the host of the image, but I would like watchtower to handle such exceptions

[eskiiom]

It happened again today ending with a crash of watchtower so I didnt get the notification but 3 or 4 running containers have been removed.

Can it be because I have 45 containers running watchtower needs to check ?
Or because it overloads the host ? (I don't have precise metrics but nothing looked obvious)

Sorry to spam this thread but it's really annoying - today I lost bookstack, ghost, and 2 piwigo...

Anyway thanks for this great tool it works flawlessly most of the time ! 😃

[0x1a8510f2]

Really similar issue here. In my case, downloaded images are occasionally pruned before the container is re-created which leads to existing containers being deleted but no new ones being created. Obviously that increases maintenance significantly for a tool which I use to reduce it. Would love if watchtower re-started the existing container and just logged an error in this case.

[piksel]

@eskiiom if watchtower crashed there is something else going on, do you have the log for the crash? (probably belongs in a new issue though)

[piksel]

@0x1a8510f2 Don't use --cleanup but do a cron scheduled docker images prune instead perhaps?

[0x1a8510f2]

This actually mainly happens when I log onto portainer panel while watchtower is doing an update and see images marked as unused. I'd delete them, only to realise that watchtower is still doing its thing and my containers are now gone. The scheduled prune would be equivalent right? So it shouldn't make a difference, unless I make it run at times watchtower isn't running, but that's not really a solution to the general problem of "if something goes wrong during the update, the containers are deleted permanently".

[piksel]

I don't really understand why this would even happen, since docker pulls images if they don't exist locally. Perhaps it's just really bad timing?
In any case, if you have both removed the new and the old images, there isn't much watchtower can do to "handle" the situation. The old container must be removed before the new one can be created.

[0x1a8510f2]

Yeah, it was bad timing essentially. In my case, only the new image was removed because only that was marked as unused, but I imagine both being removed is also possible.

Since docker doesn't natively support recreating a container, I assume watchtower stores some description of the container in between deleting the old one and creating the new one?

One option could be to write this description somewhere in the filesystem and log an error so the container can easily be re-created, but this would make the watchtower container stateful which isn't ideal obviously. Maybe logging the description in the form of a docker run command would work? Something like:

Failed to recreate container {name}. Retry manually:

docker run ...

[eskiiom]

@eskiiom if watchtower crashed there is something else going on, do you have the log for the crash? (probably belongs in a new issue though)

Hello and really sorry for the delay
Here is the log from one of the last crash

time="2022-09-20T17:12:36Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:a895c3f41cdba0b993df081e1614f38c75791159ec04ee9a362c528cecfa108c"
time="2022-09-20T17:12:41Z" level=warning msg="Could not do a head request for \"owncloud/server:latest\", falling back to regular pull." container=/Owncloud image="owncloud/server:latest"
time="2022-09-20T17:12:41Z" level=warning msg="Reason: container image info missing" container=/Owncloud image="owncloud/server:latest"
time="2022-09-20T17:14:15Z" level=info msg="Found new owncloud/server:latest image (42a03a1abaac)"
time="2022-09-20T17:14:15Z" level=info msg="Unable to update container \"/Owncloud\": no available image info. Proceeding to next."
time="2022-09-20T17:14:29Z" level=info msg="Found new lscr.io/linuxserver/bookstack:latest image (fa45e65c42dd)"
time="2022-09-20T17:15:27Z" level=info msg="Found new linuxserver/piwigo:latest image (2831dfb85d2d)"
time="2022-09-20T17:15:43Z" level=info msg="Found new ghost:latest image (f0ea389763a5)"
time="2022-09-20T17:15:48Z" level=info msg="Found new linuxserver/piwigo:latest image (2831dfb85d2d)"
time="2022-09-20T17:16:01Z" level=info msg="Found new akhilrex/podgrab:latest image (6e0094ece2a4)"
time="2022-09-20T17:16:08Z" level=info msg="Stopping /Podgrab (fcfb388705a6) with SIGTERM"
time="2022-09-20T17:16:39Z" level=info msg="Stopping /Piwigo_Thierry_Ginette (f57b12fd01e8) with SIGTERM"
time="2022-09-20T17:16:55Z" level=info msg="Stopping /ghost-ghost-1 (d15a061091a2) with SIGTERM"
time="2022-09-20T17:16:57Z" level=info msg="Stopping /Piwigo (ec4113cf9007) with SIGTERM"
time="2022-09-20T17:17:09Z" level=info msg="Stopping /Bookstack (c584c9d3bf43) with SIGTERM"
2022/09/20 17:17:31 cron: panic running job: runtime error: invalid memory address or nil pointer dereference
goroutine 16228 [running]:
github.com/robfig/cron.(*Cron).runWithRecovery.func1(0xc000076230)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:161 +0x9e
panic(0xbf89e0, 0x12aef20)
	/opt/hostedtoolcache/go/1.15.15/x64/src/runtime/panic.go:969 +0x1b9
github.com/containrrr/watchtower/pkg/container.Container.ImageID(...)
	/home/runner/work/watchtower/watchtower/pkg/container/container.go:66
github.com/containrrr/watchtower/internal/actions.stopContainersInReversedOrder(0xc000600f00, 0x20, 0x20, 0xddc440, 0xc000383a40, 0xc0003ebfb0, 0x0, 0x2540be400, 0x0, 0xc0002052d8, ...)
	/home/runner/work/watchtower/watchtower/internal/actions/update.go:130 +0x1d0
github.com/containrrr/watchtower/internal/actions.Update(0xddc440, 0xc000383a40, 0xc0003ebfb0, 0x0, 0x2540be400, 0x0, 0x12dd301, 0xc00046b740, 0xd14830, 0x49d795)
	/home/runner/work/watchtower/watchtower/internal/actions/update.go:87 +0x709
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc0003ebfb0, 0xc00054ca80)
	/home/runner/work/watchtower/watchtower/cmd/root.go:374 +0xf8
github.com/containrrr/watchtower/cmd.runUpgradesOnSchedule.func1()
	/home/runner/work/watchtower/watchtower/cmd/root.go:329 +0xb6
github.com/robfig/cron.FuncJob.Run(0xc0005ca580)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:92 +0x25
github.com/robfig/cron.(*Cron).runWithRecovery(0xc000076230, 0xdc3e00, 0xc0005ca580)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:165 +0x59
created by github.com/robfig/cron.(*Cron).run
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:199 +0x76a
time="2022-09-20T21:20:08Z" level=info msg="Waiting for running update to be finished..."
time="2022-09-20T21:20:11Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:a895c3f41cdba0b993df081e1614f38c75791159ec04ee9a362c528cecfa108c"
time="2022-09-20T21:20:11Z" level=info msg="Watchtower 1.4.0"
time="2022-09-20T21:20:11Z" level=info msg="Using notifications: smtp"
time="2022-09-20T21:20:11Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2022-09-20T21:20:11Z" level=info msg="Scheduling first run: 2022-09-21 21:20:11 +0000 UTC"
time="2022-09-20T21:20:11Z" level=info msg="Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds"
time="2022-09-21T21:20:11Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:a895c3f41cdba0b993df081e1614f38c75791159ec04ee9a362c528cecfa108c"
time="2022-09-21T21:20:19Z" level=warning msg="Could not do a head request for \"owncloud/server:latest\", falling back to regular pull." container=/Owncloud image="owncloud/server:latest"
time="2022-09-21T21:20:19Z" level=warning msg="Reason: container image info missing" container=/Owncloud image="owncloud/server:latest"
time="2022-09-21T21:21:44Z" level=info msg="Found new owncloud/server:latest image (5b5ea3a59a76)"
time="2022-09-21T21:21:44Z" level=info msg="Unable to update container \"/Owncloud\": no available image info. Proceeding to next."
time="2022-09-21T21:21:55Z" level=info msg="Found new ghcr.io/linuxserver/syncthing:latest image (9a45ba63635e)"
time="2022-09-21T21:22:08Z" level=info msg="Found new mariadb:10.6 image (670d6c59bb6a)"
time="2022-09-21T21:23:03Z" level=info msg="Found new mariadb:10.7 image (1e7307274772)"
time="2022-09-21T21:23:14Z" level=info msg="Stopping /MariaDB (def259a2f59a) with SIGTERM"
time="2022-09-21T21:23:26Z" level=info msg="Stopping /photoprism_mariadb_1 (215771f2c4ab) with SIGTERM"
time="2022-09-21T21:23:38Z" level=info msg="Stopping /syncthing (a9faf7ed7304) with SIGTERM"
2022/09/21 21:23:49 cron: panic running job: runtime error: invalid memory address or nil pointer dereference
goroutine 13 [running]:
github.com/robfig/cron.(*Cron).runWithRecovery.func1(0xc000280000)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:161 +0x9e
panic(0xbf89e0, 0x12aef20)
	/opt/hostedtoolcache/go/1.15.15/x64/src/runtime/panic.go:969 +0x1b9
github.com/containrrr/watchtower/pkg/container.Container.ImageID(...)
	/home/runner/work/watchtower/watchtower/pkg/container/container.go:66
github.com/containrrr/watchtower/internal/actions.stopContainersInReversedOrder(0xc000799b00, 0x1c, 0x20, 0xddc440, 0xc0003dc1b0, 0xc000385c60, 0x0, 0x2540be400, 0x0, 0x0, ...)
	/home/runner/work/watchtower/watchtower/internal/actions/update.go:130 +0x1d0
github.com/containrrr/watchtower/internal/actions.Update(0xddc440, 0xc0003dc1b0, 0xc000385c60, 0x0, 0x2540be400, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/runner/work/watchtower/watchtower/internal/actions/update.go:87 +0x709
github.com/containrrr/watchtower/cmd.runUpdatesWithNotifications(0xc000385c60, 0xc0004ec0e0)
	/home/runner/work/watchtower/watchtower/cmd/root.go:374 +0xf8
github.com/containrrr/watchtower/cmd.runUpgradesOnSchedule.func1()
	/home/runner/work/watchtower/watchtower/cmd/root.go:329 +0xb6
github.com/robfig/cron.FuncJob.Run(0xc0004a30e0)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:92 +0x25
github.com/robfig/cron.(*Cron).runWithRecovery(0xc000280000, 0xdc3e00, 0xc0004a30e0)
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:165 +0x59
created by github.com/robfig/cron.(*Cron).run
	/home/runner/go/pkg/mod/github.com/robfig/cron@v0.0.0-20180505203441-b41be1df6967/cron.go:199 +0x76a
time="2022-10-10T03:20:03Z" level=warning msg="Failed to retrieve container image info: Error: No such image: sha256:a895c3f41cdba0b993df081e1614f38c75791159ec04ee9a362c528cecfa108c"
time="2022-10-10T03:20:11Z" level=info msg="Watchtower 1.4.0"
time="2022-10-10T03:20:11Z" level=info msg="Using notifications: smtp"
time="2022-10-10T03:20:11Z" level=info msg="Checking all containers (except explicitly disabled with label)"
time="2022-10-10T03:20:11Z" level=info msg="Scheduling first run: 2022-10-11 03:20:11 +0000 UTC"
time="2022-10-10T03:20:11Z" level=info msg="Note that the first check will be performed in 23 hours, 59 minutes, 59 seconds"
time="2022-10-10T03:20:15Z" level=error msg="Failed to send shoutrrr notification" error="failed to send using smtp: timed out" index=0 notify=no service=smtp

I have since paused the watchtower container on this host.
All other WT instances are running just fine with less images to check. I think one of the option would be to just check and notify and let the user recreate the container manually.

[piksel]

@eskiiom The crash is due to a bug fixed in #1220 (not released in :latest), combined with something else fishy going on (the reason for the image not being available).
Ideally we would build the entire new configuration before stopping the old container so that this could never happen. I guess it wasn't a concern when developing it, since the old image shouldn't just disappear while doing the update...