Docker hub image for version 12.4 contains cryptominer [Confirmed!]

[Antender]

So, in our company we switched from this image to bitnami one (which is not technically "official" one) with minimal modifications to the descendant Dockerfile (basically 2-line fix, no changes in open ports or anything). This totally fixed the issue, no rogue processes in top, nothing. Therefore, i think this problem needs some further attention from maintainers.

P.S. Previous issue being hastly closed without waiting for investigation is pretty suspicious in itself, BTW.
Related issue: #767

[tianon]

I'm sorry you feel we were too hasty in closing the previous issue, but without some way to reproduce what you're claiming, we can only assume your infrastructure has been compromised.

Can you provide us a method to reproduce? Perhaps an executable to look for inside the published image?

[Antender]

Reproducing is actually really simple but takes some time:

1. Deploy image to a vm of your choice
2. Wait approximately 3 days
3. Docker image (postgres process specifically) spawns 8 child processes with autogenerated names 4 of which eat up lots RAM and CPU time (~100% load, tested on dual core vm)
4. Use lsof on postgres process to display outgoing http connection to port 8080 to malicious hetzner server (with apparently apache deployed).
5. Repeat steps 1 to 4 with all networking blocked except for outgoing connections to this botnet control server

[Antender]

Our own descendant image wasn't making any modifications to original Dockerfile except for one SQL file migration, so i think it's fair to say that problem would persist on ancestor image too.

[tianon]

I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it. Can you give us an exact command for running it? What specific steps are you taking to block networking in step five? A firewall in the VM, or changes to something external like a security group on the underlying hosting platform?

[Antender]

I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it.

Can't really trust your word on this one cause you're one of 2 top commiters here and probably (but not certainly!) added cryptominer in the image in the first place (if it IS in the image).

[Antender]

Using firewall in the VM software of your choice and not the guest OS would be the most bulletproof way, i think. The deployment command was the usual one we use, "docker login" to our registry then "docker swarm deploy". Yes, you can invent some more improbable theories like "your CI structure is also compromised!", but, please, use Occam's razor and investigate the simpler issue first.

[tianon]

I'm not really sure what to do to gain your trust here, but please help us help you. We need some way to reliably reproduce this -- to do so, we need the exact command you're running (especially port publishings), and exactly which firewall you're using.

The way that Docker is designed, when you publish ports from containers it uses iptables directly, which is what almost all Linux firewalls are based on. When it does so, by design it will bypass your firewall rules. See https://www.reddit.com/r/docker/comments/2fftmp/docker_will_bypass_your_firewall_by_default/ for one example (that isn't me, and isn't even recent -- 6 years ago) saying this same thing. You really cannot trust a firewall within the VM to block Docker from exposing the port. Can you please try reproducing without exposing a port on the container?

[Antender]

I'm not really sure what to do to gain your trust here, but please help us help you. We need some way to reliably reproduce this -- to do so, we need the exact command you're running (especially port publishings), and exactly which firewall you're using.

The way that Docker is designed, when you publish ports from containers it uses iptables directly, which is what almost all Linux firewalls are based on. When it does so, by design it will bypass your firewall rules. See https://www.reddit.com/r/docker/comments/2fftmp/docker_will_bypass_your_firewall_by_default/ for one example (that isn't me, and isn't even recent -- 6 years ago) saying this same thing. You really cannot trust a firewall within the VM to block Docker from exposing the port. Can you please try reproducing without exposing a port on the container?

I'm not talking about sandboxing using guest OS iptables (I stopped testing using it after previous Github discussion about ufw). We sandboxed the entire VM using VM-level settings.

And switched image in our usual CI infrastructure to bitnami one and that fixed the issue (the point you keep ignoring).

[tianon]

I'm trying to get down to the root cause of why switching to Bitnami's image fixed your issue. I'm not ignoring it, I'm trying to explore it, and you're not giving me enough information to do so.

Like I said above, I can't help you unless you help me. I have literally no incentive to embed a cryptominer in this image, and there are enough other folks in the community that review this work that attempting to do so would not go unnoticed. The amount of money we could get from a cryptominer absolutely pales compared to the loss of reputation, there's seriously no reason we would do so.

If you're not willing to have this discussion in good faith and try to help us figure out what's going on with your deployment and why switching fixed it, then I really don't know where to go from here. If you can't trust me long enough to reproduce the problem, how can I know you'll trust the solution? At this point, it honestly doesn't seem like you're interested in an actual solution within this repository, so what is the end goal here?

[Antender]

I've got an instance I've been running for months and haven't seen this issue, so there's got to be more to it. Can you give us an exact command for running it? What specific steps are you taking to block networking in step five? A firewall in the VM, or changes to something external like a security group on the underlying hosting platform?

Also your own server isn't proving that issue doesn't exist: i presume, you blocked every possible connection server could made to botnet control server, so malware couldn't start in the first place. I'm not trying to blame specifically you, I'm just asking to investigate image-related issue first before putting blame on network-related attack which is less probable.

[Antender]

The endgoal is to fix image for everyone who is not as skilled in networking as you (like myself) and trusts public "official" image. And stop someone from profitting from mining, obviously. Firewalling everything kinda fixes symptoms but isn't solving the underlying suspected cause.

[ImreSamu]

Docker hub image for version 12.4 contains cryptominer [Confirmed!]

I am using the postgis 12.4 image based on the official postgres:12.4 image.

• and I have not detected any cryptominer ,
• and have not see any cryptominers reports here: https://github.com/postgis/docker-postgis/issues

My hypothesis to test:

• what language you are using - for connecting to Postgres ?
  • PHP? Nodejs, Python?
    • imho: Any programming library, can be infected

[Antender]

My hypothesis to test:

* what language you are using - for connecting to Postgres ?
  
  * PHP? Nodejs, Python?
    
    * imho: Any programming library, [can be infected ](https://blog.sucuri.net/2018/01/malicious-cryptominers-from-github-part-2.html)

How is this supposed to be related if:

1. DB is in it's own container without any node.js code in it
2. Main postgres process spawns malicious child processes (so, presumably, postgres binary is the problem).
3. Child processes aren't using the normal postgres naming scheme for processes (which supports malicious binary theory)

[wglambert]

Without a way of reproducing the issue it would seem to be an instance of the running container being compromised similar to:
#664 (comment)
#446
redis/docker-library-redis#235 (comment)
redis/docker-library-redis#226 (comment)

I would guess that in using a different image your security settings were changed to thwart a repeat intrusion, or there is a delay with it being compromised (if no security settings were changed)
As stated before everything the image can do innately is encompassed by the Dockerfile and Entrypoint available in this repo. Additionally if the image were innately compromised I would imagine there would be significantly more users encountering this issue, it seems to be limited to your environment

[Antender]

Without a way of reproducing the issue it would seem to be an instance of the running container being compromised similar to:
#664 (comment)
#446
docker-library/redis#235 (comment)
docker-library/redis#226 (comment)

I would guess that in using a different image your security settings were changed to thwart a repeat intrusion, or there is a delay with it being compromised (if no security settings were changed)
As stated before everything the image can do innately is encompassed by the Dockerfile and Entrypoint available in this repo. Additionally if the image were innately compromised I would imagine there would be significantly more users encountering this issue, it seems to be limited to your environment

1. "hacking" theory wasn't confirmed on SO. Everyone just assumed that's what happened and comments to question on SO suggest that the user of image was doing everything he can to mitigate the issue.
2. The same story.
3. and 4) We use no redis in our deployment.

Issue 2) was closed by @tianon without asking any questions to issue creator. And he was THE person who suggested the "hacking" theory every time without actually proving it in the first place.

In issue 1) nothing proves that postgres file wasn't infected. Basically, the image user just blocked outside world connections (which i admit is a good practice!) for the image and botnet couldn't start. The problem is that 1) by default the image isn't blocking connections (isn't fixable) 2) This isn't solving the issue for less networking-savvy users. 3) Distrubuting compromised images (IF it is compromised) is still a problem in itself.

I repeat: firewalling everything fixes symptoms but leaves all the people who haven't blocked botnet master server with a problem.

Before everyone repeats "hacking" theory another time the story goes like this:

1. For months we built images in our CI and they were behaving like this but the problem wasn't very annoying cause DB container almost always was short-lived.
2. We finally decided to fix the issues not by switching passwords on DB instance, not by changing the network configuration, but by switching to another base image with default settings
3. Problem magically disappears
4. Everyone still discusses network "hacking" theory instead of inspecting image contents. Astounding!

You can close the issue, we fixed it for ourselves. I can't care anymore and waste even more time on proving fringe theories when I have all the arguments in favor of postgres binary being compromised in the first place. Just be aware, that if you read this you should have been building your own image from the start and not relying on "official" images.