Docker logs are not being logrotated

[NecronMt]

I'm using EE 4.6.3 and I believe that the docker logs are not being log rotated upon investigating my hard disk size filling up quite fast.

Under /etc/docker/daemon.json you need to add:

"log-driver": "json-file",
"log-opts": {
"max-size": "20m",
"max-file": "10",
}

Also noticed that CPU usage is higher on PHP-FPM on this new version. Any ideas?

If there is a setting that I have missed, please do let me know. Thanks

[IamParantapa1]

Same with me

[mrrobot47]

I will be looking into this in the upcoming release.

[mrrobot47]

Re: PHP issue I will be updating further here only.

[NecronMt]

seems like even after restarting the docker service using systemctl restart docker, it is still creating quite a large file (5gb in around 12hours). Any other solution?

[NecronMt]

I'm using this to get the top 20 file sizes:
sudo du -a / | sort -n -r | head -n 20

[NecronMt]

Seems like it is filling up fast with this error:
{"log":"2023-01-01 23:54:02 285107 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'hist_type' at position 9 to have type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB','JSON_HB'), found type enum('SINGLE_PREC_HB','DOUBLE_PREC_HB').\n","stream":"stderr","time":"2023-01-01T23:54:02.639869019Z"} {"log":"2023-01-01 23:54:02 285107 [ERROR] Incorrect definition of table mysql.column_stats: expected column 'histogram' at position 10 to have type longblob, found type varbinary(255).\n","stream":"stderr","time":"2023-01-01T23:54:02.640043279Z"}

my server is running Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-56-generic x86_64)

do I need to run a mysql upgrade from docker

[NecronMt]

to fix the above issue:

get the MariaDB container ID

docker ps

get the root mysql password

cat /opt/easyengine/services/docker-compose.yml | grep MYSQL_ROOT_PASSWORD

Execute the mariadb bash
docker exec -it CONTAINERID /bin/bash;

In the shell of the container, we execute the following to automatically fix a variety of problems/errors
mysql_upgrade --user=root --password=YOURPASS;

[NecronMt]

Hi any news on this one? It's weird that the hard disc is filling up full every 4 days with 20GB of free space. I'm on Ubuntu 22.04

[mrrobot47]

Fix has been shipped in the nightly build. It will be released on Monday. To update to nightly, use command:

ee cli update --nightly

You can update to the nightly build and test it over the weekend and see what is the status of the disk.
(As this is nightly build, please take server backup before updating. It is release ready and tested, still always take server backups before any updates as a rule of thumb.)

[mrrobot47]

Fixed in EE v4.6.5. To update, use command:

ee cli update

[NecronMt]

Updated but still doesn't resolve the issue coming from docker logs. You can see that on installation, the docker logs get cleared.

[mrrobot47]

Please share the output of command:

du -csh /var/lib/docker/containers/*/*-json.log

No single log file should be greater than 10MB.

[NecronMt]

agreed, those log files do not exceed 10mb. One of the files I found to be huge is /var/lib/docker/volumes/global-nginx-proxy_nginx_proxy_logs/_data/access.log also noticed that the container php and nginx logs are not being log rotated

[NecronMt]

after running: sudo du -h /var/ | sort -rh | head -10

24G /var/
19G /var/lib
18G /var/lib/docker
9.3G /var/lib/docker/overlay2
8.1G /var/lib/docker/volumes
4.6G /var/log
4.5G /var/log/journal
4.0G /var/log/journal/54c410bcea63dc302ab2ac476047b8eb

Also seems like upon further investigation it's the overlay stuff that keeps on piling up:
526M /var/lib/docker/overlay2/3e822cf5db110f8eb8e68ad8fe42de78ef4b2c291b8b67eb142427a387d6bc2b
526M /var/lib/docker/overlay2/21e21eaddcbf94772e3c22abd0d1a53719bb1e7a50d3dfc8f3f2300a9c3f11f2/merged
526M /var/lib/docker/overlay2/21e21eaddcbf94772e3c22abd0d1a53719bb1e7a50d3dfc8f3f2300a9c3f11f2
526M /var/lib/docker/overlay2/1b748a82cacc973e87a21ee69370ad916837e73703353b3af79c973f2a9982b7/merged
526M /var/lib/docker/overlay2/1b748a82cacc973e87a21ee69370ad916837e73703353b3af79c973f2a9982b7

[NecronMt]

Ok so seems like i found the culprit, the logrotations are not working correctly on all of the sites. These include nginx and php error logs. Is there any command I should run to get it to work properly?

I'm also getting a ton of errors inside of the php one with:
[27-Jan-2023 23:55:55] WARNING: [pool www] child 28 said into stderr: "NOTICE: PHP message: Connection refused"

Other than that, I checked that the logrotate.d/easyengine has all the sites listed.
ex:

/opt/easyengine/sites/MYSITE/logs/nginx/*.log
/opt/easyengine/sites/MYSITE/logs/php/*.log
{
        rotate 7
        daily
        compress
        size 5M
        maxage 30
}

it seems that only the error.log and access.log in each folder aren't rotated correctly, resulting in huge file sizes.

[NecronMt]

Can anyone help?

[ssuess]

I have the same problem, nginx access file is huge:
12G /var/lib/docker/volumes/global-nginx-proxy_nginx_proxy_logs/_data
How can I get this to rotate? Is it safe to just delete this file for now?

[ssuess]

But seriously there needs to be a way for this to logrotate WTF

[NecronMt]

I tried adding logrotate through docker but that resulted in an ever more chaotic file size change as it might have been conflicting. Not sure what else to do...

[mrrobot47]

You can do, echo "" > /path/to/log/file and then ee site reload sitename.tld instead of delete and reboot.

I'll be adding log rotate in the next release. ETA 2 weeks.

[NecronMt]

ok thanks keep us posted

[NecronMt]

Hi there any news on this one please? Thank you