MariaDB service crashes since updating to Laravel 11 oom-kill possible memory leak in Laravel 11?

[sts-ryan-holton]

Laravel Version

11.1.0

PHP Version

8.3.4

Database Driver & Version

MariaDB 10.11

Description

Hi 👋

I thought I'd just mention this in case anyone else has experienced this since updating to Laravel 11.

I use MariaDB 10.11 in production. Since updating, I'm seeing that my database has been crashing. I've been running my server for a few years now, and it's always been operating on the latest Laravel version, and also has been fine sitting between 75% and 90% memory utilisation (8GB).

Steps To Reproduce

Two noticeable things are different in Laravel 11 compared to Laravel 10:

1. Removal of dbal package
2. Mariadb driver

I wanted to figure out whether I'm missing anything. As a result of this, at a cost, I've had to increase my server's memory. I do wonder whether there's a memory leak in how the driver works now in Laravel 11?

× mariadb.service - MariaDB 10.11 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: failed (Result: oom-kill) since Thu 2024-03-28 04:00:36 EDT; 5min ago
   Duration: 1d 1h 19min 29.156s
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 766 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
    Process: 825 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
    Process: 874 ExecStart=/usr/libexec/mariadbd --basedir=/usr $MYSQLD_OPTS $_WSREP_NEW_CLUSTER (code=killed, signal=KILL)
    Process: 1443 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
   Main PID: 874 (code=killed, signal=KILL)
     Status: "Taking your SQL requests now..."
        CPU: 2h 39min 52.288s

Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   1. Back-up your data before with 'mariadb-upgrade'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   2. Start the database daemon using 'systemctl start mariadb.service'
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]:   3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]: Read more about 'mariadb-upgrade' usage at:
Mar 27 02:41:07 domain-monitor-2022 mariadb-check-upgrade[1473]: https://mariadb.com/kb/en/mysql_upgrade/
Mar 27 02:41:07 domain-monitor-2022 systemd[1]: Started MariaDB 10.11 database server.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: A process of this unit has been killed by the OOM killer.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Failed with result 'oom-kill'.
Mar 28 04:00:36 domain-monitor-2022 systemd[1]: mariadb.service: Consumed 2h 39min 52.288s CPU time.

It's been almost 3 hours since rebooting my database server, and I'm seeing the memory usage here at 1GB

● mariadb.service - MariaDB 10.11 database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; preset: disabled)
     Active: active (running) since Thu 2024-03-28 04:25:45 EDT; 2h 23min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 834 ExecStartPre=/usr/libexec/mariadb-check-socket (code=exited, status=0/SUCCESS)
    Process: 887 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir mariadb.service (code=exited, status=0/SUCCESS)
    Process: 1182 ExecStartPost=/usr/libexec/mariadb-check-upgrade (code=exited, status=0/SUCCESS)
   Main PID: 946 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 95 (limit: 100205)
     Memory: 1.0G
        CPU: 15min 19.823s
     CGroup: /system.slice/mariadb.service
             └─946 /usr/libexec/mariadbd --basedir=/usr

Mar 28 04:25:44 domain-monitor-2022 systemd[1]: Starting MariaDB 10.11 database server...
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir[887]: Database MariaDB is probably initialized in /var/lib/mysql already, nothing is done.
Mar 28 04:25:44 domain-monitor-2022 mariadb-prepare-db-dir[887]: If this is not the case, make sure the /var/lib/mysql is empty before running mariadb-prepare-db-dir.
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: The datadir located at /var/lib/mysql needs to be upgraded using 'mariadb-upgrade' tool. This can be done using the fo>
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   1. Back-up your data before with 'mariadb-upgrade'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   2. Start the database daemon using 'systemctl start mariadb.service'
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]:   3. Run 'mariadb-upgrade' with a database user that has sufficient privileges
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: Read more about 'mariadb-upgrade' usage at:
Mar 28 04:25:45 domain-monitor-2022 mariadb-check-upgrade[1212]: https://mariadb.com/kb/en/mysql_upgrade/
Mar 28 04:25:45 domain-monitor-2022 systemd[1]: Started MariaDB 10.11 database server.

[driesvints]

Hey there,

Can you first please try one of the support channels below? If you can actually identify this as a bug, feel free to open up a new issue with a link to the original one and we'll gladly help you out.

• Laracasts Forums
• Laravel.io Forums
• StackOverflow
• Discord
• Larachat
• IRC

Thanks!

[sts-ryan-holton]

No worries. Created https://stackoverflow.com/questions/78238067/mariadb-service-crashes-since-updating-to-laravel-11-oom-kill-possible-memory-le for reference

[sts-ryan-holton]

@driesvints Followed the advice on Stackoverflow. I'm being pushed back here mate :)

I would raise this as a bug report on laravel's support page, not here.

Hoping to see if others get a similar problem

[driesvints]

I've retained DB_CONNECTION=mysql in production

You need to use the new mariadb connection on Laravel v11.

[sts-ryan-holton]

Sure, I used Laravel Shift a few days ago to update. It didn't change or mention anything about this. Does this mean I need to migrate all uuid columns over to this? Not quite sure of the compatibility of this if I ever want to go back to MySQL.

Right now I see first party packages like Pulse continuing to use uuid so should be fine? And failed_jobs table uses uuid.

Note, after changing DB_CONNECTION to mariadb I'm not seeing any reduction in memory. Currently at 2.3GB after 6 hours

[driesvints]

If you use mariadb you should best use that connection. This will make sure you don't deviate from it in the future. You should update your uuid columns as in the upgrade guide. If it's the same problem then that's most likely not it. You're also free to stay on mysql if you really want that.

@hafezdivandari @staudenmeir do any of you have any ideas what could cause the memory for the new MariaDB connection to blow up like above?

@sts-ryan-holton can you please add more info to reproduce this? What sort of queries/usage is causing this?

[sts-ryan-holton]

I've switched to mariadb for now. I used Laravel Shift, and there were no problems there. A bit more about the setup:

I updated to Laravel 11 and deployed the changes. Then almost 24 hours after the deployment, mariadb.service failed completely with the error attached above. The server had 8gb of memory on a Digital Ocean droplet and 4vcpus. It's had that setup for at least 2 years. There were no problems on Laravel 9, and on Laravel 10. I'm running MariaDB 10.11

My server has always ran at around 75-90% memory. When it crashed, moments before the memory usage went up to around 98% and fell over, I restarted and 24 hours later to this morning, same thing happened. And I've never had this happen ever despite running hot on memory.

I run:

• Laravel Horizon, roughly processing between 1,000 - 2,000 jobs per minute continuously and have around 60-70 PHP processes running
• Laravel Pulse (separate Redis injest)

CPU utilisation is never really that high, and when running SHOW FULL PROCESSLIST on the DB there's only around 70 active connections at any one given time.

This morning, I updated my droplet at a £30 monthly increase to double my CPU and memory. The server has been rebooted to perform that change earlier.

Earlier this morning when opening this issue, look at the memory usage:

Tasks: 95 (limit: 100205)
     Memory: 1.0G
        CPU: 15min 19.823s

And right now:

 Tasks: 98 (limit: 100205)
     Memory: 2.5G
        CPU: 48min 52.869s

Over 1.5GB extra consumed over just 5 hours?

I have another project that's got around 4 database queues running Laravel 10, it's been up for over a week and is at 1.0GB, and I've observed this project over an hour, it goes up to 1.1GB then back down to 1.0GB.

Also worth mentioning, I haven't made any config overrides to my.cnf for MariaDB at all in the past 2 years.

My worry here is that something in Laravel 11 is causing an increase in memory usage.

Laravel Shift recommended I add the following back after

APP_TIMEZONE=Europe/London
BCRYPT_ROUNDS=10
CACHE_STORE=file
DB_CHARSET=utf8mb4
DB_COLLATION=utf8mb4_unicode_ci
DB_CONNECTION=mysql
LOG_DAILY_DAYS=3
MAIL_MAILER=smtp
MAIL_MARKDOWN_THEME=domain-monitor
QUEUE_CONNECTION=sync
QUEUE_FAILED_DRIVER=database
SESSION_DRIVER=file

And of course, I removed doctrine/dbal

[github-actions]

Thank you for reporting this issue!

As Laravel is an open source project, we rely on the community to help us diagnose and fix issues as it is not possible to research and fix every issue reported to us via GitHub.

If possible, please make a pull request fixing the issue you have described, along with corresponding tests. All pull requests are promptly reviewed by the Laravel team.

Thank you!

[sts-ryan-holton]

Just checked another server too running Centos. Except, not Laravel, it's at 1.1GB over 1 month. So I have:

• Wordpres Centos 9 -> 1.1GB over 1 month
• Laravel 10 Centos 9 -> 1.0GB over 1 week
• Laravel 10 Centos 9 (before Laravel 11) unknown but never crashed in 2 years
• Laravel 11 Centos 9 after upgrade, 2.8GB after just 8 hours 👁️

[staudenmeir]

Does this server only run MariaDB or your whole application?

How does the number of active connections develop over time? Does it only increase or stay in the same range?
Do you have the Laravel 10 numbers for comparison?

do any of you have any ideas what could cause the memory for the new MariaDB connection to blow up like above?

I don't really see how Laravel could cause these crashes since it "only" sends queries to MariaDB. Laravel itself doesn't allocate any database memory or run background tasks inside the database.

#50044 is the "most fundamental" change in Laravel 11 I can think of, but I haven't seen any other reports of increased memory usage.

Two noticeable things are different in Laravel 11 compared to Laravel 10:
Removal of dbal package

This mostly (or even only) affects migration queries.

Mariadb driver

The MySQL and MariaDB driver are identical except for the UUID column type (which only affects migration queries for new columns).