Timeout on import script

[michnovka]

when I run meili:import, after it indexes the last batch I get a timeout (very quickly, like second after it shows last message):

sudo -u nginx php -d memory_limit=1G bin/console meili:import --no-debug
Importing for index App\Entity\Log\AdminActionLog
Indexed a batch of 164 / 164 App\Entity\Log\AdminActionLog entities into app_dev_admin_action_logs index (164 indexed since start)
Setting "filterableAttributes" updated of "app_dev_admin_action_logs".
Setting "searchableAttributes" updated of "app_dev_admin_action_logs".
Setting "sortableAttributes" updated of "app_dev_admin_action_logs".
Importing for index App\Entity\Log\CronLog
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (500 indexed since start)
...
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (401500 indexed since start)
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (402000 indexed since start)
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (402500 indexed since start)
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (403000 indexed since start)
Indexed a batch of 456 / 456 App\Entity\Log\CronLog entities into app_dev_cron_logs index (403456 indexed since start)
Setting "filterableAttributes" updated of "app_dev_cron_logs".

In Tasks.php line 58:
                     
  Request timed out  
                     

Any idea what might be going on?

[norkunas]

@curquiza do you know what's going on? because I've also facing some timeout issues in production when updating settings.

@michnovka I think you would get a timeout for this even without indexing, there is something wrong with meilisearch settings api..

[norkunas]

You can of course use settings update command with a long timeout (30s works for me)

[michnovka]

Yea, I was reviewing https://github.com/meilisearch/meilisearch-symfony/pull/132/files - but the default one is 5000ms, and I am SURE it did not wait 5seconds before showing that error. 1sec max

[norkunas]

Yea, I was reviewing https://github.com/meilisearch/meilisearch-symfony/pull/132/files - but the default one is 5000ms, and I am SURE it did not wait 5seconds before showing that error. 1sec max

https://github.com/meilisearch/meilisearch-php/blob/ec43c8acf9aa7ecc5bdc81f8a32e903c46577aea/src/Endpoints/Delegates/HandlesTasks.php#L45

default is 5s even in sdk, so it is strange then. probably this error is coming from http client

[curquiza]

Hello

What happens when you change the response-timeout?

[michnovka]

You know, I gave up last time, and just dumped my DB prior to importing, as it had just useless log.

Now after upgrading to meili 1.14 and DB incompatibility between old version, I had to reimport. And the same issue is there...

[root@dockontrol-linode dockontrol]# php bin/console meili:import
Importing for index App\Entity\Log\UserActionLog
Indexed a batch of 258 / 258 App\Entity\Log\UserActionLog entities into app_dev_user_action_logs index (258 indexed since start)
Setting "filterableAttributes" updated of "app_dev_user_action_logs".
Setting "searchableAttributes" updated of "app_dev_user_action_logs".
Setting "sortableAttributes" updated of "app_dev_user_action_logs".
Importing for index App\Entity\Log\CronLog
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (500 indexed since start)
...
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (437500 indexed since start)
Indexed a batch of 500 / 500 App\Entity\Log\CronLog entities into app_dev_cron_logs index (438000 indexed since start)
Indexed a batch of 421 / 421 App\Entity\Log\CronLog entities into app_dev_cron_logs index (438421 indexed since start)
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000761 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000761},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.697939+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.748315+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000616 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000616},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.749021+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.799567+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000699 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000699},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.800354+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.850708+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000728 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000728},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.851525+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.901881+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000685 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000685},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.902674+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.953112+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.003496 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.003496},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:43.956741+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.007423+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000794 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000794},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.008326+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.058723+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.001097 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.001097},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.059943+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.110342+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000905 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000905},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.111339+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.161731+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.001005 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.001005},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.162832+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.213263+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000743 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000743},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.214106+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.264510+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000739 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000739},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.265340+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.315710+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000741 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000741},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.316539+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.366928+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000779 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000779},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.367816+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.418224+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.00062 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.00062},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.418950+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.469321+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000715 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000715},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.470121+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.520475+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000816 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000816},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.521381+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.571742+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.00073 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.00073},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.572561+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.622949+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000748 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000748},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.623787+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.674358+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000625 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000625},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.675068+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.725623+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000881 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000881},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.726591+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.776958+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000747 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000747},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.777793+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.828316+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000726 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000726},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.829131+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.879696+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000734 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000734},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.880506+02:00","extra":{}}
{"message":"Request: \"GET http://127.0.0.1:7700/tasks/962\"","context":{},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.930858+02:00","extra":{}}
{"message":"Response: \"200 http://127.0.0.1:7700/tasks/962\" 0.000769 seconds","context":{"http_code":200,"url":"http://127.0.0.1:7700/tasks/962","total_time":0.000769},"level":200,"level_name":"INFO","channel":"http_client","datetime":"2025-05-19T13:22:44.931713+02:00","extra":{}}
{"message":"Error thrown while running command \"'meili:import'\". Message: \"Request timed out\"","context":{"exception":{"class":"Meilisearch\\Exceptions\\TimeOutException","message":"Request timed out","code":408,"file":"/var/www/dockontrol/vendor/meilisearch/meilisearch-php/src/Endpoints/Tasks.php:58"},"command":"'meili:import'","message":"Request timed out"},"level":500,"level_name":"CRITICAL","channel":"console","datetime":"2025-05-19T13:22:44.987339+02:00","extra":{}}
13:22:44 CRITICAL  [console] Error thrown while running command "'meili:import'". Message: "Request timed out" ["exception" => Meilisearch\Exceptions\TimeOutException^ { …},"command" => "'meili:import'","message" => "Request timed out"]
{"message":"Command \"'meili:import'\" exited with code \"408\"","context":{"command":"'meili:import'","code":408},"level":100,"level_name":"DEBUG","channel":"console","datetime":"2025-05-19T13:22:45.013705+02:00","extra":{}}

What should I do?

[jahrsensetence]

We are running into the same issue. For us it's a show-stopper. I also have the impression that the timeout is simply not used, for us in the swap-operation. Looks like we are running into memory issues. I am not 100% sure about this but deleting the old index and then creating a new index worked much better. Also I cannot fully reproduce this on my machine as I never ran into issues with 200k documents.

Anyhow. I added sume debug outputs and saw that in my case, setting some attributes on the index caused the delays:

^ "Iteration done, current timeout: 19250 of 120000 ms"
^ "Task 1760 completed with status: succeeded"
Setting "searchableAttributes" updated of "_tmp_xyz_dev_article".

So setting searchableAttributes took 19 seconds. If I run the same thing without the existing index, this is finished "in no time".

[norkunas]

Someone needs to create a reproducer then

[niravpateljoin]

hi @norkunas

Hi, I’ve created a minimal reproducer here:

1. Clone the repository.
2. Run the fixtures:
   php bin/console doctrine:fixtures:load --no-debug --no-interaction
3. Index them in Meilisearch:
   `php bin/console meilisearch:import``

After importing some records, the process fails with:

Indexed a batch of 500 / 500 App\Entity\Post entities into app_dev_posts index (44500 indexed since start)
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/meilisearch-testing/vendor/symfony/serializer/Debug/TraceableSerializer.php on line 164

[norkunas]

@niravpateljoin hey. your reproducer misses database and php at least :)
Also your request is about OOM, not a timeout. But if you'll update it to simply run, I'll check

[niravpateljoin]

hi @norkunas yeah, I understand that I mentioned a different error message earlier — my bad.

Here are the steps to reproduce the issue:

• Symfony version: 7.3
• Database: Mariadb
• PHP version: 8.4

1. Clone the repo.
2. Set up database creds in .env.local.
3. Then run: php bin/console doctrine:database:create.
4. Then we need to run the fixture: php bin/console doctrine:fixtures:load --no-debug --no-interaction.
5. Once we have data in our database: then we simply import it in meilisearch: php -d memory_limit=1G bin/console meilisearch:import --no-debug

and the process ends with the following message:

[norkunas]

@niravpateljoin then you should add everything that is missing to docker compose, btw you should use dbal v4, it's already released long time ago

[niravpateljoin]

@norkunas

Here’s what we need to do now:

1. Clone the repo.
2. Run make init.
3. Once we have data in the database, simply import it into Meilisearch by running:
   php -d memory_limit=1G bin/console meilisearch:import --no-debug.

Even after upgrading to DBAL 4, we still encounter the same error.

[norkunas]

@niravpateljoin

$ make init
> docker compose --env-file .env up -d
[+] Running 9/9
 ✔ database Pulled                                                                                                                                                                                                                                             6.0s 
   ✔ 89dc6ea4eae2 Pull complete                                                                                                                                                                                                                                1.9s 
   ✔ 0d5da168f749 Pull complete                                                                                                                                                                                                                                1.9s 
   ✔ 1007abdfb9c9 Pull complete                                                                                                                                                                                                                                2.3s 
   ✔ 9321dc82d393 Pull complete                                                                                                                                                                                                                                2.3s 
   ✔ 713dcb928734 Pull complete                                                                                                                                                                                                                                2.3s 
   ✔ 30989e86eda4 Pull complete                                                                                                                                                                                                                                3.8s 
   ✔ 4920b5f8d410 Pull complete                                                                                                                                                                                                                                3.8s 
   ✔ 92d60c0306e8 Pull complete                                                                                                                                                                                                                                3.8s 
[+] Running 3/3
 ✔ Network meilisearch-testing_default  Created                                                                                                                                                                                                                0.1s 
 ✔ Container meilisearch                Started                                                                                                                                                                                                                0.5s 
 ✔ Container mariadb                    Started                                                                                                                                                                                                                0.5s 
⏳ Waiting for database on 127.0.0.1:3308...
/bin/sh: 1: mysqladmin: not found
/bin/sh: 1: mysqladmin: not found
/bin/sh: 1: mysqladmin: not found
...

:)