Parallelised execution of static content deploy is broken on 2.3-develop

[hostep]

Preconditions (*)

1. Tested on Magento 2.3-develop branch (works fine on version 2.3.1 btw)
2. PHP 7.1.27 with pcntl extension enabled

Steps to reproduce (*)

1. Checkout the 2.3-develop branch from Magento 2 repo (I've used commit 93a8162)
2. Run composer install
3. Run bin/magento setup:install ...
4. Run bin/magento setup:static-content:deploy -f --jobs=2

Expected result (*)

1. SCD runs without errors

Actual result (*)

$ bin/magento setup:static-content:deploy -f --jobs=2

Deploy using quick strategy
PHP Fatal error:  Uncaught TypeError: pcntl_waitpid() expects parameter 1 to be integer, boolean given in app/code/Magento/Deploy/Process/Queue.php:342
Stack trace:
#0 app/code/Magento/Deploy/Process/Queue.php(342): pcntl_waitpid(false, NULL, 1)
#1 app/code/Magento/Deploy/Process/Queue.php(234): Magento\Deploy\Process\Queue->isDeployed(Object(Magento\Deploy\Package\Package))
#2 app/code/Magento/Deploy/Process/Queue.php(215): Magento\Deploy\Process\Queue->executePackage(Object(Magento\Deploy\Package\Package), 'frontend/Magent...', Array, false)
#3 app/code/Magento/Deploy/Process/Queue.php(168): Magento\Deploy\Process\Queue->assertAndExecute('frontend/Magent...', Array, Array)
#4 app/code/Magento/Deploy/Strategy/QuickDeploy.php(76): Magento\Deploy\Process\Queue->process()
#5  in app/code/Magento/Deploy/Process/Queue.php on line 342

Fatal error: Uncaught TypeError: pcntl_waitpid() expects parameter 1 to be integer, boolean given in app/code/Magento/Deploy/Process/Queue.php:342
Stack trace:
#0 app/code/Magento/Deploy/Process/Queue.php(342): pcntl_waitpid(false, NULL, 1)
#1 app/code/Magento/Deploy/Process/Queue.php(234): Magento\Deploy\Process\Queue->isDeployed(Object(Magento\Deploy\Package\Package))
#2 app/code/Magento/Deploy/Process/Queue.php(215): Magento\Deploy\Process\Queue->executePackage(Object(Magento\Deploy\Package\Package), 'frontend/Magent...', Array, false)
#3 app/code/Magento/Deploy/Process/Queue.php(168): Magento\Deploy\Process\Queue->assertAndExecute('frontend/Magent...', Array, Array)
#4 app/code/Magento/Deploy/Strategy/QuickDeploy.php(76): Magento\Deploy\Process\Queue->process()
#5  in app/code/Magento/Deploy/Process/Queue.php on line 342

Discussion

Issue was probably introduced in 7421dfb#diff-415a71577d8921cc45d695471db3a36fL340 (not verified yet)

[m2-assistant]

Hi @hostep. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento-engcom-team give me 2.3-develop instance - upcoming 2.3.x release

For more details, please, review the Magento Contributor Assistant documentation.

@hostep do you confirm that you was able to reproduce the issue on vanilla Magento instance following steps to reproduce?

• yes
• no

[hostep]

Reverting that getPid method to the original doesn't solve the issue:

return isset($this->processIds[$package->getPath()])
            ? $this->processIds[$package->getPath()]
            : null;

But it gives a slightly different error message:

$ bin/magento setup:static-content:deploy -f --jobs=2

Deploy using quick strategy
PHP Fatal error:  Uncaught TypeError: pcntl_waitpid() expects parameter 1 to be integer, null given in app/code/Magento/Deploy/Process/Queue.php:342
Stack trace:
#0 app/code/Magento/Deploy/Process/Queue.php(342): pcntl_waitpid(NULL, NULL, 1)
#1 app/code/Magento/Deploy/Process/Queue.php(234): Magento\Deploy\Process\Queue->isDeployed(Object(Magento\Deploy\Package\Package))
#2 app/code/Magento/Deploy/Process/Queue.php(215): Magento\Deploy\Process\Queue->executePackage(Object(Magento\Deploy\Package\Package), 'frontend/Magent...', Array, false)
#3 app/code/Magento/Deploy/Process/Queue.php(168): Magento\Deploy\Process\Queue->assertAndExecute('frontend/Magent...', Array, Array)
#4 app/code/Magento/Deploy/Strategy/QuickDeploy.php(76): Magento\Deploy\Process\Queue->process()
#5 /Use in app/code/Magento/Deploy/Process/Queue.php on line 342

Fatal error: Uncaught TypeError: pcntl_waitpid() expects parameter 1 to be integer, null given in app/code/Magento/Deploy/Process/Queue.php:342
Stack trace:
#0 app/code/Magento/Deploy/Process/Queue.php(342): pcntl_waitpid(NULL, NULL, 1)
#1 app/code/Magento/Deploy/Process/Queue.php(234): Magento\Deploy\Process\Queue->isDeployed(Object(Magento\Deploy\Package\Package))
#2 app/code/Magento/Deploy/Process/Queue.php(215): Magento\Deploy\Process\Queue->executePackage(Object(Magento\Deploy\Package\Package), 'frontend/Magent...', Array, false)
#3 app/code/Magento/Deploy/Process/Queue.php(168): Magento\Deploy\Process\Queue->assertAndExecute('frontend/Magent...', Array, Array)
#4 app/code/Magento/Deploy/Strategy/QuickDeploy.php(76): Magento\Deploy\Process\Queue->process()
#5 /Use in app/code/Magento/Deploy/Process/Queue.php on line 342

Notice the difference in expects parameter 1 to be integer, boolean given vs expects parameter 1 to be integer, null given.
So there is a more complex problem going on here.

[hostep]

Git bisecting reveals the following commit to be the cause: 147d11d

And indeed, removing the line declare(strict_types=1); makes it "work" again on my machine.

But this doesn't feel right, I think there is still some deeper problem here, making the PHP execution more stricter is just revealing the problem more clearly, I believe there is still an actual issue here: passing null to pcntl_waitpid. This is probably why when I try to execute the SCD command with pcntl enabled on my local machine never seems to end, it keep running forever.

Will do some more digging.

[hostep]

I can get it more or less working again with the following code changes:

diff --git a/app/code/Magento/Deploy/Process/Queue.php b/app/code/Magento/Deploy/Process/Queue.php
index fd7aad44e0a..383b67b5d3c 100644
--- a/app/code/Magento/Deploy/Process/Queue.php
+++ b/app/code/Magento/Deploy/Process/Queue.php
@@ -340,7 +340,7 @@ class Queue
             if ($package->getState() === null) {
                 // phpcs:ignore Magento2.Functions.DiscouragedFunction
                 $pid = pcntl_waitpid($this->getPid($package), $status, WNOHANG);
-                if ($pid === $this->getPid($package)) {
+                if ($this->getPid($package) !== 0 && $pid === $this->getPid($package)) {
                     $package->setState(Package::STATE_COMPLETED);

                     unset($this->inProgress[$package->getPath()]);
@@ -354,14 +354,14 @@ class Queue
     }

     /**
-     * Returns process ID or null if not found.
+     * Returns process ID or 0 if not found.
      *
      * @param Package $package
-     * @return int|null
+     * @return int
      */
     private function getPid(Package $package)
     {
-        return isset($this->processIds[$package->getPath()]) ?? null;
+        return $this->processIds[$package->getPath()] ?? 0;
     }

     /**

But the command still seems to run endlessly on my local machine. Never quits and doesn't seem to execute anything.
This also happens on Magento 2.3.1, so I'm thinking this is a local environment problem I'm having (macOS 10.14.4, and PHP installed with MacPorts).

According to a discussion on Slack yesterday, there was at least one person who got it actually executing fine using Docker, but I don't have that setup here right now. So maybe somebody who is able to run this parallelised SCD can test on 2.3.1 and then also on 2.3-develop after applying the above changes and see if that restores the functionality again?

Thanks!

[magento-engcom-team]

✅ Confirmed by @maheshWebkul721
Thank you for verifying the issue! 👍 Your confirmation will help us to acknowledge and process this report.

[m2-assistant]

Hi @maheshWebkul721. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Verify that the issue is reproducible on 2.2-develop branch.
  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x