Write a Github Action that runs the static analysis tests as described in the https://developer.adobe.com/commerce/testing/guide/static/

[damienwebdev]

We should do this for two separate things:

These actions should be added here: https://github.com/graycoreio/github-actions-magento2

1. The current version in v2.4-develop
2. The v2.4.5-p1 release

I would expect to see a failure (or a success in Github) that looks like:

[damienwebdev]

Assigning @DavidLambauer

[DavidLambauer]

To be clear, the expected outcome of this issue is to have a GitHub workflow running all the static tests (passing) and publish the build reports. correct?

[damienwebdev]

@DavidLambauer I'm not particularly interested in the build report being publicly accessible in a "pretty format". Whatever the job outputs in CI is enough for me.

[DavidLambauer]

I started working on this one. Doing some notes-taking here. Don't mind it, yet.

• I use the graycoreio/github-actions-magento2/supported-version@main to generate the matrix for the currently supported Magento 2 Versions. This action spits out a JSON with all the different PHP/Magento(And most likely more than that) combinations. Then, I can use that output in graycoreio/github-actions-magento2/setup-magento@main to setup a Magento with the parameters that I got from the step before.
• I wonder if the Matrix is cached somewhere globally, so I can memorize it instead of wasting resources on it?
• The above setup results in a couple of different jobs. All of them are named similar and a suffix is added. Something like this: setup-magento-extension (magento/project-community-edition:>=2.4.4 <2.4.5). The UI for this is horrible. I wonder if we can simplify the name generation somehow. Something like Magento 2.4.4 / PHP 8.1 would make more sense.

[DavidLambauer]

The env variables for graycoreio/github-actions-magento2/setup-magento@main are a little bit confusing. In particular:

• There is a mode extension, and there is extensions. extensions refers to the PHP Extensions, so the naming could be php_extensions to make it more clear. The extensions variable does not affect anything right now; it is not used further.
• The coverage variable expects something like xDebug and not a simple true/false. I had to read the description twice to get it. Either, the description or the naming should be improved here.
• Some vars are named in snake_case, and some in kebab-case.

[DavidLambauer]

I have something running as a first draft. As I mentioned, I use the setup-magento actions to get all the different Magento/PHP Combos. Then, I run composer install and run the tests with cd dev/tests/static && php ./../../../vendor/bin/phpunit . testsuite/Magento/Test/Legacy testsuite/Magento/Test/Integrity.

I am not entirely sure what Tests run in Magento. When I look at this I see several static test suites. Digging deeper into the test results from Static tests for CE I can see the Allure Report, but it only mentions Integrity tests.

My latest version: https://github.com/DavidLambauer/mageos-magento2/actions/runs/3414702008/jobs/5682998132

[DavidLambauer]

Summary of what was done so far:

• I set up a Workflow that spins up Magento Instances in several different Version Combinations. I use graycoreio/github-actions-magento2/supported-version@main.
• The Workflow is available here: https://github.com/DavidLambauer/mageos-magento2/actions
• The Matrix from @damienwebdev spins up 8 different Jobs
• Each Job spins up a clean Mage-OS
• Each Job runs the following tests:
  • PHPUnit in dev/tests/unit
  • Integration Integrity Tests from dev/tests/integration/testsuite/Magento/Test/Integrity
  • Legacy Tests in dev/tests/static/testsuite/Magento/Test/Legacy
  • Static Tests in dev/tests/static

I have a couple of open todos and questions:

• The Matrix compilation should be cached somehow
• I need some help with the Workflow structure. Right now, everything is just clustered into this single Workflow file. I am sure we need smaller components not to run EVERYTHING ALWAYS.
• All the tests fail right now. Yet, I don't rely on the phpunit.xml rulesets. I tried reverse engineering the bin/magento dev:tests:run ... command and ended up with an execution that ignores the rulesets.
• The test output from the Allure Report for Static CE Tests mentions 4018 Tests, whereas my current test suite only contains 3824 tests. Am I missing some?
• I don't like the current execution. Static tests should be categorized into execution tools. So instead of having one big Static Tests Block, I would go with one for phpmd, phpstan, Copy/Paste Detector, etc...
• The Allure Reports from Magento right now seem to be broken. They don't list all the test sites but only the integrity tests. This would be nice to have running over all the different test results.
• The Tests run forever... I can't believe that a single static test run takes more than 3h... Big Yikes...

[fballiano]

@Vinai @rhoerr could we consider this issue completed?