Improve template minifier #33016

blmage · 2021-05-14T15:45:07Z

Description (*)

This PR aims to improve the template minifier by :

fixing a variety of cases that are currently handled incorrectly,
commenting on each step of the minification process to explain what it does,
increasing the exhaustiveness of the corresponding test.

Related Pull Requests

Fixed Issues (if relevant)

Fixes HTML minification breaks PHP code #32153

Manual testing scenarios (*)

See HTML minification breaks PHP code #32153 or the updated test

Questions or comments

It is very likely that there remains cases that are handled incorrectly, mostly in JS code now, but it also is probably impossible to make the process entirely safe as long as the minification happens on the templates, given the mix of PHP and HTML / CSS / JS code.

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
All automated tests passed successfully (all builds are green)

m2-assistant · 2021-05-14T15:45:10Z

Hi @blmage. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

blmage · 2021-05-14T15:52:41Z

Still need to :

ensure that // comments are only removed inside <script> tags
update the tests

but this looks promising so far, if adding a dependency on nikic/php-parser is not a problem.

blmage · 2021-05-15T13:31:56Z

@magento run all tests

magento-automated-testing · 2021-05-15T13:32:04Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

mrtuvn · 2021-05-16T02:24:19Z

@magento run all tests

magento-automated-testing · 2021-05-16T02:24:25Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

mrtuvn · 2021-05-16T06:34:26Z

@magento run Functional Tests CE, Functional Tests B2B, Functional Tests EE, WebAPI Tests

magento-automated-testing · 2021-05-16T06:34:33Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

mrtuvn · 2021-05-16T23:46:44Z

Fail tests may not related with changes from this pull request

Den4ik · 2021-05-17T07:34:26Z

@magento run WebAPI Test, Functional Tests B2B

magento-automated-testing · 2021-05-17T07:34:30Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

Den4ik · 2021-05-19T11:38:40Z

@magento run WebAPI Test, Functional Tests B2B

magento-automated-testing · 2021-05-19T11:38:44Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

Den4ik · 2021-05-20T20:38:24Z

@magento run WebAPI Test

magento-automated-testing · 2021-05-20T20:38:28Z

Failed to run the builds. Please try to re-run them later.

Den4ik · 2021-05-20T20:38:42Z

@magento run WebAPI Test

The template minifier should not fix PHP code and hide errors from the user

For example, prevent "/\/path\/to\//" from being replaced with "/\/path\/to\"

- Do not attempt to parse empty contents - Prevent nesting problems with Xdebug - Catch and silence errors

blmage · 2021-10-16T13:01:59Z

@mrtuvn @Den4ik Done!

blmage · 2021-10-16T13:02:22Z

@magento run all tests

magento-automated-testing · 2021-10-16T13:02:30Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

Den4ik · 2021-10-16T14:02:53Z

Let's wait test results

blmage · 2021-10-18T08:15:35Z

@magento run Database Compare, Functional Tests CE, Functional Tests EE, Integration Tests, Sample Data Tests CE,
Static Tests, Unit Tests, WebAPI Tests

magento-automated-testing · 2021-10-18T08:15:43Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

Den4ik · 2021-10-20T11:09:23Z

@magento run all tests

magento-automated-testing · 2021-10-20T11:09:27Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

DanielRuf · 2021-11-12T21:17:15Z

I'm surprised that Magento wants to minify source code (PHP) instead of resulting html files.

Generally the current approach is not how it should work.

At least view_preprocessed files should ideally not be minified.

At least the current setup leads to many issues and the new proposed solution here still does some black magic / voodoo with regex.

The catch statement probably throws away too complex code, PHP 8 support might have to be added, the regex does not cover the optional trailing semicolon and so on (see #32153 (comment)).

Please see also my comments at #32153

Truely, (complex) regular expressions are a safe way to shoot yourself into the foot, with a double-barrel shotgun.

DanielRuf · 2021-11-12T21:28:18Z

Generally I would also propose to support some ignore-comment syntax (like eslint-ignore, ts-ignore, htmlmin:ignore by html-minifier and more) to tell the minifier, to not touch code blocks if needed and have some control, so we don't have to patch core files in the future, if some minified code leads to such issues (there are way more cases where newlines and other elements are crucial).

maaarghk · 2023-07-07T14:48:16Z

lib/internal/Magento/Framework/View/Template/Html/Minifier.php

+                $content = $prettyPrinter->prettyPrintFile($ast);
+                $heredocs = $prettyPrinter->getDelayedHeredocs();
+            } catch (\Error $error) {
+                // Some PHP code is seemingly invalid, or too complex.


In this case doesn't it make more sense to just log a warning and return the unminified code instead of guessing that you could do a better job than the PHP parser library at minifying some mangled script using regex?

m2-community-project bot added Priority: P3 May be fixed according to the position in the backlog. Severity: S2 Major restrictions or short-term circumventions are required until a fix is available. labels May 14, 2021

magento-engcom-team added Component: View Release Line: 2.4 labels May 14, 2021

m2-community-project bot added the Progress: pending review label May 14, 2021

blmage marked this pull request as ready for review May 15, 2021 13:31

mrtuvn self-assigned this May 16, 2021

m2-community-project bot added Progress: review and removed Progress: pending review labels May 16, 2021

mrtuvn added the improvement label May 16, 2021

mrtuvn added Auto-Tests: Covered All changes in Pull Request is covered by auto-tests Award: bug fix Award: category of expertise labels May 16, 2021

Den4ik self-assigned this May 17, 2021

m2-community-project bot added the Progress: needs update label Oct 12, 2021

blmage added 12 commits October 16, 2021 14:40

Fix PHP code in template minifier test

8d18437

The template minifier should not fix PHP code and hide errors from the user

Improve detection of heredocs in minified templates

7102e87

Clean up PHP code using nikic/php-parser before minifying template HTML

e3a6ef7

Remove redundant cleaning of PHP comments

e0e5791

Remove remaining single-line comments in <script> tags only

59dfff2

Improve detection of commented PHP tags

06f667b

Do not break JS RegExps that look like comments

5fbf10b

For example, prevent "/\/path\/to\//" from being replaced with "/\/path\/to\"

Handle edge cases when parsing templates

9062eb7

- Do not attempt to parse empty contents - Prevent nesting problems with Xdebug - Catch and silence errors

Preserve all spaces inside <textarea> and <pre> tags

1bc9e69

Comment on each step of the minification process

59be884

Update template minifier tests

1ac88c3

Remove unused variable

4e50eff

blmage force-pushed the template-minifier branch from efcb392 to 4e50eff Compare October 16, 2021 12:58

Merge branch '2.4-develop' into template-minifier

33d578c

hostep mentioned this pull request Nov 12, 2021

HTML minification breaks PHP code #32153

Open

5 tasks

mrtuvn mentioned this pull request Dec 28, 2021

Minifying templates breaks when two consecutive comments exist and first is // #34880

Closed

1 task

maaarghk reviewed Jul 7, 2023

View reviewed changes

m2-community-project bot added Progress: review and removed Progress: needs update labels Jul 7, 2023

Improve template minifier #33016

Are you sure you want to change the base?

Improve template minifier #33016

Uh oh!

Conversation

blmage commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Uh oh!

m2-assistant bot commented May 14, 2021

Uh oh!

blmage commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blmage commented May 15, 2021

Uh oh!

magento-automated-testing bot commented May 15, 2021

Uh oh!

mrtuvn commented May 16, 2021

Uh oh!

magento-automated-testing bot commented May 16, 2021

Uh oh!

mrtuvn commented May 16, 2021

Uh oh!

magento-automated-testing bot commented May 16, 2021

Uh oh!

mrtuvn commented May 16, 2021

Uh oh!

Den4ik commented May 17, 2021

Uh oh!

magento-automated-testing bot commented May 17, 2021

Uh oh!

Den4ik commented May 19, 2021

Uh oh!

magento-automated-testing bot commented May 19, 2021

Uh oh!

Den4ik commented May 20, 2021

Uh oh!

magento-automated-testing bot commented May 20, 2021

Uh oh!

Den4ik commented May 20, 2021

Uh oh!

blmage commented Oct 16, 2021

Uh oh!

blmage commented Oct 16, 2021

Uh oh!

magento-automated-testing bot commented Oct 16, 2021

Uh oh!

Den4ik commented Oct 16, 2021

Uh oh!

blmage commented Oct 18, 2021

Uh oh!

magento-automated-testing bot commented Oct 18, 2021

Uh oh!

Den4ik commented Oct 20, 2021

Uh oh!

magento-automated-testing bot commented Oct 20, 2021

Uh oh!

DanielRuf commented Nov 12, 2021

Uh oh!

DanielRuf commented Nov 12, 2021

Uh oh!

maaarghk Jul 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blmage commented May 14, 2021 •

edited

Loading

blmage commented May 14, 2021 •

edited

Loading

maaarghk Jul 7, 2023 •

edited

Loading