Tokenizer/PHP: namespaced names as single token, mirroring PHP 8.0+

[jrfnl]

Description

This is a re-do of PR squizlabs/PHP_CodeSniffer#3155 with lots of updates.

In all fairness, this should be one commit for it to be atomic, but I believe that would make reviewing this PR next to impossible, which is why I've split the PR (like before) into lots of smaller, reviewable commits.

---

Util/Tokens: introduce new $nameTokens array

As both tests as well as sniffs will need to refer to these tokens quite a lot, providing a predefined token array to make this more straight-forward both within PHPCS itself, as well as for external standards seems appropriate.

PHP 8.0 | Tokenizer/PHP: namespaced names as single token

As per the proposal in squizlabs/PHP_CodeSniffer#3041.

This effectively backfills the new PHP 8.0 tokenization of identifier names for all supported PHP versions in PHPCS 4.x.

Note: this backfills takes into account that reserved keywords are now allowed to be used in namespaced names.
For that reason no check is done of the token type of the token between namespace separators, only on the contents.

Includes extensive unit tests to ensure the correct re-tokenization as well as that the rest of the tokenization is not adversely affected by this change.

Includes making sure that the following (re-)tokenizations are still stable:

• Nullable vs ternary.

Tokenizer/PHP: update union/intersection/DNF type token tokenization …

…to allow for PHP 8 identifier tokens

Includes updated test expectations for the typed constants tests.

Tokenizer/PHP: update the arrow function backfill to allow for PHP 8 identifier tokens

Includes updating the unit tests for the new reality and adding some additional tests.

Note: while the tests which were testing the handling of the fn keyword when used in namespaced names are now no longer testing the polyfill for arrow functions, I do believe the test in itself still has value, which is why I've kept it and adjusted the testNotAnArrowFunction() method to allow for expecting one of the name tokens.

Tokenizer/PHP: update the short array tokenization to allow for PHP 8 identifier tokens

Includes adding some additional tests.

Tokenizer/PHP: update various tests to allow for PHP 8 identifier tokens

Note: while some tests which were testing the handling of the specific keywords when used in namespaced names are now no longer testing the polyfill for that keyword, I do believe those tests in itself still have value, which is why I've kept them and adjusted the testNot[KEYWORD]() methods to allow for expecting one of the name tokens.

Includes removing one test which really didn't have value anymore.

File::getMethodParameters(): fix method to work with the PHP 8 identifier tokens

The existing unit tests already contain tests covering this change.

Includes updating token offset expectations in pre-existing tests for the change in tokenization.

Note: the T_NAMESPACE and T_NS_SEPARATOR cases should, by rights, be removed what with the new tokenization. However, leaving them in place provides some tolerance for types interlaced with comments/whitespace as allowed in PHP < 8.0.
As this is an often used utility method, I believe the method containing such tolerance to ensure the same results PHP cross-version is warranted.

File::getMethodProperties(): fix method to work with the PHP 8 identifier tokens

The existing unit tests already contain tests covering this change.

Includes updating token offset expectations in pre-existing tests for the change in tokenization.

File::getMemberProperties(): fix method to work with the PHP 8 identifier tokens

The existing unit tests already contain tests covering this change.

Includes updating token offset expectations in pre-existing tests for the change in tokenization.

File::isReference(): fix method to work with the PHP 8 identifier tokens

The existing unit tests already contain tests covering this change.

File::findExtendedClassName(): fix method to work with the PHP 8 identifier tokens

Includes adding an additional test for a PHP 8 identifier name type previously not covered by tests.

File::findImplementedInterfaceNames(): fix method to work with the PHP 8 identifier tokens

Includes adding an additional test for a PHP 8 identifier name type previously not covered by tests.

File::find[Start|End]OfStatement(): update tests to allow for PHP 8 identifier tokens

File::getTokenAsString(): update tests to allow for PHP 8 identifier tokens

Tokens::$functionNameTokens: add the new PHP 8 identifier name tokens

Includes adding/updating tests for various sniffs using the Tokens::$functionNameTokens token array to make sure the behaviour is unchanged (without adding the new tokens to the token array, the behaviour would have been changed).

.... and then a lot of other commits updating individual sniffs to allow for the new tokenization....

Suggested changelog entry

Added:

• Tokens::NAME_TOKENS containing an array with the tokens used for identifier names.

Changed:

• PHPCS now uses the PHP >= 8.0 native method for tokenizing (namespaced) identifier names.
  • Before PHP 8.0, PHP would tokenize namespaced names using T_STRING and T_NS_SEPARATOR.
  • From PHP 8.0, PHP uses the tokens T_NAME_FULLY_QUALIFIED, T_NAME_RELATIVE, and T_NAME_QUALIFIED instead.
  • PHPCS now uses these new PHP 8.0 tokens no matter what version of PHP is being used to run PHPCS.
  • Custom sniffs that use T_STRING and T_NS_SEPARATOR tokens to look for namespaced names will need to be modified.
• Tokens::$functionNameTokens now includes the identifier name tokens.

Fixed:

• Fixed bug : File::findExtendedClassName() will no longer break on namespace relative class names.
• Fixed bug : File::findImplementedInterfaceNames() will no longer break on namespace relative interface names.
• Fixed bug : Various sniffs now have better support for ignoring/examining qualified function calls.

Related issues/external references

Fixes squizlabs/PHP_CodeSniffer#3041
Fixes #133