PHP Warning: Unterminated comment starting line 2 in tolerant-php-parser/src/PhpTokenizer.php on line 61

[felixfbecker]

I was working on running benchmarks for the language server in CI. One of the frameworks that is benchmarked is the parser itself. But it fails with a bunch of PHP warnings that seem to be emitted by token_get_all inside PhpTokenizer.php

PHP Warning:  Unterminated comment starting line 2 in /home/travis/build/[secure]/php-language-server/vendor/microsoft/tolerant-php-parser/src/PhpTokenizer.php on line 61
PHP Stack trace:
PHP   1. {main}() /home/travis/build/[secure]/php-language-server/Performance.php:0
PHP   2. LanguageServer\PhpDocument->__construct() /home/travis/build/[secure]/php-language-server/Performance.php:52
PHP   3. LanguageServer\PhpDocument->updateContent() /home/travis/build/[secure]/php-language-server/src/PhpDocument.php:105
PHP   4. LanguageServer\TreeAnalyzer->__construct() /home/travis/build/[secure]/php-language-server/src/PhpDocument.php:147
PHP   5. Microsoft\PhpParser\Parser->parseSourceFile() /home/travis/build/[secure]/php-language-server/src/TreeAnalyzer.php:55
PHP   6. Microsoft\PhpParser\TokenStreamProviderFactory::GetTokenStreamProvider() /home/travis/build/[secure]/php-language-server/vendor/microsoft/tolerant-php-parser/src/Parser.php:146
PHP   7. Microsoft\PhpParser\PhpTokenizer->__construct() /home/travis/build/[secure]/php-language-server/vendor/microsoft/tolerant-php-parser/src/TokenStreamProviderFactory.php:12
PHP   8. Microsoft\PhpParser\PhpTokenizer::getTokensArrayFromContent() /home/travis/build/[secure]/php-language-server/vendor/microsoft/tolerant-php-parser/src/PhpTokenizer.php:25
PHP   9. token_get_all() /home/travis/build/[secure]/php-language-server/vendor/microsoft/tolerant-php-parser/src/PhpTokenizer.php:61

See https://travis-ci.org/felixfbecker/php-language-server/jobs/294780146#L843

[roblourens]

I also see those when running the parser tests, and they are annoying but seem harmless. I think your tests are just failing because

PHP Fatal error:  Uncaught UnexpectedValueException: RecursiveDirectoryIterator::__construct(/home/travis/build/[secure]/php-language-server/validation/frameworks/CodeIgniter): failed to open dir: No such file or directory in /home/travis/build/[secure]/php-language-server/Performance.php:20
Stack trace:
#0 /home/travis/build/[secure]/php-language-server/Performance.php(20): RecursiveDirectoryIterator->__construct('/home/travis/bu...')
#1 {main}
  thrown in /home/travis/build/[secure]/php-language-server/Performance.php on line 20

[felixfbecker]

The language server has an errorhandler defined that turns warnings into exceptions, so I wouldn't say they are harmless. PHP warnings are a legacy concept, if the warning is not important, it should be swallowed, if the warning is of importance it should be reported as part of the ParserDiagnostic reporting mechanism

[roblourens]

These are just written to stderr by token_get_all. It looks like the @ operator suppresses it. Is that the right approach?

[felixfbecker]

Yeah, @ suppresses warnings and they can be inspected with error_get_last(). I wonder where the error comes from though? It is invalid syntax in the parser code? Should it be reported as a diagnostic?

[roblourens]

Ok, makes sense. In theory it would come from the parser parsing something with an unterminated comment. I'm not sure about your case, but when I see it in the parser tests, it's because we have tests over files with invalid syntax.

[felixfbecker]

So it should become a diagnostic, right?

[roblourens]

Yeah, any syntax error should already get a diagnostic from the DiagnosticProvider, but that's handled elsewhere.

[staabm]

Just a heads up: @ can make things slow/slower, and should be prevented when possible.

[felixfbecker]

@staabm I considered that but I don't see a better alternative