Backfill the T_YIELD token for PHP < 5.5

[jrfnl]

Looking at PR #1458, #1491, #1492 the yield keyword is not backfilled to tokenize as T_YIELD for old PHP versions.

[ktomk]

@jrfnl What about "T_YIELD_FROM" ?

[michalbundyra]

@ktomk I can't see T_YIELD_FROM token on PHP token list
http://php.net/manual/en/tokens.php

Can you show any code example when PHP uses this token?

[ktomk]

@webimpress: I just wrote a PHPCS sniff that dealt with it yesterday at work. Probably that is PHPCS specific and not necessary for your change?

/edit: Regarding the code-example you asked for: https://3v4l.org/NhegF

<?php

var_dump(T_YIELD_FROM);

Output for 7.0.0 - 7.2.0alpha3

int(269)

Output for hhvm-3.15.4

int(439)

Output for 5.6.0 - 5.6.30

Notice: Use of undefined constant T_YIELD_FROM - assumed 'T_YIELD_FROM' in /in/NhegF on line 3

string(12) "T_YIELD_FROM"

[michalbundyra]

@ktomk Please see my updated PR #1524

[ktomk]

Wow, that was quick 👍 If you still have some time I think it's worth to cover both tokens with a test, if I understand the change correctly it must be after the change that tokenization should yield the same tokens regardless of PHP version. I don't know but there should be existing tests for tokenization to create from, just let me know if you see any issues or have other feedback.

[michalbundyra]

I'm just starting my holidays and I can't have a look on it for next week. If you like add tests please do that in separate PR based on my changes. I've tested it locally (with xdebug) and all seems to be fine. I haven't seen any "tokenizer unit tests", so I'm not sure how it should be done.

…

On 15 Jul 2017, 15:59 +0200, Tom Klingenberg ***@***.***>, wrote: Wow, that was quick 👍 If you still have some time I think it's worth to cover both tokens with a test, if I understand the change correctly it must be after the change that tokenization should yield the same tokens regardless of PHP version. I don't know but there should be existing tests for tokenization to create from, just let me know if you see any issues or have other feedback. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ktomk]

@webimpress Well then I'd say have a nice time! I will take a look what I can do and will naturally check that against your changes! I might be able to add some more general tests, too. Happy holidays.

[jrfnl]

@ktomk Useful addition, thanks for suggesting it.
@webimpress Thanks for your quick turn around adding it to the PR!

For future reference: yield from was added in PHP 7.0 AFAICS:

• http://php.net/manual/en/language.generators.syntax.php#control-structures.yield.from
• https://wiki.php.net/rfc/generator-delegation

[michalbundyra]

@ktomk Maybe we don't need any extra tests now, because as noted by @jrfnl and as I updated PR #1492 the token is used there. When #1524 will be merged then tests in #1492 should pass.

[ktomk]

That's great to read, I was not aware how the changes inline with multiple reports here and that there are tests already.