Tokenizer: derefencing incorrectly identified as short array

[jrfnl]

The array dereferencing square brackets in the following code are incorrectly changed to short array tokens.

// Array and string literal dereferencing.
echo [1, 2, 3][0];
echo 'PHP'[0];

Debug output:

        *** START LEVEL MAP ***
        Process token 0 on line 1 [col:1;len:5;lvl:0;]: T_OPEN_TAG => <?php\n
        Process token 1 on line 2 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n
        Process token 2 on line 3 [col:1;len:42;lvl:0;]: T_COMMENT => // Array and string literal de
referencing.\n
        Process token 3 on line 4 [col:1;len:4;lvl:0;]: T_ECHO => echo
        Process token 4 on line 4 [col:5;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 5 on line 4 [col:6;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 6 on line 4 [col:7;len:1;lvl:0;]: T_LNUMBER => 1
        Process token 7 on line 4 [col:8;len:1;lvl:0;]: T_COMMA => ,
        Process token 8 on line 4 [col:9;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 9 on line 4 [col:10;len:1;lvl:0;]: T_LNUMBER => 2
        Process token 10 on line 4 [col:11;len:1;lvl:0;]: T_COMMA => ,
        Process token 11 on line 4 [col:12;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 12 on line 4 [col:13;len:1;lvl:0;]: T_LNUMBER => 3
        Process token 13 on line 4 [col:14;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 14 on line 4 [col:15;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 15 on line 4 [col:16;len:1;lvl:0;]: T_LNUMBER => 0
        Process token 16 on line 4 [col:17;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 17 on line 4 [col:18;len:1;lvl:0;]: T_SEMICOLON => ;
        Process token 18 on line 4 [col:19;len:0;lvl:0;]: T_WHITESPACE => \n
        Process token 19 on line 5 [col:1;len:4;lvl:0;]: T_ECHO => echo
        Process token 20 on line 5 [col:5;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 21 on line 5 [col:6;len:5;lvl:0;]: T_CONSTANT_ENCAPSED_STRING => 'PHP'
        Process token 22 on line 5 [col:11;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 23 on line 5 [col:12;len:1;lvl:0;]: T_LNUMBER => 0
        Process token 24 on line 5 [col:13;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 25 on line 5 [col:14;len:1;lvl:0;]: T_SEMICOLON => ;
        *** END LEVEL MAP ***
        *** START ADDITIONAL PHP PROCESSING ***
        * token 22 on line 5 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 24 on line 5 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        * token 14 on line 4 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 16 on line 4 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        * token 5 on line 4 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 13 on line 4 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        *** END ADDITIONAL PHP PROCESSING ***

[gsherwood]

I've committed a fix for this. The 7.1 tests are only failing because of the PHPUnit version.

Thanks for reporting this.

[jrfnl]

@gsherwood Thanks for the quick fix!

[jrfnl]

Uh oh... looks like this fix broke the tokenizer for normal array referencing....

The first set of square brackets on each line in the below example code are now tokenized as short array openers/closers.

echo $this->addedCustomFunctions['nonce'];
echo $this->deprecated_functions[ $function_name ]['version'];

Debug output:

        Process token 0 on line 1 [col:1;len:5;lvl:0;]: T_OPEN_TAG => <?php\n
        Process token 1 on line 2 [col:1;len:0;lvl:0;]: T_WHITESPACE => \n
        Process token 2 on line 3 [col:1;len:4;lvl:0;]: T_ECHO => echo
        Process token 3 on line 3 [col:5;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 4 on line 3 [col:6;len:5;lvl:0;]: T_VARIABLE => $this
        Process token 5 on line 3 [col:11;len:2;lvl:0;]: T_OBJECT_OPERATOR => ->
        Process token 6 on line 3 [col:13;len:20;lvl:0;]: T_STRING => addedCustomFunctions
        Process token 7 on line 3 [col:33;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 8 on line 3 [col:34;len:7;lvl:0;]: T_CONSTANT_ENCAPSED_STRING => 'nonce'
        Process token 9 on line 3 [col:41;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 10 on line 3 [col:42;len:1;lvl:0;]: T_SEMICOLON => ;
        Process token 11 on line 3 [col:43;len:0;lvl:0;]: T_WHITESPACE => \n
        Process token 12 on line 4 [col:1;len:4;lvl:0;]: T_ECHO => echo
        Process token 13 on line 4 [col:5;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 14 on line 4 [col:6;len:5;lvl:0;]: T_VARIABLE => $this
        Process token 15 on line 4 [col:11;len:2;lvl:0;]: T_OBJECT_OPERATOR => ->
        Process token 16 on line 4 [col:13;len:20;lvl:0;]: T_STRING => deprecated_functions
        Process token 17 on line 4 [col:33;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 18 on line 4 [col:34;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 19 on line 4 [col:35;len:14;lvl:0;]: T_VARIABLE => $function_name
        Process token 20 on line 4 [col:49;len:1;lvl:0;]: T_WHITESPACE =>
        Process token 21 on line 4 [col:50;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 22 on line 4 [col:51;len:1;lvl:0;]: T_OPEN_SQUARE_BRACKET => [
        Process token 23 on line 4 [col:52;len:9;lvl:0;]: T_CONSTANT_ENCAPSED_STRING => 'version'
        Process token 24 on line 4 [col:61;len:1;lvl:0;]: T_CLOSE_SQUARE_BRACKET => ]
        Process token 25 on line 4 [col:62;len:1;lvl:0;]: T_SEMICOLON => ;
        *** END LEVEL MAP ***
        *** START ADDITIONAL PHP PROCESSING ***
        * token 17 on line 4 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 21 on line 4 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        * token 7 on line 3 changed from T_OPEN_SQUARE_BRACKET to T_OPEN_SHORT_ARRAY
        * token 9 on line 3 changed from T_CLOSE_SQUARE_BRACKET to T_CLOSE_SHORT_ARRAY
        *** END ADDITIONAL PHP PROCESSING ***

[gsherwood]

Thanks. I couldn't figure out why a bit of code was in the tokenizer, and I had no tests failing after removing it, but this is exactly the case it was checking for. I just couldn't see the syntax in my head - bit of a brain explosion.

Anyway, I've put some tests in to catch the problem now, and fixed it again. Please let me know if you see any more array issues from this.

[jrfnl]

@gsherwood Thanks ever so much for the quick fix. Will let you know if I find any other breakages.