New diagnostic error on PIC X..

[delevoye]

LINKED WITH #1911, #1749 and #1899

Describe the bug
In DATA DIVISION, two consecutive dots at the end of a variable declaration should yield a parsing error.
IBM error: "X."was not a valid combination of"PICTURE"string characters. A"PICTURE"string of"S9(1)"was assumed.

Sample

       IDENTIFICATION DIVISION.
       PROGRAM-ID. Pgm.
       ENVIRONMENT DIVISION.
       CONFIGURATION SECTION.
       SPECIAL-NAMES. DECIMAL-POINT IS COMMA.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 A pic X.
      *Error: "X."was not a valid combination of"PICTURE"string
      *       characters. A"PICTURE"string of"S9(1)"was assumed.
       01 B pic X..
      *Error: "."was invalid. Scanning was resumed at the next area
      *       "A" item, level-number or the start of the next clause.
       01 C pic X. .
      
      *Error: "."was invalid. Scanning was resumed at the next area
      *       "A" item, level-number or the start of the next clause.
      *Warning: A blank was missing before character "." in column 16.
      *         A blank was assumed.
       01 var1..
          05 var2 pic XX.
      *Error: "VAR4"was invalid. Scanning was resumed at the next area
      *       "A"item, level-number, or the start of the next clause.
      *Error : A duplicate"PICTURE"clause was found in a data
      *        description entry. The duplicate clause was discarded.
      *Error : Character(s)"5"were found as"PICTURE"string character(s).
      *       A"PICTURE"string of"S9(1)"was assumed.
          05 var3 pic X.05 var4 pic X.
      
      * OK
       01 var1Bis.
      * OK
          05 var5 pic X. 05 var6 pic X.
      
       PROCEDURE DIVISION.
           move A to B
           .
       END PROGRAM Pgm.

Without decimal point is comma, we get the same erors:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. Pgm.
       ENVIRONMENT DIVISION.
       CONFIGURATION SECTION.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 A pic X.
      *Error: "X."was not a valid combination of"PICTURE"string
      *       characters. A"PICTURE"string of"S9(1)"was assumed.
       01 B pic X..
      *Error: "."was invalid. Scanning was resumed at the next area
      *       "A" item, level-number or the start of the next clause.
       01 C pic X. .
      
      *Error: "."was invalid. Scanning was resumed at the next area
      *       "A" item, level-number or the start of the next clause.
      *Warning: A blank was missing before character "." in column 16.
      *         A blank was assumed.
       01 var1..
          05 var2 pic XX.
      *Error: "VAR4"was invalid. Scanning was resumed at the next area
      *       "A"item, level-number, or the start of the next clause.
      *Error : A duplicate"PICTURE"clause was found in a data
      *        description entry. The duplicate clause was discarded.
      *Error : Character(s)"5"were found as"PICTURE"string character(s).
      *       A"PICTURE"string of"S9(1)"was assumed.
          05 var3 pic X.05 var4 pic X.
      
      * OK
       01 var1Bis.
      * OK
          05 var5 pic X. 05 var6 pic X.
      
       PROCEDURE DIVISION.
           move A to B
           .
       END PROGRAM Pgm.

EDIT : more complete example.

[smedilol]

@delevoye It's linked with #1749 and #1899
You also need to test with DECIMAL-POINT IS COMMA as it can changes how is interpreted the dot.

       IDENTIFICATION DIVISION.
       PROGRAM-ID. DVZZMFT0.
       ENVIRONMENT DIVISION.
       CONFIGURATION SECTION.
       SPECIAL-NAMES. DECIMAL-POINT IS COMMA.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
      *Ko compilation error
       01 var1 PIC X..
           
       END PROGRAM DVZZMFT0.

In fact you can focus only on DECIMAL-POINT IS COMMA tests as this is the standard for us.

I think the technical solution proposed in #1899 can still be implemented as a first step to reproduce IBM compiler behavior.
Then others special checks must be done when we are in data division.

Another interesting test that mix 3 issues together:

       IDENTIFICATION DIVISION.
       PROGRAM-ID.Pgm.
       ENVIRONMENT DIVISION.
       CONFIGURATION SECTION.
       SPECIAL-NAMES. DECIMAL-POINT IS COMMA.
       DATA DIVISION.
       WORKING-STORAGE SECTION.
       01 A pic X.
       01 B pic X..
       01 C pic X. .
      
       01 var1..
          05 var2 pic XX.
          05 var3 pic X.05 var4 pic X.
       PROCEDURE DIVISION..
           move A to B..
           move A to B...move A to B.
      
           .
       END PROGRAM Pgm.

[fm-117]

Checked with new picture validator, the errors are correctly created. Need to re-check after picture validator is fully integrated into develop.

[fm-117]

With the new PictureValidator (and some minor modifications), here are the diagnostics for the first two samples:

• Line 11[17,18] <27, Error, Syntax> - Syntax error : Combination of symbols 'X.' was not recognized as a valid PICTURE string
• Line 14[20,20] <27, Error, Syntax> - Syntax error : extraneous input '.' expecting {ProgramIdentification, ProgramEnd, ClassIdentification, ClassEnd, FactoryEnd, ObjectIdentification, ObjectEnd, MethodEnd, ProcedureDivisionHeader, WorkingStorageSectionHeader, LocalStorageSectionHeader, LinkageSectionHeader, FileDescriptionEntry, DataDescriptionEntry, DataRedefinesEntry, DataRenamesEntry, DataConditionEntry, ExecStatement, FunctionDeclarationEnd, GlobalStorageSectionHeader}
• Line 20[15,15] <12, Warning, Tokens> - A blank was missing before character "." in column 16. A blank was assumed.
• Line 28[23,26] <27, Error, Syntax> - Syntax error : Invalid PICTURE string 'X.05': character '5' at position '4' was not expected
• Line 28[28,31] <27, Error, Syntax> - Syntax error : extraneous input 'var4' expecting {separator, statement starting keyword, keyword}

This is correct. However no error is reported on line 20, only a Warning. This is related to how CUP parser recovers from error:

• on line 14, the CUP parser could not handle the extraneous '.' and stopped building the Node tree so no error is reported from node phase after that point. This is confirmed when commenting the line 14.

Last sample will be included as a new unit test.