Difference in decoding PIC 9(9)V99 field between spark-cobol_2.11 and spark-cobol_2.12 (NULL in 2.12)

[sandhyaalusuri]

Description

We are observing a difference in output when upgrading from Spark (Scala 2.11) to Spark (Scala 2.12) using Cobrix.
A field defined as PIC 9(9)V99 in the copybook is decoded correctly in spark-cobol_2.11, but the same field is returned as NULL in spark-cobol_2.12.

Spark 2.x (Scala 2.11): Cobrix spark-cobol_2.11-2.2.2
Spark 3.x (Scala 2.12): Cobrix tested spark-cobol_2.12-2.2.2, spark-cobol_2.12-2.6.3.

Copybook snippet
05 AMOUNT-FIELD PIC 9(9)V99.

Observed behavior
Version Output
spark-cobol_2.11 110323.11
spark-cobol_2.12 NULL

We would like to understand whether this is:

1. an intentional change in strictness/validation,
2. expected behavior in newer Cobrix versions,
3. or if additional options are required to maintain backward-compatible behavior.

[yruslan]

Hi, @sandhyaalusuri ,

Please, run the execution with .option("debug", "true") and send the HEX value of the raw data for both versions of Cobrix.

[sandhyaalusuri]

Hi @yruslan,

Thank you for your quick response.

We ran the execution with .option("debug", "true").
In spark-cobol_2.11-2.2.2, the field is resolved as 113023.51 and the corresponding HEX value of the raw data is 3131333032332E3531.
we will share spark-cobol_2.12-2.2.2 details shortly

[sandhyaalusuri]

We are using cp037 as the code page for the respective copybook.

[yruslan]

Great, thanks!
From the looks the the raw data:

1. This is an ASCII encoded number.
2. It has an explicit dot ('.') symbol.
3. For such a number the correct PIC having the same size is 9(8).99 (rather than 9(9)V99). The V character represents an implicit decimal point where the point itself is omitter. Just a guess from my side is maybe the copybook is created for an EBCDIC file, and you are trying to apply it to the already converted ASCII file. In EBCDIC formats implicit decimal points are more common. In ASCII formats explicit decimal points are more common.
4. cp037 code page is an EBCDIC format. The raw number should look like F1F1F3... rather than 313133...

[sandhyaalusuri]

Thank you for the clarification.
For Spark-3
in copy book , i have changed it type to string (PIC X(11)) and got results as 113023NA ,corresponding hexa value is F1F1F3F0F2F3D5C1404040.
similarly ,when datatype is 9(9) V99 ,got result as NULL ,corresponding hexa value is F1F1F3F0F2F3D5C1404040 .

[yruslan]

Thanks for the info.

It makes it much more clear. Since Cobrix 2.2.2 there were no changes breaking changes related to parsing of decimal numbers.

The decoder became more strict at the request of users in order to avoid accidentally parsing of invalid string tokens as numbers. The string 113023NA is not a valid number, hence you have null as the result.

It could be that your data is shifted for the copybook by 5 bytes, or the actual size of the number is 5 digits smaller, e.g. 9(4)V99. In any case, the 'N', 'A', and 3 spaces need to be removed in order to the number to be successfully parsed. The raw number should look like F1F1F3F0F2F3 for 1130.32, or F1F1F3F0F2F3F5F1 for 113032.51).

[sandhyaalusuri]

Thanks for update.

If we try to shift the position , we are observing the following error :
There are some files in /path/_C10 that are NOT DIVISIBLE by the RECORD SIZE calculated from the copybook (5970 bytes per record). Check the logs for the names of the files.

Kindly guide us on this issue, as it is a blocker for our Spark 3 upgrade.

[yruslan]

You can retain the record size by replacing non-numberic characters with a string field, or a FILLER:
For example, from:

05 AMOUNT-FIELD PIC 9(9)V99.

to:

05 AMOUNT-FIELD PIC 9(4)V99.
05 AMOUNT-EXTRA PIC X(5).

or

05 AMOUNT-FIELD PIC 9(4)V99.
05 FILLER       PIC X(5).

but it also could be that PIC 9(9)V99 is the correct one and the shift happened in one of previous fields.

[sandhyaalusuri]

Characters in the column are not same for all rows and the output should be similar to 2.11_2.2.2.
we really appreciate if you have any suggestions.

[yruslan]

The only suggestion in this case would be to convert this field to strings AMOUNT-FIELD PIC X(11)., and use Spark casting or UDFs to handle conversion to decimal numbers with corder cases.

[yruslan]

Or, you can create a second REDEFINE filed that would retain the original value so you would need only to fix null values in the first field, e.g:

05 AMOUNT-FIELD PIC 9(4)V99.
05 AMOUNT-STR   PIC X(11) REDEFINES AMOUNT-FIELD.

[yruslan]

I have another idea.
Try

.option("strict_sign_overpunching", "false")

with the original code - that should restore to the old behavior.

[sandhyaalusuri]

Thanks for your suggestions ,i have already tried the above options but no change in the results.

how will i determine its length ,in some cases its returns different values like
spark_cobol_2.11-2.2.2 --> 4.66
spark_cobol_2.12-2.2.2 --> NULL ,string MFF
hexa value D4C6C64040404040404040

[yruslan]

I see that

.option("strict_sign_overpunching", "false")

still not relaxed enough to allow multiple punched signs in the same number. But we can implement the relaxed behavior for this mode in order to keep maximum backwards compatiblity.