Unable to get the correct numeric data parsed through Cobrix.

[cshah79]

Background [Optional]

Hi Ruslan,
We are trying to replace a legacy process which is ingesting binary data (VSAM RRD) we receive from our source and processing it through Informatica currently, by doing the same processing using Databricks and Cobrix.
I have the attached the RRD file describing the Record length and the fields which we got from our Source application. This along with the current file config in Informatica is how we have created the attached copybook.
I have also attached a sample record comparison of the same file processed through Informatica (highlighted in yellow) vs Cobrix.

output_comparison_infa_vs_corbix.xlsx
dlr_chain_sub_copybook.txt
source_raw_data.txt
DLR_CHAIN_SUB_DLY1_RRD.txt

Question

The main issue we are having is that any numeric data in the file is not getting interpreted correctly when we process this file though Cobrix. Can you please let me know where we might be going wrong with this?

[yruslan]

Thanks for the detailed question with the actual copybook and data! It helped to find the issue quickly.
COMP-5 numbers are expected to be big endian (DLY_TBL_SEQUENCE_NUM = 48 = 0x00 0x00 0x00 0x30), while the actual data contained little-endian data (DLY_TBL_SEQUENCE_NUM = 48 = 0x30 0x00 0x00 0x00).

In big-endian encoding 0x30 0x00 0x00 0x00 = 805306368, that's why all numerical data decoded incorrectly.

Luckily, there is a workaround. Please, use COMP-9 instead of COMP-5 in such copybooks. Let me know if it works.

[cshah79]

Thanks for the detailed question with the actual copybook and data! It helped to find the issue quickly. COMP-5 numbers are expected to be big endian (DLY_TBL_SEQUENCE_NUM = 48 = 0x00 0x00 0x00 0x30), while the actual data contained little-endian data (DLY_TBL_SEQUENCE_NUM = 48 = 0x30 0x00 0x00 0x00).

In big-endian encoding 0x30 0x00 0x00 0x00 = 805306368, that's why all numerical data decoded incorrectly.

Luckily, there is a workaround. Please, use COMP-9 instead of COMP-5 in such copybooks. Let me know if it works.

Thanks, I looked at the hex values available through debug option and makes sense what you just observed data coming in as little-endian format.

I updated copybook to use COMP-9, but it errored out .

Py4JJavaError: An error occurred while calling o484.load.
: za.co.absa.cobrix.cobol.parser.exceptions.SyntaxErrorException: Syntax error in the copybook at line 3: Invalid input 'COMP-9' at position 3:54

Cobrix version: za.co.absa.cobrix:spark-cobol_2.12:2.4.3
Apache Spark 3.1.1
Copybook attached.
dlr_chain_sub_copybook3.txt

[yruslan]

This is a very good catch! 👍 It seems it is a regression parser bug. We will fix it shortly.

[yruslan]

Should be fixed in 2.4.6. Please, check and let me know if it worked for you.