Debug column is not getting generated for redefined field

Describe the bug

Debug column is not created for the redefined column without having any sub level when debug option is enabled.

To Reproduce

For copybook as below

10 field1 PIC X.
10 field2 redefines field1 PIC X.

The debug column is not generated for the column field2

[yruslan]

Hi, the debug column for field2 should be the same as for field1 since they redefine each other.

What's your exact use case? Are field1 and field2 have different sizes?

for one copybook its of same size. I am assuming if there will be one more indicator column then based on that indicator field1 and fild2 may contain 2 different values though their size is same.

Let my copybook is as below.

01 TRANSDATA.
10 CURRENCY PIC X(3).
10 INDICATOR PIC X(1).
10 field1 PIC X.
10 field2 redefines field1 PIC X.

and my data is as below in ASCII

123PA
123CB

As field2 redefines field1 then based on the INDICATOR column user needs to decide whether to pick field1 or field2.

val df = spark
.read
.format("cobol")
.option("is_text", "true")
.option("copybook", "copybookpath")
.option("encoding", "ascii")
.option("truncate_comments","false")
.option("debug","true")
.option("schema_retention_policy", "collapse_root")
.load("datapath")
df.show();

In the above case field2_debug is not generated. I do understand that based on the INDICATOR column i can only refer field1
but as far as attribute is concerned if i want to use field2 attribute_debug column i cannt do that.

in case the copybook will be as follows

01 TRANSDATA.
10 CURRENCY PIC X(3).
10 INDICATOR PIC X(1).
10 field1 PIC X(2).
10 field2 redefines field1 PIC X.

and the data is

123PAA
123CBB
123CfQ

the output will be

In the above case i need definitely the field2_debug column as the field 1 and field2 value wont be same.

So in any of the case if i would generate redefined debug column then the process can be streamlined and redefined debug column can be handled.

Raised a PR #331 for removing the condition that restricts the redefined debug field generation in parser

closing this issue as PR is merged