Copybook.parseSimple not dropping the fillers in output ast

String copybookContents = "01 RECORD.
05 FILLER PIC X(1).
05 COMPANY_PREFIX PIC X(3).
05 FILLER PIC X(1).
05 FILLER PIC X(1).
05 COMPANY_NAME PIC X(9)."

Group grp = CopybookParser.parseSimple(copybookContents, true,true,commentPolicy.apply(false,6,72)).ast();

in the above scenario as drop_value_fillers is true then the output ast also should not contain the FILLERS. But output ast is providing each column details.

[yruslan]

Thanks for the bug report. Dropping of fillers is a feature of 'spark-cobol'. Fillers should be deopped in Spark schema while still can be present in the AST.
But your expectation makes sense. We can drop fillers from AST as well.

[yruslan]

Hi, I was revisiting this issue.
Each AST element has isFiller flag, so you can always ignore all fields if the flag is true.

Is it necessary for your use case to actually remove fillers from the AST if you have this flag?

That's how we are handling in our code while parsing the ast. It's a good to have feature to drive via parsesimple

[yruslan]

Could you please elaborate on what are you trying to achieve?

Every use case that I can think of can be done by just ignoring fields (AST statements) for which isFiller == true.

We are currently parsing the copybook using ast and converting to json. We are using parsesimple to get the ast. if based on the user options if we can drop or retain the filler in ast then for us its not required to handle while converting back to json.

[yruslan]

Sure, but using node.isFiller can be used to achieve the same, isn't it?

cobrix/cobol-parser/src/main/scala/za/co/absa/cobrix/cobol/parser/ast/Statement.scala

Line 81 in 3014923

def isFiller: Boolean

What's the code snippet that you are using?

[yruslan]

I see why you are asking for the feature. There are parameters that say 'dropGroupFiller' and 'dropValueFillers' but nothing actually dropped, just marked. It makes sense. We can implement the dropping as well

You are correct. Currently we are parsing and dropping the fillers by utilizing the isFiller. currently the parameter we are sending for dropping filler to parseSimple is not functional.

[yruslan]

This is implemented.

Please, check the latest master. Note that the signature of the method (parseSimple) has changed. It now reflects what is actually being done.

Spark schemas doe not support having 2 column names having tha same name so FILLERs need to be renamed in order to be retained. So the method allow controlling if fillers are going to be renamed, and a separate flag that controls if non-renamed fields should be dropped.