variable length record optimization

[sree018]

Background [Optional]

we have 76 GB variable width variable length (BDW+RDW) with multi-segmented file(47 segments) , file contains 470,000,000 records with 700 columns. I am trying to convert parquet file. It's creating single index(1)( single partition). For parsing file and able to see data correctly with df.show()

Question

How to do I parallelize job across executors?

df.write taking single thread while write into parquet file.

options used

1. inputsplit records
2. input split size

[yruslan]

Hi, by default, Cobrix should be able to parallelize its work, so we'd need some additional context to understand why it is not happening in your case. Please, add the following to the issue description:

• The version of Cobrix used
• The full code snippet that you used to read the file
• The command line you used to run the application

[sree018]

@yruslan thanks for response

The version of Cobrix used: 2.5.1, scala_211,spark_2.4.0

val cobrixOptions:Map[String,String]=Map("record_format"->"VB",
"bdw_adjustment"->"-4,"
"rdw_adjustment"->"-4,"
"is_rdw_big_endian"-> "true",
"is_bdw_big_endian"->"true",
"input_split_size_mb"->"30")

val df=spark.read.format("za.co.absa.cobrix.spark.cobol.source").option("copybook",copyBookPath).options(cobrixOptions).load(path)

val df1=SparkUtils.flattenSchema(df,true)

df1.write.mode("overwrite").parquet(outboundpath)

[yruslan]

Interesting... the index consists of a single item only. This usually happens when the input file is not decoded correctly.

How many records 'df2.count' returns?

[sree018]

450,786,280

[yruslan]

Looks good, thanks! Investigating

[yruslan]

The issue is confirmed. We will fix it in the next release.