Skip to content

Add support for parsing copybooks given Spark options #672

@yruslan

Description

@yruslan

Background

Sometime we want to use RDDs and Spark schemas separately for processing input files. In this case it is important to generate Spark schema that matches the record schema exactly. But the parser accepts its own set of options, and Spark reader for the 'cobol' format accepts options via '.option()'. It would be useful for the copybook parser to also be able to parse via options get from a Map[String. String], with the same semantics as the Spark cobol format reader.

Feature

Add support for parsing copybooks given Spark options.

Example

val sparkOptions = Map("generate_record_id" -> "true")
val cobolSchema = CobolSchema.fromSparkOptions(sparkOptions)
val sparkSchema = cobolSchema.getSparkSchema

Proposed Solution

As per example

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions