-
Notifications
You must be signed in to change notification settings - Fork 82
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Background
Sometime we want to use RDDs and Spark schemas separately for processing input files. In this case it is important to generate Spark schema that matches the record schema exactly. But the parser accepts its own set of options, and Spark reader for the 'cobol' format accepts options via '.option()'. It would be useful for the copybook parser to also be able to parse via options get from a Map[String. String], with the same semantics as the Spark cobol format reader.
Feature
Add support for parsing copybooks given Spark options.
Example
val sparkOptions = Map("generate_record_id" -> "true")
val cobolSchema = CobolSchema.fromSparkOptions(sparkOptions)
val sparkSchema = cobolSchema.getSparkSchema
Proposed Solution
As per example
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request