Skip to content

3. Add custom numeric transformation

@dawrutowicz edited this page Jun 8, 2019 · 1 revision

There are types that are not shipped with the library since they need configuration made by the user.

Such example is a BigDecimal type - it needs precision and scale.

In such case, you need to define an implicit SparkDataType. It is just a wrapper for Spark data type that can be used for automatic resolution.

Add definition for BigDecimal:

import cleanframes.instances.SparkDataType

implicit val decimalCleaner = new SparkDataType[BigDecimal] {
  override def getDataType: DataType = DecimalType(4, 2)
}

Rest usage of the library remains the same.

Full running code example can be found here.

Clone this wiki locally