`List<T>.toDataFrame()` sorts lexicographically when primary constructor has `@ColumnName` annotations

[hantsy]

When upgrading to v0.15, the exporting csv is broken.

We defined the DataSchema as the following:

@DataSchema
data class CsvExampleDataModel(
    @ColumnName("Amount")
    val amount: BigDecimal? = null,
    @ColumnName("Last 4")
    val last4: String? = null,
    @ColumnName("ACH Type")
    val type: ACHType? = null
)

In the exported CSV, the column names are ordered by letters, not using the nature order in the CsvExampleDataModel.

Give the following example codes.

  val exportedCsv = listOf(
      CsvExampleDataModel(BigDecimal("10.2"), "1234", ACHType.SAME_DAY),
      CsvExampleDataModel(BigDecimal("11.20"), "4567", ACHType.NEXT_DAY)
  ).toDataFrame().toCsv()
  println("expected csv: $exportedCsv")

And print result in console like this:

expected csv: ACH Type,Amount,Last 4
SAME_DAY,10.2,1234
NEXT_DAY,11.20,4567

Check the examples: https://github.com/hantsy/dataframe-sandbox/blob/master/src/main/kotlin/com/example/CsvExample.kt#L37-L41

[Jolanrensen]

Thanks! Good find :)

This behavior can also be seen in DataFrame 0.14, it's not new. 0.13.1 still has the old behavior as I can see.

It has nothing to do with CSV though, the behavior can already be observed when just converting a list of data classes to DataFrame.

It seems to have been caused by this change where I reworked how properties in classes (and getter functions in Java, since we cannot get their order by reflection) should be ordered when converted to a DataFrame. It seems I check for the order of the columnName inside the primary constructor, but the column name gets changed by adding the @ColumnName annotation, so it can no longer be found in the constructor.

Should be an easy fix :)

[hantsy]

Yeah, we still use v0.13.1 in our project now. And we need to use the raw column order defined in the schema model.

[Jolanrensen]

@hantsy should you need a temporary solution, you can always move columns to the right index. Or you can completely reorder your columns by any metric you decide. For instance, this piece of code reorders all columns by the order of the properties in CsvExampleDataModel:

val constructorOrder: List<String> =
    (CsvExampleDataModel::class as KClass<*>)
        .declaredMemberProperties
        .map { it.findAnnotation<ColumnName>()?.name ?: it.name }

df.reorderColumnsBy {
    constructorOrder.indexOf(it.name)
}

[alwyn]

In my case I want the columns to be in the same order as they are defined in the constructor. The above temporary solution didn't work for that since declaredMemberProperties is an unordered map. Parameters or the constructor doesn't seem to have annotation information attached, so one has to process in order of the parameters and then find the property that match the parameter in the member properties collection.

In my case I had to do the following:

val constructorOrder: List<String> =
         (CsvExampleDataModel::class as KClass<*>)
             .primaryConstructor!!
             .parameters.map { param ->
                 val property = CsvExampleDataModel::class.declaredMemberProperties.first { it.name == param.name }
                 property.findAnnotation<ColumnName>()?.name ?: property.name
             }
df.reorderColumnsBy {
     constructorOrder.indexOf(it.name)
}

[Jolanrensen]

In my case I want the columns to be in the same order as they are defined in the constructor. The above temporary solution didn't work for that since declaredMemberProperties is an unordered map. Parameters or the constructor doesn't seem to have annotation information attached, so one has to process in order of the parameters and then find the property that match the parameter in the member properties collection.

In my case I had to do the following:

val constructorOrder: List<String> =
         (CsvExampleDataModel::class as KClass<*>)
             .primaryConstructor!!
             .parameters.map { param ->
                 val property = CsvExampleDataModel::class.declaredMemberProperties.first { it.name == param.name }
                 property.findAnnotation<ColumnName>()?.name ?: property.name
             }
df.reorderColumnsBy {
     constructorOrder.indexOf(it.name)
}

ah yes, you're right. I forgot about that. We use a similar solution to this in DataFrame: retrieve properties from memberProperties and sort them by primaryConstructor order.