Skip to content

Bug on propagating arrow field metadata  #3964

Closed
@metesynnada

Description

@metesynnada

Describe the bug
While converting DFSchema to Schema, field metadata is not preserved. It may cause future bugs on methods like alias.

To Reproduce
Create a DFSchema that has fields with metadata.

let mut a: DFField = DFField::new(Some("table1"), "a", DataType::Int64, false);
let mut b: DFField = DFField::new(Some("table1"), "b", DataType::Int64, false);
let mut a_metadata = BTreeMap::new();
a_metadata.insert("key".to_string(), "value".to_string());
a.field.set_metadata(Some(a_metadata));
let mut b_metadata = BTreeMap::new();
b_metadata.insert("key".to_string(), "value".to_string());
b.field.set_metadata(Some(b_metadata));
let df_schema = Arc::new(
    DFSchema::new_with_metadata(
        [a, b].to_vec(),
        HashMap::new(),
    )
        .unwrap(),
);
let schema: Schema = df_schema.as_ref().clone().into();
let a_df = df_schema.fields.get(0).unwrap().field();
let a_arrow = schema.fields.get(0).unwrap();
assert_eq!(a_df.metadata(), a_arrow.metadata())

Expected behavior
Fields preserve the metadata.

Additional context
It will be solved by changing From<DFSchema> for Schema method.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions