Example writing a complex structure to parquet.

[tghunt]

Hello all,

Does anyone know of an example writing complex records to parquet files using the parquet crate; Where the record would be something like "field1, field 2, field3, Vec<struct<string, Vec<struct<string, string>>>>". The record example reflects how data would exists in BigQuery after ingestion.

[tustvold]

Have you considered using the arrow APIs? These will handle the record shredding for you and so should be easier to get started with

[tustvold]

Technically yes, practically you would need to perform the record shredding yourself, which is highly non-trivial. Is the additional nesting such a deal breaker for you? It is fairly standard practice AFAICT...

[tghunt]

Actually it is a deal breaker. I know everything in Rust is still newish but I may have to revert back to Java (dislike its state of affairs) in order to build out the parquet files as needed. As I have shown in the various images, the parquet files are ending up with arbitrary levels which in BigQuery means there are extra steps required to unnest data. What should be a simple one level unnest to get at data becomes a multi-level unnest. While it is not quite an issue from a software perspective, it is from a data science interactive perspective since the clean usability of the data is obscured with the arbitrary levels.

[tustvold]

My knowledge of BigQuery is limited, but as there is still the same number of repeated fields, I wouldn't have expected the arrow schema to require additional calls to unnest, which is only required for arrays, not non-repeated fields?

I know everything in Rust is still newish

FWIW this isn't a Rust issue per-se, we are simply following the parquet standard in this regard 😅

[tghunt]

With the avro crate I can generate the schema and layout out the data in the following manner:

The unnesting is only required for the level of data/details you are after and the nested levels can be named appropriately.

I was hoping to be able to write out a parquet file that would mimic this. If it was not for the uncontrollable arbitrary items appearing, what we have explored would work. In our examples I had many_name_value_pairs which would be tests in the image above. With the Mapbuilder I would be able to rename entry to a test name, but if I have to store maps within a listbuilder I would end up with the extra list and items.

/*
toml adds:
apache-avro = "0.15.0"
serde = "1.0.163"
serde_json = "1.0.96"
*/
use std::fs::{File};
use apache_avro::{Schema, Writer};
use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct KeyValue {
    pub keyword: String,
    pub value: String
}

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Tests {
    test_name: String,
    test_details: Vec<KeyValue>
}

#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Record {
    pub time: i64,
    pub id1: String,
    pub id2: String,
    pub tests: Vec<Tests>
}

pub fn main() {

    let mut record = Record{
        time: 123,
        id1: "id_1".to_string(),
        id2: "id_2".to_string(),
        tests: vec![],
    };

    for i in 0..5 {
        let test = format!("test_{}", i);

        let mut results = Vec::<KeyValue>::new();
        for i2 in 0..4 {
            results.push(KeyValue{ keyword: format!("result_{}", i2), value: format!("value_{}", i2) });
        }
        record.tests.push(Tests{ test_name: test, test_details: results })
    }
    let schema = get_schema();
    let serde_schema = Schema::parse_str(schema.as_str()).unwrap();
    let file = File::create("sample.avro").unwrap();
    let mut writer = Writer::new(&serde_schema, file);

    writer.append_ser(record).unwrap();
    writer.flush().expect("Flush to file failed");
}


/// Function that provides the table schema.
fn get_schema() -> String {
    let schema: String = r#"{
            "type": "record",
            "name": "Root",
            "fields": [
                {"name": "time", "type": "long", "logicalType": "timestamp"},
                {"name": "id1", "type": "string"},
                {"name": "id2", "type": "string"},
                {"name": "tests",
                    "type": {
                        "type": "array",
                        "items": {
                            "type": "record",
                            "namespace": "root.tests",
                            "name": "test",
                            "fields": [{
                                    "name": "test_name",
                                    "type": "string",
                                    "default": "null"
                                }, {
                                    "name": "test_details",
                                    "type": {
                                        "type": "array",
                                        "items": {
                                            "type": "record",
                                            "namespace": "root.tests.test_details",
                                            "name": "Test_Details",
                                            "fields": [{
                                                    "name": "keyword",
                                                    "type": "string",
                                                    "default": "null"
                                                }, {
                                                    "name": "value",
                                                    "type": "string",
                                                    "default": "null"
                                                }
                                            ]
                                        }
                                    }
                                }
                            ]
                        }
                    }
                }
            ]
        }"#.to_string();
    schema
}

[tustvold]

If it was not for the uncontrollable arbitrary items appearing, what we have explored would work.

I am afraid I can't really offer you any further advice on this, other than to continue to strongly encourage you to follow the parquet schema conventions and adopt the list representation specified here, as this will ensure ecosystem interoperability. I would not expect there to be a material difference between querying the two schemas, as they will both have the same underlying dremel representation.

That being said if you are unable to follow the parquet conventions for schema design, you may want to email the parquet mailing list for advice on how best to achieve this, along with the long-term implications of such a design

[tghunt]

Is there a working example of this code somwhere? I found the link from this the last page of the arrow and parquet blog.

[tustvold]

Change

let b2 = DataType::List(Box::new(Field::new("element", DataType::Int32, false)));

To

let b2 = DataType::List(Arc::new(Field::new("element", DataType::Int32, false)));

And the array construction compiles

[tghunt]

Thank you.

I am not sure, but it appears the ListBuilder being assigned for b2 is changing the signature of b2 and the name "element" is being changed to "item". If I change the name "element" to item and false to true to match what is in the struct it will run and I can print out the values object.

I was also able to get the values object out to a parquet via RecordBatch and again, there are arbitrary levels being added from the ListBuilder.

I am not seeing a way to get the structure of from the article using arrow objects.

[tustvold]

You may be right, the parquet writer follows the specification and always maps a list to a 3 element structure, so as to correctly encode both list and element nullability.

Whilst it supports reading 2 or 1 element lists for backwards compatibility, the arrow APIs will not write them