Avro schema parser uses type name instead of field name [AVRO]

[EmilyMatt]

Describe the bug

If I provide a schema like so:

{
  "namespace": "ns1",
  "name": "main",
  "type": "record",
  "fields": [
    {
      "name": "f1",
      "type": {
        "type": "record",
        "namespace": "ns2",
        "name": "record2",
        "fields": [
          {
            "name": "f1_1",
            "type": "string"
          }
        ]
      }
    }
  ]
}

The schema parser will use "record2" as the field name, despite it actually being the type name, the field name should be "f1", this means conversion from ArrowSchemas that don't contain the schema json in the metadata will always fail, and in general the schemas will not be applicable to the output recordbatch.

To Reproduce

Create an avro file with the above writer schema, then use the following arrow schema to create a reader_schema

Schema::new(vec![
            Field::new(
                "f1",
                DataType::Struct(
                    vec![
                        Field::new("f1_1", DataType::Utf8, false),
                    ]
                    .into(),
                ),
                false,
            )
]).with_metadata(HashMap::from([(AVRO_NAMESPACE_METADATA_KEY.into(), "ns1".into()), (AVRO_NAME_METADATA_KEY.into(), "main".into())]));

(Use AvroSchema::try_from() etc.)

It will error on mismatch in field names because the writer schema will have the field "f1" correctly, but the newly created reader_schema will have "record2"

Expected behavior

The field name should be propagated correctly

Additional context

[jecsand838]

@EmilyMatt Thank you for opening this issue. I'll take a look at it today.

[jecsand838]

take

[jecsand838]

@EmilyMatt I carefully reviewed this issue + the code over the past few hours. I realized the child metadata required for f1 was missing in your reproduction description.

Including this metadata would like this:

        let mut md = HashMap::new();
        md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(), "ns1".to_string());
        md.insert(AVRO_NAME_METADATA_KEY.to_string(), "main".to_string());

        // ======= Child Field Metadata
        let mut f1_md = HashMap::new();
        f1_md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(), "ns2".to_string());
        f1_md.insert(AVRO_NAME_METADATA_KEY.to_string(), "record2".to_string());
        // =======

        let arrow_reader_schema = ArrowSchema::new_with_metadata(
            vec![
                ArrowField::new(
                    "f1",
                    DataType::Struct(Fields::from(vec![ArrowField::new(
                        "f1_1",
                        DataType::Utf8,
                        false,
                    )])),
                    false,
                )
                .with_metadata(f1_md), // inject here
            ],
            md, // root field metadata
        );

I know this is burdensome and potentially heavy on the caller. The challenge here has been inter-oping Avro's type naming system with Arrow Schema. I couldn't find another way to do it which:

1. Didn't involve using metadata.
2. Wouldn't cause explicit failures / extra complexity for callers just wanting to encode Arrow data--not originally from an Avro source--into Avro.

The reason for the name generation logic was to make the implementation less rigid for callers going from pure Arrow (with the metadata missing) to Avro (i.e. point 2). My thinking was if there wasn't an original Avro schema, then generating one in this manner wouldn't necessarily be incorrect so long as it was in spec and repeatable with optional overrides (such as via the metadata).

Additionally I included the Arrow Schema's SCHEMA_METADATA_KEY logic as a flexible alternative for callers with the complete Avro schema so they could just inject it into the Arrow Schema for downstream use. This is approach did come in handy moving the DataFusion work forward.

Basically I was attempting to balance flexibility, practicality, and correctness, which was somewhat challenging in this scenario. Furthermore, I intended to take this rougher metadata based foundation and extend the crate's public api to gradually offload complexity from the caller while maintaining backwards compatibility. #8923 is example of that in practice.

However, I'm 100% open to and hoping for a better approach and already planned to refactor the Arrow Schema -> AvroSchema conversion logic in arrow-avro/src/schema.rs anyway due to it's brittleness. (Which is why I didn't catch the missing metadata sooner lol. If there's a bug in arrow-avro, I'm 98% confident it would pop-up in that schema conversion logic, so was already expecting it).

Anyway when I ran this reproducing test below in the arrow-avro/src/schema.rs file, with the child metadata now inserted, it passed with the reader and writer schemas matching exactly.

    #[test]
    fn test_reader_schema_from_arrow_preserves_nested_field_name() {
        // Writer schema as described in issue
        let writer_json = r#"{
          "namespace": "ns1",
          "name": "main",
          "type": "record",
          "fields": [
            {
              "name": "f1",
              "type": {
                "type": "record",
                "namespace": "ns2",
                "name": "record2",
                "fields": [
                  {
                    "name": "f1_1",
                    "type": "string"
                  }
                ]
              }
            }
          ]
        }"#;
        // Parse the writer schema into the internal Avro representation
        let writer_schema: Schema =
            serde_json::from_str(writer_json).expect("writer schema JSON should parse");
        let mut md = HashMap::new();
        md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(), "ns1".to_string());
        md.insert(AVRO_NAME_METADATA_KEY.to_string(), "main".to_string());
        let mut f1_md = HashMap::new();
        f1_md.insert(AVRO_NAMESPACE_METADATA_KEY.to_string(), "ns2".to_string());
        f1_md.insert(AVRO_NAME_METADATA_KEY.to_string(), "record2".to_string());
        let arrow_reader_schema = ArrowSchema::new_with_metadata(
            vec![
                ArrowField::new(
                    "f1",
                    DataType::Struct(Fields::from(vec![ArrowField::new(
                        "f1_1",
                        DataType::Utf8,
                        false,
                    )])),
                    false,
                )
                .with_metadata(f1_md),
            ],
            md,
        );
        // Build the Avro reader schema from the Arrow schema
        let reader_avro = AvroSchema::try_from(&arrow_reader_schema)
            .expect("Arrow -> Avro schema conversion should succeed");
        let reader_schema: Schema = reader_avro
            .schema()
            .expect("generated Avro schema JSON should parse");
        // Both writer and reader should be top-level records
        let writer_record = match &writer_schema {
            Schema::Complex(ComplexType::Record(r)) => r,
            other => panic!("expected writer to be a record, got: {other:?}"),
        };
        let reader_record = match &reader_schema {
            Schema::Complex(ComplexType::Record(r)) => r,
            other => panic!("expected reader to be a record, got: {other:?}"),
        };
        // Sanity: writer field name is "f1"
        assert_eq!(writer_record.fields.len(), 1);
        assert_eq!(writer_record.fields[0].name, "f1");
        assert_eq!(
            reader_record.fields[0].name, "f1",
            "Reader schema must preserve the Arrow field name `f1` and \
                 must not substitute the nested record's type name (`record2`) as the field name"
        );
        println!("{reader_schema:#?}\n\n{writer_schema:#?}");
        // Critical assert: If the bug is present, the reader schema and writer schema will not match
        assert_eq!(reader_schema, writer_schema);
        let _resolved = AvroFieldBuilder::new(&writer_schema)
            .with_reader_schema(&reader_schema)
            .with_utf8view(false)
            .with_strict_mode(false)
            .build()
            .expect(
                "Writer and generated reader schemas should resolve without field name mismatches",
            );
    }

I do think the schema.rs constant names + relevant docs being better / more descriptive would help here as well.

Let me know if you think this resolves the issue.

CC: @alamb @nathaniel-d-ef

[EmilyMatt]

Honestly I think my main interest is the cases where the ArrowSchema does not contain any metadata.
If I provide a valid arrow schema, do an AvroSchema::try_from on it, and provide it as the read_schema, I expected the two to be compatible.
Currently it crashes, which is my main issue^^

[jecsand838]

Honestly I think my main interest is the cases where the ArrowSchema does not contain any metadata. If I provide a valid arrow schema, do an AvroSchema::try_from on it, and provide it as the read_schema, I expected the two to be compatible. Currently it crashes, which is my main issue^^

That makes sense and I 100% understand where you're coming from.

Right now AvroSchema::try_from(&ArrowSchema) can only work with whatever information is actually present in the ArrowSchema. When there’s no Avro metadata at all, the Arrow side won't contain:

1. The original Avro record type names, namespaces, etc. (i.e. ns2.record2), or
2. Any indication that a given Struct used to be a named Avro record.

Because AvroSchema::try_from was originally written for the Writer, missing Schema metadata was interpreted as there being no originally sourced Avro schema, therefore we were free to generate one with the information available.

So for something like your nested f1 struct, AvroSchema::try_from has to invent a legal Avro record name. The current implementation does that by using the field name ("f1") when avro.name / avro.namespace aren’t present. That means the generated reader schema ends up with:

writer: nested record name  = "record2" (namespace "ns2")
reader: nested record name  = "f1"      (no namespace)

When you then feed that generated schema into the resolver as read_schema, Avro resolution correctly fails with a record‑name mismatch. From Avro’s perspective those are two different types, even though the shape matches.

Essentially there are two slightly different expectations here:

1. What the code currently guarantees:
   From an Arrow schema with no/missing Avro metadata, arrow-avro will generate a valid Avro schema that correctly describes the Arrow layout, but we can't promise it will be compatible with a pre‑existing Avro writer schema that used specific named record types / namespaces, etc.

2. More streamlined approach:
   If the ArrowSchema is structurally compatible with the writer schema, then AvroSchema::try_from + read_schema should just work, even without metadata.

For schemas that don’t rely on Avro named types, those two line up and things work fine. For named records (like your record2), they diverge because Arrow simply doesn’t carry all the needed information unless we stash them in metadata.

Practically speaking, the patterns available are:

1. If you already have the Avro writer schema JSON, reuse that in the read_schema (or store it in SCHEMA_METADATA_KEY on the Arrow Schema), or
2. When you do want to round‑trip via Arrow, attach avro.name / avro.namespace on the relevant struct/dictionary fields so the generated reader schema can match the writer’s names.

I totally agree this is not obvious from the API, and from your perspective it just looks like you passed a perfectly valid Arrow schema and the resolution crashed. So at a minimum we should:

• Document this limitation more clearly, and
• Consider whether we can support a more permissive resolution mode for reader schemas derived from Arrow (i.e. fall back to structural matching when the reader has no avro.name/avro.namespace), without breaking spec‑compliant cases.
• Extend the public API to simplify as much of this metadata related complexity as possible. Perhaps new logic to make it easy to derive a resolution compatible reader schema from a specific writer schema?

If you’re interested, we could track that as a follow‑up enhancement: something along the lines of "support using pure Arrow schemas as reader schemas without requiring Avro naming metadata", and then discuss what trade‑offs / flags that would need.