Destination S3 (avro/parquet format): handle empty schemas

[edgao]

from https://github.com/airbytehq/oncall/issues/6299

currently, schemas like {type: array, items: {}} throw an illegal arg exception because of the empty {} nested schema. This PR:

• Adds a JSON type to represent that schema
• adds functionality to the mappers to handle that schema
• updates the avro schema+record mappers to convert JSON to string

(.... eventually we should just use AirbyteType here 😅 which has much nicer handling for array/object/primitive types in general)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 30, 2024 6:06pm

[johnny-schmidt]

Should we be explicit about the fact that we're matching an empty {}? Or do you want to use this as a catch-all

[edgao]

more a catchall. So, I guess I should rename it to unknown in that case 🤔

[johnny-schmidt]

This is true in the sense that Parquet doesn't have a Json type per se. It has a json-logical type that can be applied to strings, which we don't use, because there's no way to represent it in the Avro schema.

[edgao]

gotcha. I'll update the comment to reflect that

[johnny-schmidt]

This looks correct to me. You could also consider adding these types to the DATs for avro and parquet.

The relevant test method is DestinationAcceptanceTest.testProblematicTypes (I'm bad at naming sorry).

And the workflow for adding them would be

• add fields of that type to the end of the test catalog file v0/problematic_types_configured_catalog.json (you can step into it from the test in intellij)
• add corresponding fields to the end of v0/problematic_types_messages_in.txt. There are only three rows, which are basically Variant1, Variant2 (if applicable), and Nulled
• add the expected avro output to the end of the three rows in v0/problematic_types_coerced_schemaless_schema.json (should really be renamed)
• add the expected parquet output to the end of the three rows in v0/problematic_types_disjoint_union_schema.json
• (expected output in this case would be the same: stringified json blob)

[johnny-schmidt]

Other than possibly amending the DATs, this is good to go.

[edgao]

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/10636936796
✅ Successfully published Java CDK version=0.44.20!

[manuito]

Hi. Since this release on Airbyte Cloud we are facing issues : the sync_id is not anymore in destination file name. Seen by some other users => https://airbytehq.slack.com/archives/C021JANJ6TY/p1725177438711539

I have opened issue #45084
It's a very impactful regression for us 🙏