ENH: avro to/from serialization

[jreback]

discussed in #3525

shiny new fast version of avro might be interesting: adtech-labs/cyavro#1
by @mariusvniekerk

[jreback]

cc @wesm

[wesm]

any performance numbers on cyavro vs pyavroc vs fastavro?

[VelizarVESSELINOV]

👍

[manugarri]

👍

[Khrol]

👍

[manugarri]

any updates on this? I am willing to put time to implement this, however would need some pointers, specially regarding:

• Would we cast pandas dtypes into avro automatically? Or would we enforce a user defined schema? I believe the pandas approach would be the former, but it would be good (and easier) to allow for a specific schema. Casting the types would require some assumptions (avro has support for a set of primitives that might not match pandas').

• Would read_avro support for headerless data (i.e. streaming data without the header)? Would make sense, to allow this feature as long as the user provides the schema.

[wesm]

One possible route for this is https://issues.apache.org/jira/browse/ARROW-1209 -- @mariusvniekerk has started working on this, and it would be easier to accommodate Avro's types in Arrow and deal with the marshaling to/from pandas DataFrame at a central location in Arrow-land.

[jbrockmendel]

Is this a live topic or is it subsumed by more recent arrow ecosystem developments?

[wesm]

Arrow-based Avro read/write isn't shipping yet, @emkornfield is interested in this as it relates to various Google services (among other uses). Hard to predict the timeline for that, though.

[emkornfield]

Yes, I'm hoping to make some progress on this towards the end of this month/early next month.