Proposal: Add a method to directly create an arrow::RecordBatch

[alamb]

Again, thank you for this library.

arrow makes heavy use of RecordBatch (while arrow2 removed the equivalent concept and deals in Vecs of Arrays).

It is possible to make RecordBatch from the APIs in this crate, as I demonstrate in #131, I think it would a nicer API if this crate offered a convenience method for doing so directly

Here is what you have to do today (copied from #131)

// Determine Arrow schema
let fields =
  SerdeArrowSchema::from_type::<Record>(TracingOptions::default())?
  .to_arrow_fields()

// Convert to Arrow arrays
let arrays = serde_arrow::to_arrow(&fields, &records)?;

// Form a RecordBatch
let schema = Schema::new(&fields);
let batch = RecordBatch::try_new(schema.into(), arrays)?;

Proposal

I propose adding a to_batch method that coverts directly to arrow RecordBatch:

pub fn to_batch<T: Serialize + ?Sized>(fields: &[Field], items: &T) -> Result<RecordBatch> {
...
}

Example use:

// Determine Arrow schema
let fields =
  SerdeArrowSchema::from_type::<Record>(TracingOptions::default())?
  .to_arrow_fields()

// Convert to RecordBatch
let batch = serde_arrow::to_batch(&fields, &records)?;

[alamb]

If this is acceptable to you @chmp I would be happy to create a PR

[chmp]

Hey @alamb.

Thanks for the offer, in principle I would be open to an PR, but only reluctantly.

To explain my reservations: due to the constant API churn I consider a larger exposure to the arrow API a risk. Initially serde_arrow was built around the arrow API, but after multiple updates breaking my code (within a six week timeframe), I took the decision to minimize the arrow API used. That's the reason why all builders are implemented from scratch. And that's the reason why I removed the RecordBatch based API in serde_arrow some time ago.

The regular major version release is also the reason why this whole non-additive feature issue exists (at least I didn't think of a solution yet that does not involve releasing a breaking change every 6 weeks myself).

[alamb]

No worries -- your explantion makes sense. I think having an example (e.g. #131) would provide 90% of the value. Maybe over time depending on how peple use the crate it would make sense to offer additional APIs, but I can see the value in keeping things simple

[chmp]

Btw. the to_arrow function already includes the corresponding docs. But better to include a more prominent example.

[chmp]

See also #137

[chmp]

Added a to_record_batch on main

[chmp]

Released as serde_arrow=0.11.0