parquet: Add ArrowWriterOptions to skip embedding the arrow metadata

[evenyag]

Which issue does this PR close?

Closes #5296.

Rationale for this change

What changes are included in this PR?

Adds the following APIs

• ArrowWriter::try_new_with_options()
• AsyncArrowWriter::try_new_with_options()
• ArrowWriterOptions

Are there any user-facing changes?

Users can

• use ArrowWriterOptions::with_skip_arrow_metadata() to set skip_arrow_metadata to true and skip embedding the arrow schema.
• use ArrowWriterOptions::with_properties() to configure the parquet writer.

[tustvold]

This looks good to me thank you, I do think we should probably move WriterProperties on to ArrowWriterOptions though, as forcing users to juggle two options collections seems cumbersome. Apologies for the slow review, been a bit swamped lately

[tustvold]

I wonder if we could move this on to ArrowWriterOptions?

[evenyag]

Sure, I'll try it. The cpp FileWriter API separates these two parameters so I kept try_new_with_options() similar to theirs.

::arrow::Result<std::unique_ptr<FileWriter>> Open(
    const ::arrow::Schema &schema,
    MemoryPool *pool,
    std::shared_ptr<::arrow::io::OutputStream> sink,
    std::shared_ptr<WriterProperties> properties = default_writer_properties(),
    std::shared_ptr<ArrowWriterProperties> arrow_properties = default_arrow_writer_properties()
)

[waynexia]

I do think we should probably move WriterProperties on to ArrowWriterOptions

This approach looks good to me. And we can also consider deprecating the old try_new method or making a breaking change to remove it, since these two are very similar

[tustvold]

Thank you for this