Brings java lance schema into java module

[majin1102]

For now, lance java(including spark) uses arrow schema to build dataset and retrieval from dataset.

However, when we need to config compression or other field level configurations, the field ids need to be known, which is stored in lance dominated schema. That's why we need to bring lance schema into java module, which has been done in python and rust modules.

The type transformation between java and rust could be dirty. Try best to make it light

[jackye1995]

makes sense to me

[westonpace]

I don't have any concern if we want to do this (as you said, we've done this in python and rust).

However, it does get rather tedious keeping track of two different schemas all the time. Compression and field configuration is done through the Arrow schema already (via field metadata). So the main thing missing is field ids.

Another option might be to export field ids into the arrow schema as field metadata as well.

However, I don't feel too strongly about this, if we want to stick to the multiple schema types plan.

[jackye1995]

Another option might be to export field ids into the arrow schema as field metadata as well.

My understanding is that people usually craft a record batch in some way and use the derived schema to write, and it is consolidated inside the Lance library. I feel an issue that would arise is it's hard to know which arrow schema is a "Lance arrow schema", which is not. It feels cleaner to have a dedicated LaneSchema struct at that point. Is this a valid concern?

[westonpace]

My understanding is that people usually craft a record batch in some way and use the derived schema to write, and it is consolidated inside the Lance library. I feel an issue that would arise is it's hard to know which arrow schema is a "Lance arrow schema", which is not. It feels cleaner to have a dedicated LaneSchema struct at that point. Is this a valid concern?

I think it is valid. In my mental model the only "lance schema" (with field ids) is the dataset schema. If a schema is attached to a record batch then it is just an arrow schema and shouldn't have field ids.