[data] add dataloader for lance datasource

[Jay-ju]

Why are these changes needed?

The storage format of lance will be used in multimodal preprocessing and training. When used for training, it needs to be loaded by Dataloader. Here are two relatively important characteristics:

1. lance supports point query and can be randomly shuffled.
2. After lance is adapted to dataloader, it can load data in a streaming manner to prevent memory overflow and excessive loading time of training data.

Therefore, this provides how lance can form a Dataset for torch Dataloader implementation, enabling the unification of ray train and ray data.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[gvspraveen]

ooc can you show example of how this work with ray.train.torch. TorchTrainer?. Currently it takes Ray Dataset as input and not Datasource.

[Jay-ju]

It is indeed using Dataaset.

[raulchen]

In Ray Data, a Datasource is to used to create a Ray Dataset, ray.data.read_datasource(...).
Users are not supposed to directly use the Datasource class for training ingestion.
If you want to do this, I think you can directly create a torch Dataset based on LanceDB without using Ray Data.

[Jay-ju]

From a theoretical perspective, directly creating a torch Dataset is also possible. it is inherited from Torch Dataset. The reason for placing it here is mainly that it can be directly converted into a dataset through the datasource, facilitating the implementation of the ray train + ray data mode.
WDYT？

[raulchen]

sorry I still don't think this makes sense.
because "Users are not supposed to directly use the Datasource class for training ingestion."

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

• If you'd like to keep this open, just leave any comment, and the stale label will be removed.