[Datasets] Add basic e2e Datasets example on NYC taxi dataset

[clarkzinzow]

This PR adds a dedicated docs page for examples, and adds a basic e2e tabular data processing example on the NYC taxi dataset.

The goal of this example is to demonstrate basic data reading, inspection, transformations, and shuffling, along with ingestion into dummy model trainers and doing dummy batch inference, for tabular (Parquet) data. The example doesn't use Ray AIR since AIR is still alpha, and doesn't (yet) use Ray Train in order obviate the need for:

1. performing real feature engineering which be overly advanced for this example,
2. defining a model that works with the NYC taxi data which would be non-trivial.

We can either complete the port to Ray Train using a dummy linear model in this example, or do that as a more targeted ML ingest example. I'm currently planning on the latter, but I'm happy to discuss the former!

Docs preview: https://ray--24874.org.readthedocs.build/en/24874/data/examples/index.html

TODOs

• (Maybe?) Port to linear Torch model and do full Ray Train + model inference

[clarkzinzow]

cc @maxpumperla for the examples page and general structure.

[ericl]

I enjoyed reading through this one. A couple operational comments:

• Seems a bunch of references aren't rendering properly, double check them?
• I see a bunch of yellow warnings from requests, is there a way to hide those?

[clarkzinzow]

Seems a bunch of references aren't rendering properly, double check them?

Hmm strange, they are all straightforward references. 🤔 I'll dig into that.

I see a bunch of yellow warnings from requests, is there a way to hide those?

I'll try disabling warnings.

[ericl]

On the example data, could we also mention the ability to read the entire directory / partitioning support? Specifying individual files seems not the common case.

[clarkzinzow]

@pcmoritz Do you want to take a quick sign-off pass on this example?

https://ray--24874.org.readthedocs.build/en/24874/data/examples/nyc_taxi_basic_processing.html

This is supposed to be one of the "basic data processing" examples, where we're going to do one of these for each data type (tabular, text, imagery) per the examples plan, where we're planning on waiting to do e2e ML workload examples (with full feature engineering + non-dummy trainers and non-dummy batch inferrers) until the AIR examples land.

[clarkzinzow]

@ericl @jianoaix Ping on data team final review/approval.

[ericl]

Can you delete the cell from the top of the notebook?

# flake8: noqa
import warnings
import os

# Suppress noisy requests warnings.
warnings.filterwarnings("ignore")
os.environ["PYTHONWARNINGS"] = "ignore"

Maybe mark as hidden or delete it manually from JSON.

There are also a bunch of ugly stuff like

2022-05-17 19:42:53,496	WARNING read_api.py:252 -- The number of blocks in this dataset (2) limits its parallelism to 2 concurrent tasks. This is much less than the number of available CPU slots in the cluster. Use `.repartition(n)` to increase the number of dataset blocks.

(scheduler +12m43s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.

(scheduler +13m18s) Warning: The following resource request cannot be scheduled right now: {'CPU': 1.0}. This is likely due to all cluster resources being claimed by actors. Consider creating fewer actors or adding more nodes to this Ray cluster.

That should be removed.

[pcmoritz]

Thanks a lot for putting these examples together -- they look great. I have the same comments as Eric about the verbose outputs and the cell at the top -- we should get rid of those.

Also I was wondering if and why the batch_format="pandas" part in some of the calls was needed.

The pictures for the datasets are a very nice touch!

For the second example it would be great to mention what the execution model for create_shuffle_pipeline is (am I correct in assuming that the DatasetPipeline is a lazy object/expression that gets executed when iter_batches is called?).

Ideally the second example would also use a real dataset and then e.g. train with XGBoost, but we can leave this as a follow up item.

Also, are these examples tested in CI/release tests?

[clarkzinzow]

Also I was wondering if and why the batch_format="pandas" part in some of the calls was needed.

Not required, I was including it to make it explicit that we're applying a Pandas UDF, but the default batch format will provide Pandas batches to the UDF. For the sake of maximally narrow API calls, are you voting that this explicit batch format be removed?

For the second example it would be great to mention what the execution model for create_shuffle_pipeline is (am I correct in assuming that the DatasetPipeline is a lazy object/expression that gets executed when iter_batches is called?).

Ideally the second example would also use a real dataset and then e.g. train with XGBoost, but we can leave this as a follow up item.

FYI that is an existing example (from many months ago) that I'm planning on tweaking in another PR, but I'll make a note of that feedback!

Also, are these examples tested in CI/release tests?

I'll add the Bazel target now for these example notebooks to run; if the second example (large-scale ML ingest) fails in CI, then I'll disable the Bazel target until I submit the follow-up PR tweaking that example.

[clarkzinzow]

I guess we don't have support for the note directive in notebooks?

[clarkzinzow]

@ericl @jianoaix @pcmoritz Ping for final review, I've implemented all above feedback, PTAL: https://ray--24874.org.readthedocs.build/en/24874/data/examples/nyc_taxi_basic_processing.html

[clarkzinzow]

@ericl @maxpumperla @jjyao @scv119 Need a code-owner review here.

[clarkzinzow]

@ericl @maxpumperla @jjyao @scv119 Ping again.

[maxpumperla]

looks really solid to me