[Data] Update Data+Train Benchmark #39276
Conversation
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
|
Could you update the PR description? Also, I'm thinking we should actually update the benchmarks to not have to use a GPU, since we're currently not even testing this part. Can we switch to an m5 instance type with the same number of CPUs? There should be a way to override the |
| parser.add_argument( | ||
| "--read-task-cpus", | ||
| default=1, | ||
| type=int, | ||
| type=float, |
There was a problem hiding this comment.
Shall we scale the target max block size by this value?
There was a problem hiding this comment.
@stephanie-wang which max block size are you referring to here? and what would be an appropriate scaling factor?
There was a problem hiding this comment.
DataContext.target_max_block_size.
You can just multiply it by this value to scale it properly.
| if args.use_torch: | ||
| torch_num_workers = args.torch_num_workers | ||
| if torch_num_workers is None: | ||
| torch_num_workers = 256 |
There was a problem hiding this comment.
Probably best to make this in terms of the CPU count and whether the data is stored locally, instead of hardcoding.
Let's also share the code with use_mosaic?
There was a problem hiding this comment.
discussed with @c21 , for now, we will use 256 workers for all cases directly using TorchLoader. implemented logic based on CPU count for mosaic
There was a problem hiding this comment.
Could we just share the torch_num_workers for both? Mosaic is also using a torch dataloader.
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
|
|
||
| head_node_type: | ||
| name: head_node | ||
| instance_type: m5.4xlarge |
There was a problem hiding this comment.
I think there is a way to pass a custom num_gpus here, so that way we can still use --use-gpu for the script.
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Add Torch/Mosaic data loading benchmarks
Add various command line parameters for customizing the benchmark run (e.g. data loader type, dataset size/type/location, dataset size per worker, etc.)
Add utility functions to generate image/parquet datasets based on data size per worker
Increased release test timeouts accordingly
---------
Signed-off-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Cheng Su <scnju13@gmail.com>
Co-authored-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.