Preserve formatting in concatenated IterableDataset #7522
Conversation
…have consistent formatting
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
lhoestq
left a comment
lhoestq
left a comment
There was a problem hiding this comment.
Cool ! I took the liberty to update the branch with the changes from main, and added come comments:
src/datasets/iterable_dataset.py
Outdated
| _check_column_names([col_name for dset in dsets for col_name in dset.features]) | ||
|
|
||
| # Check format is consistent; if so, will set format for concatenated dataset | ||
| format_type_set = {dset._formatting.format_type for dset in dsets} |
There was a problem hiding this comment.
dset._formatting doesn't always exist
So if dset._formatting is None for all the datasets, the formatting of the output dataset should be None as well
There was a problem hiding this comment.
I've added a check in the set comprehension, so that it skips elements where dset._formatting is None. This means that we will try to set the output format to that of the inputs who do have dset._formatting. Is that reasonable? or we should set the output format to None in this case?
There was a problem hiding this comment.
any discrepancy should result in formatting=None, if we want to be consistent with the Dataset API
There was a problem hiding this comment.
Sounds good! I've reworked the logic to account for that.
src/datasets/iterable_dataset.py
Outdated
|
|
||
| # Check format is consistent; if so, will set format for concatenated dataset | ||
| format_type_set = {dset._formatting.format_type for dset in dsets} | ||
| format_type = format_type_set.pop() if len(format_type_set) == 1 else None |
There was a problem hiding this comment.
if the datasets have disparate formats, maybe you can add this logging info (same as for concatenating Dataset objects):
logger.info("Some of the datasets have disparate format. Resetting the format of the concatenated dataset.")…enated dataset format to None. Add log line for inputs with inconsistent format.
3 participants
Fixes #7515