Skip to content

Forward max_buffer_input_shards from IterableDatasetDict.shuffle - #8594

Merged
lhoestq merged 1 commit into
huggingface:mainfrom
vineethsaivs:fix/iterable-dataset-dict-shuffle-input-shards
Sep 10, 2026
Merged

lhoestq merged 1 commit into
huggingface:mainfrom
vineethsaivs:fix/iterable-dataset-dict-shuffle-input-shards

Conversation

@vineethsaivs

Copy link
Copy Markdown
Contributor

What breaks

load_dataset(..., streaming=True) without a split returns an IterableDatasetDict, and shuffling it cannot reach the shard-interleaving control:

ds = load_dataset("HuggingFaceFW/fineweb", streaming=True)
ds = ds.shuffle(seed=42, buffer_size=10_000, max_buffer_input_shards=4)
# TypeError: IterableDatasetDict.shuffle() got an unexpected keyword argument 'max_buffer_input_shards'

#8194 added max_buffer_input_shards to IterableDataset.shuffle and updated docs/source/stream.mdx, but not dataset_dict.py, so the per-split method has the argument and the dict wrapper silently does not. Every other IterableDatasetDict.shuffle argument is forwarded.

Second, smaller thing in the same argument

Its docstring entry in iterable_dataset.py reads

max_buffer_input_shards (`int`, defaults to `101000`):

The signature says max_buffer_input_shards: int = 10, and stream.mdx says "default is 10". 101000 is the 10 and the neighbouring buffer_size default of 1000 run together.

The change

Accept the argument on IterableDatasetDict.shuffle with the same default, document it with the same wording, and pass it to each split. Fix the stated default in the IterableDataset.shuffle docstring.

Test

test_iterable_dataset_dict_shuffle_forwards_max_buffer_input_shards in tests/test_iterable_dataset.py, next to the existing test_iterable_dataset_shuffle_buffer_uses_multiple_input_shards and using its assertion: with max_buffer_input_shards=1 the first ten examples must come from at most two shards, which only holds if the argument reaches the split.

tests/test_iterable_dataset.py -k input_shards
  before: 1 failed, 1 passed
          TypeError: IterableDatasetDict.shuffle() got an unexpected keyword argument 'max_buffer_input_shards'
  after:  2 passed

tests/test_iterable_dataset.py tests/test_dataset_dict.py
  559 passed, 9 skipped

ruff check and ruff format --check are clean on the three files.

`IterableDataset.shuffle` grew a `max_buffer_input_shards` argument in huggingface#8194 and
the dict wrapper was not updated, so calling it on the result of
`load_dataset(..., streaming=True)` raises TypeError.

The same argument's docstring entry states the default as 101000; the signature
says 10, and the streaming docs say 10.

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch !

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq
lhoestq merged commit 636178b into huggingface:main Sep 10, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants