Skip to content

Don't include files list DatasetInfo (and remove old stuff) - #8128

Merged
lhoestq merged 1 commit into
mainfrom
dont-include-files-list-in-info
Apr 16, 2026
Merged

lhoestq merged 1 commit into
mainfrom
dont-include-files-list-in-info

Conversation

@lhoestq

@lhoestq lhoestq commented Apr 10, 2026

Copy link
Copy Markdown
Member

this will fix the viewer when it saves the dataset info in mongo, for datasets with a lot of files this can make the mongo doc too big for mongo

I also removed old deprecated stuff linked to files list like the old concept of "post-processing" and cleaned the checksum recording logic

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq
lhoestq merged commit 4ff88bf into main Apr 16, 2026
2 of 15 checks passed
@lhoestq
lhoestq deleted the dont-include-files-list-in-info branch April 16, 2026 09:23
harupy added a commit to harupy/mlflow that referenced this pull request May 12, 2026
`datasets` 4.8.5 stopped populating `DatasetInfo.download_checksums`
(huggingface/datasets#8128), so the revision assertion in
`test_from_huggingface_dataset_constructs_expected_dataset_with_revision`
trips on `None`. Switch the canonical revision check to
`mlflow_ds_new.source.revision`, and keep the checksum check only when
the field is still populated.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>
harupy added a commit to harupy/mlflow that referenced this pull request May 12, 2026
`datasets` 4.8.5 stopped populating `DatasetInfo.download_checksums`
(huggingface/datasets#8128), so the revision assertion in
`test_from_huggingface_dataset_constructs_expected_dataset_with_revision`
trips on `None`. Switch the canonical revision check to
`mlflow_ds_new.source.revision`, and keep the checksum check only when
the field is still populated.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: harupy <17039389+harupy@users.noreply.github.com>
DaoDaoNoCode added a commit to DaoDaoNoCode/mlflow that referenced this pull request May 15, 2026
datasets 4.8.5 stopped populating DatasetInfo.download_checksums
(huggingface/datasets#8128), so the revision assertion trips on
None with a TypeError. Switch the revision check to
mlflow_ds_new.source.revision, which is the canonical source of
truth and equally captures the intent of the assertion.

Cherry-picked from mlflow/mlflow@87182fb

Signed-off-by: Juntao Wang <juntwang@redhat.com>
DaoDaoNoCode added a commit to DaoDaoNoCode/mlflow that referenced this pull request May 15, 2026
datasets 4.8.5 stopped populating DatasetInfo.download_checksums
(huggingface/datasets#8128), so the revision assertion trips on
None with a TypeError. Switch the revision check to
mlflow_ds_new.source.revision, which is the canonical source of
truth and equally captures the intent of the assertion.

Cherry-picked from mlflow/mlflow@87182fb (mlflow#23228)

Signed-off-by: Juntao Wang <juntwang@redhat.com>
mprahl pushed a commit to opendatahub-io/mlflow that referenced this pull request May 15, 2026
datasets 4.8.5 stopped populating DatasetInfo.download_checksums
(huggingface/datasets#8128), so the revision assertion trips on
None with a TypeError. Switch the revision check to
mlflow_ds_new.source.revision, which is the canonical source of
truth and equally captures the intent of the assertion.

Cherry-picked from mlflow/mlflow@87182fb (mlflow#23228)

Signed-off-by: Juntao Wang <juntwang@redhat.com>
Elsword016 pushed a commit to Elsword016/datasets that referenced this pull request Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants