Skip to content

[httpx migration] Import httpx from huggingface_hub, require huggingface_hub>=1.31.0 - #8598

Merged
lhoestq merged 3 commits into
huggingface:mainfrom
Wauplin:fix/httpx-imports
Sep 11, 2026
Merged

lhoestq merged 3 commits into
huggingface:mainfrom
Wauplin:fix/httpx-imports

Conversation

@Wauplin

@Wauplin Wauplin commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Direct httpx imports will break when huggingface_hub migrates to httpx2. This updates them to use the compatibility re-export introduced in huggingface/huggingface_hub#4803, following the migration plan in huggingface/huggingface_hub#4802. Goal is to make datasets compatible with both huggingface_hub 1.x and 2.x when the switch to httpx2 happens.

Following #8598 (comment), this also bumps the minimum huggingface_hub version and removes the code that was only there to keep working with older releases.

Minimum version

Bumped to 1.31.0 (in setup.py and in the "minimum versions" CI job) rather than 1.30.0: huggingface_hub.utils.httpx landed in 1.31.0, not in 1.30.0 as the migration plan anticipated.

requests and httpx are also gone from the dependencies. requests was only needed for huggingface_hub 0.x, and pinning httpx<1.0.0 here would defeat the purpose of the re-export — datasets should not constrain the httpx major version, huggingface_hub owns that now.

Legacy code removed

  • the config.HF_HUB_VERSION >= 1.6.0 gates in arrow_dataset.py, dataset_dict.py, iterable_dataset.py, load.py, tests/fixtures/hub.py and tests/test_iterable_dataset.py, with the BucketNotFoundError = None / HfFileSystemResolvedRepositoryPath = HfFileSystemResolvedPath fallbacks and the matching raise ImportError("... requires huggingface_hub>=1.6.0") guards
  • except EntryNotFoundError around dirfs.glob(...), marked # needed for huggingface_hub<=1.7.1
  • the requests exceptions caught next to their httpx counterparts in load.py and in CONNECTION_ERRORS_TO_RETRY
  • proxies=download_config.proxies passed to hf_hub_download, which huggingface_hub has silently ignored since 1.0. DownloadConfig.proxies is kept for backward compatibility, with a docstring pointing at HTTP_PROXY / set_client_factory
  • in the test helpers: IS_HF_HUB_1_x and its two branches in offline(), require_buckets_support_in_huggingface_hub (buckets are always available now), the requests half of xfail_if_500_502_http_error, and the huggingface_hub.file_download.HUGGINGFACE_CO_URL_TEMPLATE monkeypatch fallback

Once huggingface_hub 2.0 is out, the last step of the plan is to widen the pin to <3.0.

ðŸĪ– Generated with Claude Code

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@Wauplin

Wauplin commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Failing CI seems unrelated

FAILED tests/test_hub.py::test_delete_from_hub - huggingface_hub.errors.HfHubHTTPError: Server error '502 Bad Gateway' for url 'https://hub-ci.huggingface.co/api/datasets/__DUMMY_DATASETS_USER__/test-dataset-14a7d7-17890532107233/commit/main'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502
==== 1 failed, 3159 passed, 180 skipped, 232 warnings in 322.78s (0:05:22) =====

@Wauplin
Wauplin marked this pull request as ready for review September 10, 2026 15:57
@Wauplin
Wauplin requested a review from lhoestq September 10, 2026 15:57
@lhoestq

lhoestq commented Sep 10, 2026

Copy link
Copy Markdown
Member

cool ! I guess we need to bump the minimal version from 0.25.0 to 1.30.0 (both in setup.py and in the "minimum version" CI)

`huggingface_hub.utils.httpx` only exists in huggingface_hub>=1.31.0, so the
re-export used to stay forward-compatible with httpx2 requires bumping the
minimum version anyway. Bump it to 1.31.0 (setup.py + minimum-versions CI) and
remove everything that was only there to keep working with older releases:

- `config.HF_HUB_VERSION >= 1.6.0` gates around `BucketNotFoundError` and the
  `HfFileSystemResolved{Bucket,Repository}Path` classes, plus the
  `raise ImportError("... requires huggingface_hub>=1.6.0")` guards
- `except EntryNotFoundError` around `dirfs.glob`, needed for hfh<=1.7.1
- `requests` exceptions caught next to their `httpx` counterparts, the
  `requests` dependency itself, and the direct `httpx` pin (it now comes from
  huggingface_hub, so datasets no longer constrains the httpx major)
- `proxies=` passed to `hf_hub_download`, silently ignored since hfh 1.0
- `IS_HF_HUB_1_x`, `require_buckets_support_in_huggingface_hub` and the 0.x
  branches in the test helpers

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Wauplin Wauplin changed the title [httpx migration] Import httpx from huggingface_hub [httpx migration] Import httpx from huggingface_hub, require huggingface_hub>=1.31.0 Sep 11, 2026
@Wauplin

Wauplin commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author

@lhoestq good idea. PR now removes all dead code that was needed for legacy versions of huggingface_hub (especially to gracefully raise on versions not supporting buckets, and on versions running on requests)

Failing CI all come from HTTP 500 from staging (tests are flaky on hfh as well nowadays 😎)

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants