Repository navigation
fix(registry): follow redirects only within the configured registry origin - #3785
Daniel Meppiel (danielmeppiel) merged 9 commits into
Conversation
âĶrigin Add one origin-bounded redirect owner, origin_bounded_request, in utils/http_credentials.py. Every hop is sent with allow_redirects=False; 301/302/303/307/308 are followed only to the same scheme, host and effective port under the configured base path (RFC 3986 dot-segment, %2e/%2f and '//' normalization, IDNA host compare, no userinfo, no IPv6 zone IDs, no HTTPS downgrade), at most 3 hops. 307/308 keep the method and body; 301/302/303 are followed only for GET and HEAD. Credentials are re-sent only on in-bound hops, and credential-bearing requests never pick up ambient .netrc credentials. Proxy-Authorization joins the credential header set. Route the registry client (/versions, archive download, lockfile replay, publish) through the owner, so a gateway that answers 307 to the decoded same-origin path now works. Registry publish now follows redirects only within the configured registry origin. Marketplace metadata fetches use the shared credential-scoped HTTP owner, and the marketplace client session follow moves into the owner. The lockfile keeps the canonical pre-redirect resolved_url. Guard: transport-platform-origin-bounded-redirect (new checks module, registered on credential-origin-scope) with a frozen, reasoned inventory of sites that refuse every redirect or carry no credential. apm-spec-waiver: redirect follow policy is transport hardening with no OpenAPM v0.1 requirement Fixes #3726 Co-authored-by: AKASH KUMAR <33580834+ashky23@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶ-pr2-redirect-owner
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
ðĄ Changes recommended
Critical origin and path validation bypasses can expose registry credentials to unintended destinations, while supposedly anonymous requests can send ambient .netrc credentials and lifecycle tests fail to verify authenticated delivery.
5 open issues
Reject encoded traversal remaining after the decode limit · New Reject paths escaping the base under alternate normalization · New Use Requests-compatible IDNA normalization for origin checks · New Disable ambient .netrc credentials on anonymous requests · New Bind the fixture registry and assert credential delivery · New
What changed in this PR
Addresses #3726 by centralizing redirect handling so registry operations can follow same-origin gateway redirects while enforcing origin, base-path, method, and hop limits.
Changes:
- Routes registry requests and marketplace fetches through a shared redirect owner.
- Adds redirect regression tests, lifecycle fixtures, and an architecture guard.
- Documents redirect behavior and gateway configuration guidance.
| File | Description |
|---|---|
tests/âutils/âmetadata_transport.py |
Adapts existing metadata mocks to the shared HTTP seam. |
tests/âutils/âlocal_scripted_https_proxy.py |
Adds a recording HTTPS proxy for hermetic lifecycle tests. |
tests/âunit/âutils/âtest_http_credentials.py |
Tests redirect bounds, credentials, methods, and hop limits. |
tests/âunit/âregistry/âtest_client_redirects.py |
Covers registry lookup, replay, and publish redirects. |
tests/âunit/âmarketplace/âtest_metadata_enrichment.py |
Migrates metadata mocks to the new transport seam. |
tests/âunit/âmarketplace/âtest_marketplace_commands.py |
Updates session doubles for shared-owner requests. |
tests/âunit/âmarketplace/âtest_marketplace_client.py |
Updates mocks and tests credential handling across redirects. |
tests/âunit/âmarketplace/âtest_builder.py |
Migrates builder metadata mocks. |
tests/âunit/âmarketplace/âtest_builder_security.py |
Tests refusal of cross-origin metadata redirects. |
tests/âintegration/âtest_wave7_policy_registry_coverage.py |
Updates marketplace request patch locations. |
tests/âintegration/âtest_registry_redirect_origin_lifecycle.py |
Adds install, replay, and publish redirect scenarios. |
tests/âintegration/âtest_marketplace_metadata_origin_lifecycle.py |
Adds metadata credential-isolation lifecycle coverage. |
tests/âintegration/âtest_marketplace_metadata_auth_fallback.py |
Preserves fallback coverage through the new seam. |
tests/âintegration/âtest_marketplace_builder_phase3w4.py |
Migrates builder transport mocks. |
tests/âintegration/âtest_marketplace_builder_hermetic.py |
Updates hermetic metadata mocks. |
tests/âintegration/âtest_architecture_owner_rule_mutations.py |
Adds a redirect-owner bypass mutation. |
src/âapm_cli/âutils/âhttp_credentials.py |
Introduces bounded redirect handling and URL normalization. |
src/âapm_cli/âmarketplace/âclient.py |
Uses the shared owner with anonymous cross-origin following. |
src/âapm_cli/âmarketplace/âbuilder.py |
Moves metadata fetching from urllib to the shared owner. |
src/âapm_cli/âdeps/âregistry/âclient.py |
Centralizes sending and maps redirect refusals to registry errors. |
scripts/âarchitecture_linter/âgroups/âtransport_platform.py |
Registers the redirect guard. |
scripts/âarchitecture_linter/âchecks/âtransport_origin_bounded_redirect.py |
Enforces shared-owner routing and explicit exceptions. |
packages/âapm-guide/â.apm/âskills/âapm-usage/âauthentication.md |
Documents registry redirect credential boundaries. |
docs/âsrc/âcontent/âdocs/âreference/âregistry-http-api.md |
Defines redirect behavior and gateway guidance. |
docs/âsrc/âcontent/âdocs/âguides/âregistries.md |
Explains redirects for encoded identities. |
docs/âsrc/âcontent/âdocs/âenterprise/âregistry-proxy.md |
Adds redirect-refusal troubleshooting guidance. |
CHANGELOG.md |
Records the redirect fix and metadata transport changes. |
.apm/âarchitecture/âowners/âtransport-auth-platform.json |
Assigns redirect decisions to the credential-origin owner. |
Review effort: Balanced
âĶcross-origin follow origin_bounded_request gains ambient_credentials=False so an anonymous auth-chain step never picks up .netrc credentials. Marketplace _http_get exposes follow_cross_origin_without_credentials per call so Azure DevOps REST can hand an unfollowed sign-in redirect back to the auth chain. Registers the origin-bounded-redirect guard in the frozen runner inventory and syncs the red-team registry redirect scenario (bound registry credentials). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Refuse paths whose encoded delimiters survive the decode limit. - Require containment under both dot-segment/slash-collapse orders. - Compare non-ASCII hosts by exact text instead of an IDNA mapping. - Never apply ambient credentials on an anonymous cross-origin hop; marketplace metadata enrichment opts out of ambient credentials. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Python Architect | 0 | 1 | 0 | Single-owner discipline upheld; one pre-existing HTTPS check in the marketplace client duplicates the owner. |
| Test Coverage Expert | 0 | 0 | 0 | Every focus surface has unit and lifecycle regression traps; the guard has a mutation case. |
| Supply Chain Security Expert | 0 | 1 | 0 | Owner is well designed; the request-body hardening on anonymous cross-origin follow is now folded. |
| Auth Expert | 0 | 0 | 0 | AuthResolver and RegistryAuthContext remain the only credential sources; #3790 compatibility is clean. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Top 1 follow-ups
- [Python Architect] Consolidate the marketplace client's pre-existing check on the final URL's HTTPS scheme into the origin-bounded owner when the T1 egress-guard PR lands. -- Harmless defense in depth today, but a maintenance-cost duplicate once the egress guard covers every remaining redirect site.
Recommendation
Ship after CI is green. The only remaining follow-up, consolidating the duplicate HTTPS check, is explicitly deferred to the egress-guard PR. All four panelists converged, the hardening was folded with a regression test, and coverage has no gaps on any critical surface. Merging unblocks sibling #3790.
Full per-persona findings
Python Architect
- [recommended] The marketplace client keeps a redundant check on the final redirect URL's HTTPS scheme, which duplicates the owner's enforcement, at
src/apm_cli/marketplace/client.py:336
The owner already refuses an HTTPS downgrade, and the anonymous cross-origin path requires HTTPS. The second check cannot fire, but it obscures which component owns the decision.
Suggested: Remove it or turn it into an assertion when the egress guard lands. (Deferred: pre-existing, with its own test.)
Test Coverage Expert
No findings.
Supply Chain Security Expert
- [recommended] A request body is not cleared when anonymous cross-origin follow strips the credential headers, at
src/apm_cli/utils/http_credentials.py
No current caller combines the two. Making the owner reject the combination makes its contract explicit.
Suggested: Reject a body combined withfollow_cross_origin_without_credentials. (Folded at 37a9bb3, testtest_request_body_cannot_use_anonymous_cross_origin_follow.)
Auth Expert
No findings.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.
Re-verified the sweep's recorded findings on environment-variables, targets-matrix, install-packages, registries, registry-http-api and governance-guide against current main (after #3785, #3774, #3782, #3743, #3797 merged) and fixed what is still drifted. Refs #3800 Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


TL;DR
A registry behind a gateway that answers
307to the decoded, same-origin path (for example EnvoyUNESCAPE_AND_REDIRECTon%2F) now installs, replays and publishes instead of failing withregistry HTTP 307. One owner,origin_bounded_requestinutils/http_credentials.py, now makes every redirect follow decision for the registry client, the marketplace client and marketplace metadata enrichment. Any redirect outside the configured origin or base path is refused with a clear message.Fixes #3726
Note
Scope record: #3726 (comment) (0.36.1 auth track, PR 2 of 3). Reported with a working prototype by AKASH KUMAR (@ashky23), credited as co-author.
Description
RegistryClient._requestandfetch_from_urlpassedallow_redirects=Falseand raised on any>= 300, so a same-origin gateway307failed the install before the registry was reached ([BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect)Â #3726).publish_versionrelied on therequestsdefault redirect handling instead of an explicit origin policy.urllibdirectly instead of the shared credential-scoped HTTP owner.allow_redirects=False(registry),max_redirects = 5on the marketplace session, andurllibdefaults (builder).Why it matters: the repo rule is "Every durable decision ... has exactly ONE canonical owner", and the Registry HTTP API already says "Servers MUST decode percent-encoded path segments before lookup." A gateway doing that by redirect is legitimate when it stays on the registry origin.
Issue and approved scope
Issue: Fixes #3726
Human scope-approval comment: #3726 (comment)
This PR completes the issue (same-origin follow for
/versions, download, lockfile replay and publish; refusal outside the registry; gateway guidance in docs). The registry wire form for nested identities is out of scope per the record.Type of change
Approach
origin_bounded_request(session, method, url, *, base_url, headers, data, ...): every hopallow_redirects=False; follow 301/302/303/307/308 only whenorigin_bound_refusal(target, base_url)isNone; max 3 hops.%2e/%2f/%5c/%25,\â/,//collapse, IDNA-lowercased host, effective port; refuse userinfo, IPv6 zone IDs, non-HTTP(S) and HTTPSâHTTP.auth, so ambient.netrccannot replace the chosen credential whileHTTPS_PROXYand CA-bundle env stay honored.Proxy-AuthorizationjoinsCREDENTIAL_HEADER_NAMES./versions, download, lockfile replay, publish), the marketplace client_http_get, and builder metadata fetches through the owner.fetch_from_url's origin check moves into the owner.Implementation
src/apm_cli/utils/http_credentials.py: additive. Addsorigin_bounded_request,origin_bound_refusal,origin_root,origin_label,RedirectRefusedError(carriesreason,originand the unfollowed response; its message never echoes the target URL). ExistingCredentialScopedSessionstays as defense in depth.src/apm_cli/deps/registry/client.py: one_sendpath through the owner; default session is nowCredentialScopedSession. Refusals map toregistry redirected outside the configured registry (<origin>); refusing,... only 307 and 308 may redirect a PUT ..., or... more than 3 times .... Wire form unchanged;resolved_urlstaysarchive_url(...)(canonical, pre-redirect).src/apm_cli/marketplace/client.py:_http_getuses the owner withfollow_cross_origin_without_credentials=True, preserving today's anonymous cross-origin follow (credentials stripped) for marketplace URLs and the ADO sign-in page detection; credentials stay on the request origin. The session-levelmax_redirects = 5is gone.src/apm_cli/marketplace/builder.py:urllibreplaced by the owner on aCredentialScopedSession; strict origin bounds; failure causes mapped fromrequestserrors.transport-platform-origin-bounded-redirectin newscripts/architecture_linter/checks/transport_origin_bounded_redirect.py, registered oncredential-origin-scope(decision text extended with "redirect follow decision"). It requires the owner contract and consumer routing, forbidsallow_redirects=/.max_redirects =outside the owner except a frozen, reasoned, count-exact inventory (auth status probe, lifecycle SSRF dispatch, Artifactory boundary HEAD, policy fetches, version checker, archive), and forbidsurlopenin any module that builds a credential header. One sortedMutationCase.KEEP_UNCHANGED), registry-proxy troubleshooting row, registries guide pitfall,apm-usageauth skill line. CHANGELOG[Unreleased] Fixed.Diagrams
Legend: how one request moves through the owner; the only loop back is an in-bounds, method-safe hop under the hop limit.
flowchart LR A["registry client / marketplace client / builder"] --> B["origin_bounded_request: send hop, allow_redirects=False"] B --> C{"3xx with Location?"} C -->|"no"| R["return response"] C -->|"yes"| D{"origin_bound_refusal(target, base_url)"} D -->|"in bounds"| E{"307/308, or GET/HEAD?"} E -->|"yes, hops < 3"| B E -->|"no"| X["RedirectRefusedError"] D -->|"out of bounds"| X X --> M["registry redirected outside the configured registry; refusing"]Trade-offs
auth, nottrust_env=False.trust_env=Falsewould also dropHTTPS_PROXYandREQUESTS_CA_BUNDLE, breaking registries behind corporate proxies (the lifecycle tests themselves run throughHTTPS_PROXY). The no-opauthblocks only the netrc lookup.marketplace/client.pyonly.allow_redirects=sites (PR 3 migrates the frozen inventory).install --frozenregistry replay is reported as a frozen MCP manifest mismatch; the real download error is only visible with--verbose. That is install plumbing and is tracked separately by the auth track.Benefits
/versions, download,--frozenreplay and publish follow a same-origin gateway307.allow_redirects=or credentialedurlopenfailslint-architecture-boundaries.sh.Testing
Fail-before / pass-after evidence
tests/integration/test_registry_redirect_origin_lifecycle.py::test_registry_install_follows_same_origin_gateway_redirect--frozenreplay via same-origin 307...::test_registry_lock_replay_follows_same_origin_redirect_without_lock_churn[same-origin-307]...::test_publish_upload_never_leaves_registry_origin...::test_publish_upload_is_not_downgraded_by_302apm pack)tests/integration/test_marketplace_metadata_origin_lifecycle.pytests/unit/registry/test_client_redirects.pytests/unit/marketplace/test_builder_security.py::test_metadata_fetch_does_not_follow_redirect_to_another_origintests/unit/utils/test_http_credentials.pyNo live registry or gateway was used; behavior is proven with hermetic loopback servers and a recording HTTPS proxy. Lifecycle fixtures (
tests/utils/local_scripted_https_proxy.py, registry scenarios) come from the 0.36.1 red-team lane.Spec conformance (OpenAPM v0.1)
CONFORMANCE.{md,json}regeneratedapm-spec-waiverin the commit).How to test
apm experimental enable registries, then run the [BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect)Â #3726 repro (gw.py+id: gitlab.com/acme/infra-prompts): the server sees the encoded request, then the decoded one, and parsing proceeds.Locationto another host: install fails withregistry redirected outside the configured registry (http://127.0.0.1:8098); refusingand the other host gets no request.bash scripts/lint-architecture-boundaries.shpasses; addallow_redirects=Trueanywhere undersrc/apm_cli/and it fails withtransport-platform-origin-bounded-redirect.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com