Skip to content

fix(registry): follow redirects only within the configured registry origin - #3785

Merged
Daniel Meppiel (danielmeppiel) merged 9 commits into
mainfrom
danielmeppiel-0361-t1-pr2-redirect-owner
Nov 17, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 9 commits into
mainfrom
danielmeppiel-0361-t1-pr2-redirect-owner

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

TL;DR

A registry behind a gateway that answers 307 to the decoded, same-origin path (for example Envoy UNESCAPE_AND_REDIRECT on %2F) now installs, replays and publishes instead of failing with registry HTTP 307. One owner, origin_bounded_request in utils/http_credentials.py, now makes every redirect follow decision for the registry client, the marketplace client and marketplace metadata enrichment. Any redirect outside the configured origin or base path is refused with a clear message.

Fixes #3726

Note

Scope record: #3726 (comment) (0.36.1 auth track, PR 2 of 3). Reported with a working prototype by AKASH KUMAR (@ashky23), credited as co-author.

Description

  • RegistryClient._request and fetch_from_url passed allow_redirects=False and raised on any >= 300, so a same-origin gateway 307 failed the install before the registry was reached ([BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect) #3726).
  • publish_version relied on the requests default redirect handling instead of an explicit origin policy.
  • Marketplace metadata enrichment used urllib directly instead of the shared credential-scoped HTTP owner.
  • [!] The redirect decision had three owners: allow_redirects=False (registry), max_redirects = 5 on the marketplace session, and urllib defaults (builder).

Why it matters: the repo rule is "Every durable decision ... has exactly ONE canonical owner", and the Registry HTTP API already says "Servers MUST decode percent-encoded path segments before lookup." A gateway doing that by redirect is legitimate when it stays on the registry origin.

Issue and approved scope

Issue: Fixes #3726

Human scope-approval comment: #3726 (comment)

This PR completes the issue (same-origin follow for /versions, download, lockfile replay and publish; refusal outside the registry; gateway guidance in docs). The registry wire form for nested identities is out of scope per the record.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Approach

# Fix
1 New owner origin_bounded_request(session, method, url, *, base_url, headers, data, ...): every hop allow_redirects=False; follow 301/302/303/307/308 only when origin_bound_refusal(target, base_url) is None; max 3 hops.
2 Normalization before the containment check: RFC 3986 dot-segment removal, repeated decoding of %2e/%2f/%5c/%25, \ → /, // collapse, IDNA-lowercased host, effective port; refuse userinfo, IPv6 zone IDs, non-HTTP(S) and HTTPS→HTTP.
3 307/308 keep method and body; 301/302/303 only for GET/HEAD (a PUT is refused, never silently replayed as GET).
4 Credentials are re-sent only on in-bound hops. Credential-bearing hops pass an explicit no-op auth, so ambient .netrc cannot replace the chosen credential while HTTPS_PROXY and CA-bundle env stay honored. Proxy-Authorization joins CREDENTIAL_HEADER_NAMES.
5 Route the registry client (/versions, download, lockfile replay, publish), the marketplace client _http_get, and builder metadata fetches through the owner. fetch_from_url's origin check moves into the owner.

Implementation

  • src/apm_cli/utils/http_credentials.py: additive. Adds origin_bounded_request, origin_bound_refusal, origin_root, origin_label, RedirectRefusedError (carries reason, origin and the unfollowed response; its message never echoes the target URL). Existing CredentialScopedSession stays as defense in depth.
  • src/apm_cli/deps/registry/client.py: one _send path through the owner; default session is now CredentialScopedSession. Refusals map to registry redirected outside the configured registry (<origin>); refusing, ... only 307 and 308 may redirect a PUT ..., or ... more than 3 times .... Wire form unchanged; resolved_url stays archive_url(...) (canonical, pre-redirect).
  • src/apm_cli/marketplace/client.py: _http_get uses the owner with follow_cross_origin_without_credentials=True, preserving today's anonymous cross-origin follow (credentials stripped) for marketplace URLs and the ADO sign-in page detection; credentials stay on the request origin. The session-level max_redirects = 5 is gone.
  • src/apm_cli/marketplace/builder.py: urllib replaced by the owner on a CredentialScopedSession; strict origin bounds; failure causes mapped from requests errors.
  • Guard transport-platform-origin-bounded-redirect in new scripts/architecture_linter/checks/transport_origin_bounded_redirect.py, registered on credential-origin-scope (decision text extended with "redirect follow decision"). It requires the owner contract and consumer routing, forbids allow_redirects= / .max_redirects = outside the owner except a frozen, reasoned, count-exact inventory (auth status probe, lifecycle SSRF dispatch, Artifactory boundary HEAD, policy fetches, version checker, archive), and forbids urlopen in any module that builds a credential header. One sorted MutationCase.
  • Docs: Registry HTTP API §1.7 Redirects (gateway guidance: Envoy's default is KEEP_UNCHANGED), registry-proxy troubleshooting row, registries guide pitfall, apm-usage auth skill line. CHANGELOG [Unreleased] Fixed.

Diagrams

Legend: how one request moves through the owner; the only loop back is an in-bounds, method-safe hop under the hop limit.

flowchart LR
    A["registry client / marketplace client / builder"] --> B["origin_bounded_request: send hop, allow_redirects=False"]
    B --> C{"3xx with Location?"}
    C -->|"no"| R["return response"]
    C -->|"yes"| D{"origin_bound_refusal(target, base_url)"}
    D -->|"in bounds"| E{"307/308, or GET/HEAD?"}
    E -->|"yes, hops < 3"| B
    E -->|"no"| X["RedirectRefusedError"]
    D -->|"out of bounds"| X
    X --> M["registry redirected outside the configured registry; refusing"]
Loading

Trade-offs

  • Netrc suppression via explicit no-op auth, not trust_env=False. trust_env=False would also drop HTTPS_PROXY and REQUESTS_CA_BUNDLE, breaking registries behind corporate proxies (the lifecycle tests themselves run through HTTPS_PROXY). The no-op auth blocks only the netrc lookup.
  • Marketplace keeps an anonymous cross-origin follow mode. Marketplace URLs legitimately redirect to CDNs, and ADO sign-in detection depends on it; credentials are still stripped on those hops. The guard reserves this mode for marketplace/client.py only.
  • Hop limit 5 → 3 for marketplace. One consistent ceiling.
  • Out of scope: registry wire form for nested identities (D6), ADO / policy call sites (PR 1), remaining allow_redirects= sites (PR 3 migrates the frozen inventory).
  • Not fixed here: a failed install --frozen registry replay is reported as a frozen MCP manifest mismatch; the real download error is only visible with --verbose. That is install plumbing and is tracked separately by the auth track.

Benefits

  1. [BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect) #3726 repro installs: /versions, download, --frozen replay and publish follow a same-origin gateway 307.
  2. No credential, request or publish body reaches another origin or base path (lifecycle-proven against a recording HTTPS proxy).
  3. One owner + static guard: a new allow_redirects= or credentialed urlopen fails lint-architecture-boundaries.sh.

Testing

  • Tested locally
  • All existing tests pass (targeted suites below)
  • Added tests for new functionality
Fail-before / pass-after evidence
Scenario Test Base (v0.36.0 src) This PR
Install via same-origin gateway 307 (#3726) tests/integration/test_registry_redirect_origin_lifecycle.py::test_registry_install_follows_same_origin_gateway_redirect FAIL pass
--frozen replay via same-origin 307 ...::test_registry_lock_replay_follows_same_origin_redirect_without_lock_churn[same-origin-307] FAIL pass
Publish redirected to another origin ...::test_publish_upload_never_leaves_registry_origin FAIL pass
Publish answered with 302 ...::test_publish_upload_is_not_downgraded_by_302 FAIL pass
Metadata redirect keeps credential on origin (apm pack) tests/integration/test_marketplace_metadata_origin_lifecycle.py FAIL pass
Regression locks (cross-origin / outside-base refusal, publish 307 keeps PUT+body, direct replay) same file pass pass
Registry client hops (17 cases) tests/unit/registry/test_client_redirects.py 17 FAIL pass
Builder metadata redirect tests/unit/marketplace/test_builder_security.py::test_metadata_fetch_does_not_follow_redirect_to_another_origin FAIL pass
Owner table (normalization, method, hops, netrc) tests/unit/utils/test_http_credentials.py n/a (new owner) 36 pass
APM_E2E_TESTS=1 APM_BINARY_PATH=$PWD/.venv/bin/apm pytest -p no:xdist tests/integration/test_registry_redirect_origin_lifecycle.py   -> 8 passed
APM_BINARY_PATH=... pytest -p no:xdist tests/integration/test_marketplace_metadata_origin_lifecycle.py                           -> 1 passed
same two files on base src                                                                                                     -> 5 failed, 4 passed
pytest -p no:xdist tests/unit/registry tests/unit/utils/test_http_credentials.py                                               -> 318 passed, 2 xfailed
pytest -p no:xdist tests/unit/marketplace + registry/marketplace suites -> 2878 passed; 6 wave7 tests that patched the old session .get migrated to .request -> 7 passed
pytest -p no:xdist 16 marketplace/registry integration files (incl. binary lifecycle)                                          -> 547 passed (after test seam migration)
test_architecture_owner_rule_mutations.py [origin-bounded case + matrix coverage]                                              -> passed
uvx ruff@0.15.12 check / format --check; pylint R0801; lint-auth-signals.sh; lint-architecture-boundaries.sh                    -> clean
check_test_assertions.py / check_exact_test_duplicates.py                                                                       -> clean

No live registry or gateway was used; behavior is proven with hermetic loopback servers and a recording HTTPS proxy. Lifecycle fixtures (tests/utils/local_scripted_https_proxy.py, registry scenarios) come from the 0.36.1 red-team lane.

Spec conformance (OpenAPM v0.1)

  • Spec edit
  • Manifest edit
  • Test edit
  • CONFORMANCE.{md,json} regenerated
  • N/A -- this PR does not change OpenAPM-observable behaviour (client transport hardening; apm-spec-waiver in the commit).

How to test

  1. apm experimental enable registries, then run the [BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect) #3726 repro (gw.py + id: gitlab.com/acme/infra-prompts): the server sees the encoded request, then the decoded one, and parsing proceeds.
  2. Change the gateway Location to another host: install fails with registry redirected outside the configured registry (http://127.0.0.1:8098); refusing and the other host gets no request.
  3. bash scripts/lint-architecture-boundaries.sh passes; add allow_redirects=True anywhere under src/apm_cli/ and it fails with transport-platform-origin-bounded-redirect.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

â€Ķrigin

Add one origin-bounded redirect owner, origin_bounded_request, in
utils/http_credentials.py. Every hop is sent with allow_redirects=False;
301/302/303/307/308 are followed only to the same scheme, host and
effective port under the configured base path (RFC 3986 dot-segment,
%2e/%2f and '//' normalization, IDNA host compare, no userinfo, no
IPv6 zone IDs, no HTTPS downgrade), at most 3 hops. 307/308 keep the
method and body; 301/302/303 are followed only for GET and HEAD.
Credentials are re-sent only on in-bound hops, and credential-bearing
requests never pick up ambient .netrc credentials. Proxy-Authorization
joins the credential header set.

Route the registry client (/versions, archive download, lockfile
replay, publish) through the owner, so a gateway that answers 307 to
the decoded same-origin path now works. Registry publish now follows
redirects only within the configured registry origin. Marketplace
metadata fetches use the shared credential-scoped HTTP owner, and the
marketplace client session follow moves into the owner. The lockfile
keeps the canonical pre-redirect resolved_url.

Guard: transport-platform-origin-bounded-redirect (new checks module,
registered on credential-origin-scope) with a frozen, reasoned
inventory of sites that refuse every redirect or carry no credential.

apm-spec-waiver: redirect follow policy is transport hardening with no OpenAPM v0.1 requirement

Fixes #3726

Co-authored-by: AKASH KUMAR <33580834+ashky23@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 16:19
@danielmeppiel Daniel Meppiel (danielmeppiel) added this to the 0.36.1 milestone Nov 17, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ðŸŸĄ Changes recommended

Critical origin and path validation bypasses can expose registry credentials to unintended destinations, while supposedly anonymous requests can send ambient .netrc credentials and lifecycle tests fail to verify authenticated delivery.

5 open issues
What changed in this PR

Addresses #3726 by centralizing redirect handling so registry operations can follow same-origin gateway redirects while enforcing origin, base-path, method, and hop limits.

Changes:

  • Routes registry requests and marketplace fetches through a shared redirect owner.
  • Adds redirect regression tests, lifecycle fixtures, and an architecture guard.
  • Documents redirect behavior and gateway configuration guidance.
File Description
tests/​utils/​metadata_transport.py Adapts existing metadata mocks to the shared HTTP seam.
tests/​utils/​local_scripted_https_proxy.py Adds a recording HTTPS proxy for hermetic lifecycle tests.
tests/​unit/​utils/​test_http_credentials.py Tests redirect bounds, credentials, methods, and hop limits.
tests/​unit/​registry/​test_client_redirects.py Covers registry lookup, replay, and publish redirects.
tests/​unit/​marketplace/​test_metadata_enrichment.py Migrates metadata mocks to the new transport seam.
tests/​unit/​marketplace/​test_marketplace_commands.py Updates session doubles for shared-owner requests.
tests/​unit/​marketplace/​test_marketplace_client.py Updates mocks and tests credential handling across redirects.
tests/​unit/​marketplace/​test_builder.py Migrates builder metadata mocks.
tests/​unit/​marketplace/​test_builder_security.py Tests refusal of cross-origin metadata redirects.
tests/​integration/​test_wave7_policy_registry_coverage.py Updates marketplace request patch locations.
tests/​integration/​test_registry_redirect_origin_lifecycle.py Adds install, replay, and publish redirect scenarios.
tests/​integration/​test_marketplace_metadata_origin_lifecycle.py Adds metadata credential-isolation lifecycle coverage.
tests/​integration/​test_marketplace_metadata_auth_fallback.py Preserves fallback coverage through the new seam.
tests/​integration/​test_marketplace_builder_phase3w4.py Migrates builder transport mocks.
tests/​integration/​test_marketplace_builder_hermetic.py Updates hermetic metadata mocks.
tests/​integration/​test_architecture_owner_rule_mutations.py Adds a redirect-owner bypass mutation.
src/​apm_cli/​utils/​http_credentials.py Introduces bounded redirect handling and URL normalization.
src/​apm_cli/​marketplace/​client.py Uses the shared owner with anonymous cross-origin following.
src/​apm_cli/​marketplace/​builder.py Moves metadata fetching from urllib to the shared owner.
src/​apm_cli/​deps/​registry/​client.py Centralizes sending and maps redirect refusals to registry errors.
scripts/​architecture_linter/​groups/​transport_platform.py Registers the redirect guard.
scripts/​architecture_linter/​checks/​transport_origin_bounded_redirect.py Enforces shared-owner routing and explicit exceptions.
packages/​apm-guide/​.apm/​skills/​apm-usage/​authentication.md Documents registry redirect credential boundaries.
docs/​src/​content/​docs/​reference/​registry-http-api.md Defines redirect behavior and gateway guidance.
docs/​src/​content/​docs/​guides/​registries.md Explains redirects for encoded identities.
docs/​src/​content/​docs/​enterprise/​registry-proxy.md Adds redirect-refusal troubleshooting guidance.
CHANGELOG.md Records the redirect fix and metadata transport changes.
.apm/​architecture/​owners/​transport-auth-platform.json Assigns redirect decisions to the credential-origin owner.

Review effort: Balanced

Comment thread src/apm_cli/utils/http_credentials.py Resolved
Comment thread src/apm_cli/utils/http_credentials.py Resolved
Comment thread src/apm_cli/utils/http_credentials.py Resolved
Comment thread src/apm_cli/utils/http_credentials.py Resolved
â€Ķcross-origin follow

origin_bounded_request gains ambient_credentials=False so an anonymous
auth-chain step never picks up .netrc credentials. Marketplace _http_get
exposes follow_cross_origin_without_credentials per call so Azure DevOps
REST can hand an unfollowed sign-in redirect back to the auth chain.
Registers the origin-bounded-redirect guard in the frozen runner
inventory and syncs the red-team registry redirect scenario (bound
registry credentials).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Refuse paths whose encoded delimiters survive the decode limit.
- Require containment under both dot-segment/slash-collapse orders.
- Compare non-ASCII hosts by exact text instead of an IDNA mapping.
- Never apply ambient credentials on an anonymous cross-origin hop;
  marketplace metadata enrichment opts out of ambient credentials.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

APM Review Panel: ship_with_followups

Origin-bounded redirect owner keeps credential-bearing HTTP on the configured registry origin. A four-persona lean panel converged with zero blocking findings.

panel-mode=lean; personas=python-architect,test-coverage-expert,supply-chain-security-expert,auth-expert,apm-ceo

cc Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

All four panelists agree this PR is sound. The supply-chain-security-expert raised one recommended hardening: a request body combined with anonymous cross-origin follow. It was folded at 37a9bb3 with a regression test before synthesis. The auth-expert confirmed that AuthResolver and RegistryAuthContext remain the only credential sources, and that the PR is compatible with sibling #3790. The test-coverage-expert found no gaps. Same-origin 307 install, frozen replay, publish, refusal paths, method change, netrc exclusion, guard mutation and the canonical resolved_url all carry unit and lifecycle traps. Nine lifecycle scenarios pass against a local HTTPS proxy with hermetic stubs, and the lint mirror is green.

The python-architect confirmed the single-owner discipline and flagged one pre-existing duplicate: the marketplace client's own check on the final URL's HTTPS scheme. It predates this PR, has its own unit test and is harmless defense in depth, so it is deferred to the T1 egress-guard PR, which consolidates the remaining egress. All five Copilot inline threads (decode limit, dot/slash order, non-ASCII host comparison, ambient netrc, lifecycle credential binding) were fixed with regression tests and resolved.

Aligned with: Secure by default: credential-bearing redirects stay bounded to the configured registry origin with no opt-in, and HTTPS downgrade is refused. Governed by policy: guard transport-platform-origin-bounded-redirect enforces the owner with a frozen, count-exact exempt inventory.

Panel summary

Persona B R N Takeaway
Python Architect 0 1 0 Single-owner discipline upheld; one pre-existing HTTPS check in the marketplace client duplicates the owner.
Test Coverage Expert 0 0 0 Every focus surface has unit and lifecycle regression traps; the guard has a mutation case.
Supply Chain Security Expert 0 1 0 Owner is well designed; the request-body hardening on anonymous cross-origin follow is now folded.
Auth Expert 0 0 0 AuthResolver and RegistryAuthContext remain the only credential sources; #3790 compatibility is clean.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Top 1 follow-ups

  1. [Python Architect] Consolidate the marketplace client's pre-existing check on the final URL's HTTPS scheme into the origin-bounded owner when the T1 egress-guard PR lands. -- Harmless defense in depth today, but a maintenance-cost duplicate once the egress guard covers every remaining redirect site.

Recommendation

Ship after CI is green. The only remaining follow-up, consolidating the duplicate HTTPS check, is explicitly deferred to the egress-guard PR. All four panelists converged, the hardening was folded with a regression test, and coverage has no gaps on any critical surface. Merging unblocks sibling #3790.


Full per-persona findings

Python Architect

  • [recommended] The marketplace client keeps a redundant check on the final redirect URL's HTTPS scheme, which duplicates the owner's enforcement, at src/apm_cli/marketplace/client.py:336
    The owner already refuses an HTTPS downgrade, and the anonymous cross-origin path requires HTTPS. The second check cannot fire, but it obscures which component owns the decision.
    Suggested: Remove it or turn it into an assertion when the egress guard lands. (Deferred: pre-existing, with its own test.)

Test Coverage Expert

No findings.

Supply Chain Security Expert

  • [recommended] A request body is not cleared when anonymous cross-origin follow strips the credential headers, at src/apm_cli/utils/http_credentials.py
    No current caller combines the two. Making the owner reject the combination makes its contract explicit.
    Suggested: Reject a body combined with follow_cross_origin_without_credentials. (Folded at 37a9bb3, test test_request_body_cannot_use_anonymous_cross_origin_follow.)

Auth Expert

No findings.

This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.


Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 8d5df11 into main Nov 17, 2026
26 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-0361-t1-pr2-redirect-owner branch November 17, 2026 20:11
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 18, 2026
Re-verified the sweep's recorded findings on environment-variables,
targets-matrix, install-packages, registries, registry-http-api and
governance-guide against current main (after #3785, #3774, #3782,
#3743, #3797 merged) and fixed what is still drifted.

Refs #3800

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Registry client fails on a gateway 307 for percent-encoded identities (e.g. Envoy's default UnescapeAndRedirect)

2 participants