Repository navigation
fix(git): materialize package links from Git objects on every host - #3603
Daniel Meppiel (danielmeppiel) merged 12 commits into
Conversation
Dependency checkouts now pin core.symlinks=false, so Git writes each in-repo link as link text and never needs OS symlink capability. One owner, utils/git_sparse.materialize_tracked_symlinks, resolves every tracked mode-120000 entry in the package scope against the checked-out commit's tree (POSIX component-wise resolution, containment, 40-hop and directory-cycle bounds) and writes the target's checkout bytes and executable bit, or expands a directory link. The result is the tree a POSIX checkout yields when its links are followed, so content_hash is identical on Linux, macOS and Windows without Developer Mode, and --frozen replays a lockfile from any platform. Escaping, broken and cyclic links are rejected before hashing. Shared cache shards materialize leniently and the package consumer validates strictly; shards without the new apm.linkMaterialization marker heal on hit. Out-of-cone targets resolve from objects, so sparse cones no longer widen (#2707). The now-unreachable symlink capability diagnostic from #3545 is removed. Fixes #3457 apm-spec-waiver: Git checkout link handling is not an OpenAPM statement; materialized links reach req-sc-015 as regular, scanned files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶndows MAX_PATH Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
ðĄ Changes recommended
Unresolved findings include Windows path traversal enabling host-file overwrites, unbounded symlink expansion risking memory exhaustion, and incorrect symlink handling in path resolution and file transport.
4 open issues
What changed in this PR
This PR replaces OS-dependent Git symlinks with copies resolved from commit objects, addressing cross-platform package hash differences in #3457 and sparse-checkout link failures in #2707.
Changes:
- Pins Git to write link text, then centrally resolves and materializes file and directory links.
- Integrates materialization and validation into cache and downloader paths, including migration of older cache entries.
- Updates cross-platform tests, native Windows acceptance checks, architecture guards, and documentation.
| File | Description |
|---|---|
tests/âutils/âgit_link_fixture.py |
Adds portable Git-link fixtures and expected content hashes. |
tests/âunit/âutils/âtest_git_sparse.py |
Tests resolution, containment, cycles, and idempotence. |
tests/âunit/âtest_windows_native_symlink_workflow.py |
Updates native job names and timeout assertions. |
tests/âunit/âtest_windows_native_symlink_probe.py |
Updates lifecycle and writer-mutation assertions. |
tests/âunit/âinstall/âtest_symlink_capability_install.py |
Removes obsolete capability-error tests. |
tests/âunit/âdeps/âtest_shared_clone_cache.py |
Mocks materialization for object-free checkout tests. |
tests/âunit/âdeps/âtest_github_downloader_symlink_capability.py |
Removes obsolete capability-error routing tests. |
tests/âunit/âdeps/âtest_github_downloader_phase3.py |
Verifies sparse materialization delegation. |
tests/âunit/âdeps/âtest_github_downloader_error_handling.py |
Updates mocks and checkout signatures. |
tests/âunit/âdeps/âtest_bare_cache_sparse.py |
Verifies out-of-cone content without widening. |
tests/âunit/âcache/âtest_proxy_compat.py |
Updates cache materialization markers. |
tests/âunit/âcache/âtest_git_symlink_pin.py |
Verifies the link-text pin. |
tests/âunit/âcache/âtest_git_symlink_config.py |
Tests pin precedence and materialized content hashes. |
tests/âunit/âcache/âtest_git_cache.py |
Updates cache mocks and marker fixtures. |
tests/âunit/âcache/âtest_git_cache_symlinks.py |
Tests golden hashes, cache healing, and privilege independence. |
tests/âunit/âcache/âtest_git_cache_sparse.py |
Tests narrow cones and materialization-free cache hits. |
tests/âunit/âcache/âtest_git_cache_recency.py |
Removes tests for deleted cache-hit validation. |
tests/âunit/âcache/âtest_git_cache_hardening.py |
Updates valid-cache fixtures. |
tests/âtest_github_downloader.py |
Updates materialization mocks and checkout assertions. |
tests/âintegration/âtest_windows_native_symlink.py |
Requires successful materialization without symlink privileges. |
tests/âintegration/âtest_sparse_cone_symlink_repair.py |
Updates sparse-link and invalid-target regressions. |
tests/âintegration/âtest_git_object_links_lifecycle.py |
Adds install, cold-cache frozen replay, and rejection coverage. |
tests/âintegration/âtest_git_cache_hermetic.py |
Updates cache-hit marker fixtures. |
tests/âintegration/âtest_architecture_owner_rule_mutations.py |
Updates mutation intent wording. |
src/âapm_cli/âutils/âgit_sparse.py |
Implements Git-object link planning, writing, and validation. |
src/âapm_cli/âutils/âgit_env.py |
Pins link text and removes capability diagnostics. |
src/âapm_cli/âdeps/âgithub_downloader.py |
Integrates link materialization and consumer validation. |
src/âapm_cli/âdeps/âbare_cache.py |
Materializes links after shared-bare checkout. |
src/âapm_cli/âcache/âgit_cache.py |
Materializes staged checkouts and recognizes migration markers. |
scripts/âwindows_native_symlink_probe_entry.py |
Adds lifecycle coverage and writer mutation controls. |
scripts/âwindows_native_standard_user_symlink_gate.ps1 |
Extends the native acceptance timeout. |
scripts/âarchitecture_linter/âchecks/âtransport_sparse_and_updates.py |
Guards canonical materialization and validation routing. |
scripts/âarchitecture_linter/âchecks/âtransport_auth_platform.py |
Updates pin and materialization requirements. |
packages/âapm-guide/â.apm/âskills/âapm-usage/âdependencies.md |
Documents links installed as content. |
docs/âsrc/âcontent/âdocs/âcontributing/âintegration-testing.md |
Describes updated native Windows acceptance checks. |
docs/âsrc/âcontent/âdocs/âconsumer/âinstall-packages.md |
Documents privilege-free installs and invalid-link errors. |
CHANGELOG.md |
Adds the cross-platform materialization release note. |
.github/âworkflows/âci.yml |
Renames the native acceptance step and extends timeouts. |
.apm/âarchitecture/âowners/âtransport-auth-platform.json |
Updates the canonical link-handling responsibility. |
Review effort: Balanced
âĶect-symlink-materialization
âĶfetches Review follow-ups on the Git-object link owner: - a shared 100000-entry budget rejects acyclic fan-out before it grows exponentially; - expanded source and destination entry names must be plain portable paths (no "..", backslash, colon or NUL), so a POSIX-authored "..\\..\\x" cannot escape the checkout on Windows; - a component after a file, including "", "." and "..", is ENOTDIR; - link blobs load on demand, and out-of-tree reads take attributes from the commit (--attr-source=HEAD, Git 2.40+, with fallback); - the path: sparse file transport resolves a tracked link through the owner instead of returning its link text. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶect-symlink-materialization
âĶport contract Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶo Windows Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶ names Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶect-symlink-materialization
âĶfallback Panel follow-ups (test-coverage-expert): - the real-binary lifecycle now runs for a whole-repo and a subdirectory package whose links leave the package path; both pin a golden content_hash, replay with --frozen from a cold cache and reject escaping, broken and cyclic links before any lock is written; - a real-Git test drives download_subdirectory_package through the full-clone fallback and asserts the installed tree is link content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶect-symlink-materialization
APM Review Panel:
|
| Persona | B | R | N | Takeaway |
|---|---|---|---|---|
| Supply Chain Security Expert | 0 | 0 | 0 | Defense-in-depth containment: path guards, hop/write budgets, cycle detection, strict consumer rejection, Git-object provenance. No security regressions found. |
| Test Coverage Expert | 0 | 2 | 0 | Core logic and whole-repo lifecycle well covered; subdirectory lifecycle e2e and full-clone fallback were the gaps (both folded in e7256c3). |
| Python Architect | 0 | 0 | 0 | Single-owner link materialization through git_sparse.py; all checkout paths route through it; lenient/strict split linter-enforced; dead code removed. |
B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.
Recommendation
All three panelists converged and no findings remain open: both test-coverage items are folded with passing, mutation-checked tests. CI on the prior head was fully green, including Windows Compatibility Gate and Windows Native Symlink Acceptance. The fold commit adds tests only. Recommend merging once CI confirms green on the fold head. No follow-ups to track.
Full per-persona findings
Supply Chain Security Expert
No findings.
Test Coverage Expert
- [recommended] No subdirectory lifecycle e2e test verifying golden content_hash or --frozen cross-platform replay at
tests/integration/test_git_object_links_lifecycle.py
Subdirectory (monorepo) installs take the sparse checkout -> lenient materialize -> strict validate path; only whole-repo had an e2e golden-hash and --frozen test.
Suggested: Parametrize the lifecycle over a subdirectory package using the update-index --cacheinfo fixture approach.
Proof (missing at):tests/integration/test_git_object_links_lifecycle.py::test_subdir_links_install_as_content_and_frozen_replays_the_cross_platform_hash-- proves: apm install of a monorepo subdirectory package with links produces the same content_hash on every platform and --frozen replays the lock from a cold cache [portability-by-manifest,devx]
Folded: e7256c3 --test_links_install_as_content_and_frozen_replays_the_cross_platform_hash[subdirectory]pins sha256:53a8b0e1..., andtest_unresolvable_link_fails_before_hashingruns for both layouts. - [recommended] github_downloader subdir full-clone fallback path has mocked-only link materialization at
src/apm_cli/deps/github_downloader.py:1591
The fallbackmaterialize_tracked_symlinks(strict=False)call had no real-Git test.
Suggested: Drive download_subdirectory_package through the full-clone fallback with a real local bare repo.
Proof (missing at):tests/integration/test_sparse_cone_symlink_repair.py::test_full_clone_fallback_materializes_out_of_cone_links-- proves: when sparse checkout fails for a monorepo package with links, the full-clone fallback still installs link content [portability-by-manifest,devx]
Folded: e7256c3; removing the fallback call makes the test fail.
Python Architect
No findings.
This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.
Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.
Resolves conflicts with #3603 (links materialized from Git objects) by keeping both intents: core.symlinks=false plus the apm.linkMaterialization marker from #3603, and the core.autocrlf=false + core.eol=lf line-ending pins from #3270. Cache shards now require all four entries; the snapshot materializer replaces host symlink and line-ending entries; the git-child-environment needles cover both the link materializer and the line-ending owner. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#3603 resolves a tracked link whose target a sparse cone left out with `git cat-file --filters`, which applies checkout conversion. Callers such as the downloader's sparse fallback and git_file_transport pass no `-c` pins, so under a host `core.eol=crlf` (or the Windows native default) a link to a `text=auto` file was written with CRLF and its content_hash diverged across platforms, the #3270 failure on a new materialization path. The resolver now appends git_line_ending_pin_args to its own Git calls, so every caller gets the committed LF bytes. Regression: test_link_target_outside_sparse_cone_ignores_host_eol (global and env-count host core.eol=crlf; host-default control). Mutation-break: without the pin both crlf cases fail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After #3603 the shard checkout runs materialize_tracked_symlinks, which parses bytes; TestCreateCheckout's text-mode subprocess mocks fed it str and three cases failed on main as well, alongside a stale config-call count. The class fixture now stubs the resolver (covered by real-git tests) and the counts derive from _checkout_materialization_pins, so the line-ending and link pins are both accounted for. The bare spawn test returns bytes when the caller did not ask for text. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#3603's sparse-symlink rule required _LINK_MATERIALIZATION_PIN three times in git_cache.py (definition, persisted pins, cache-hit check). With the #3270 line-ending pins both the persisted list and the cache-hit check route through the single _checkout_materialization_pins(), so the marker now appears in its definition and that list, and the rule also requires the helper at its definition, persistence and validation sites. Dropping the marker from the list still reports a violation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶorepo-de Resolve git_cache.py against #3603: main replaced the sparse-widen finalizer with materialize_tracked_symlinks. Keep the #3071 completeness probe on cache hits and write-dedup, and run the post-checkout unmaterialized-path probe before link materialization so an empty promisor tree still raises the auth-classifiable IncompleteCheckoutError. Drop the recency test for the removed finalizer, as main did. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶatforms (#3599) * fix(git): pin core.eol=lf so text=auto package hashes match across platforms Under .gitattributes text / text=auto, Git writes the line ending that core.eol selects. Its default (native) is CRLF on Windows, and the #2971 core.autocrlf=false pin does not override it, so a package committed with LF was materialized with CRLF on Windows, recorded a different content_hash and failed `apm install --frozen` across platforms. - utils/git_env owns the line-ending pins (core.autocrlf=false and core.eol=lf) as -c args and in the network config snapshot, which replaces host autocrlf/eol entries like the core.symlinks pin. - GitCache, shared-bare materialization and the uncached fallback clone/checkout carry the pins; cache shards record them and older shards without core.eol are rematerialized on hit. - Explicit eol=lf/eol=crlf attributes, binary paths and committed CRLF blobs keep their repository-defined bytes. - Real-git component tests and installed-CLI lifecycle tests run in the windows_compat lane; docs describe the guarantee and the reviewed `apm install --update` recovery for locks with pre-fix CRLF hashes. Fixes #3270 apm-spec-waiver: remote package content_hash is reserved for OpenAPM v0.2; this fixes platform materialization only Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): note text=auto line-ending hash fix (#3599) apm-spec-waiver: remote package content_hash is reserved for OpenAPM v0.2; this fixes platform materialization only Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs: point stale CRLF lock recovery at the troubleshooting steps A plain `apm install --update` does not re-record a stale CRLF hash when the installed copy still matches the locked SHA (install/phases/integrate.py skips the re-download), and it re-resolves mutable refs. The packaged usage guide and the CHANGELOG promised a same-pin in-place refresh; both now point at the `Content hash mismatch` recovery steps, which require a fresh consumer or removal of the installed copy and review of the SHA and hash diff. install-packages.md links the lockfile content_hash definition instead of restating it, and notes that older cached checkouts are re-created on their next use. Addresses Copilot inline on packages/apm-guide/.apm/skills/apm-usage/ dependencies.md:97 and panel follow-ups (doc-writer, supply-chain, performance). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * perf(cache): persist bare consumer pins via clone --config materialize_from_bare runs once per consumer and is not amortized across installs. Writing the two line-ending pins with one `git config` spawn each added a process per dependency on every install (Windows pays the most per spawn). `git clone --config` records the same values in the new repository's config with no extra process; the checkout still carries the `-c` pins that outrank env snapshots. A comment notes why the symlink pin travels only as `-c` there. The rematerialization of a cache shard that lacks the pins now logs its elapsed time, so the one-time cost after upgrading is observable. The bare regression test asserts the consumer config records both pins; mutation-break: dropping the `--config` args fails it. Addresses panel follow-ups (performance-expert, python-architect). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(architecture): prove the line-ending pin guard has teeth The PR extends transport-platform-git-child-environment with needles for git_line_ending_pin_* in git_env.py, git_cache.py and bare_cache.py and a forbid scan for core.eol literals outside git_env.py, but no test proved those checks fire. Three source-override mutations now do: a core.eol literal in bare_cache.py, git_cache.py bypassing git_line_ending_pin_args, and the config snapshot no longer appending the line-ending pins. Each must report a violation with the matching message. Mutation-break: restoring the pre-PR linter check makes all three fail. Addresses panel follow-up (test-coverage-expert). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cache): trap extra git config spawns on the bare consumer path A mocked materialize_from_bare must carry the line-ending pins as clone --config arguments and spawn no separate `git config core.*` process, so a later change that reintroduces per-key spawns on this non-amortized path is a reviewed decision. Mutation-break: the previous per-key loop fails it. Addresses panel follow-up (performance-expert nit). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(git): pin line endings when resolving link targets from Git objects #3603 resolves a tracked link whose target a sparse cone left out with `git cat-file --filters`, which applies checkout conversion. Callers such as the downloader's sparse fallback and git_file_transport pass no `-c` pins, so under a host `core.eol=crlf` (or the Windows native default) a link to a `text=auto` file was written with CRLF and its content_hash diverged across platforms, the #3270 failure on a new materialization path. The resolver now appends git_line_ending_pin_args to its own Git calls, so every caller gets the committed LF bytes. Regression: test_link_target_outside_sparse_cone_ignores_host_eol (global and env-count host core.eol=crlf; host-default control). Mutation-break: without the pin both crlf cases fail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(cache): keep checkout mocks off the link resolver After #3603 the shard checkout runs materialize_tracked_symlinks, which parses bytes; TestCreateCheckout's text-mode subprocess mocks fed it str and three cases failed on main as well, alongside a stale config-call count. The class fixture now stubs the resolver (covered by real-git tests) and the counts derive from _checkout_materialization_pins, so the line-ending and link pins are both accounted for. The bare spawn test returns bytes when the caller did not ask for text. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(architecture): count the link marker through the shared pin list #3603's sparse-symlink rule required _LINK_MATERIALIZATION_PIN three times in git_cache.py (definition, persisted pins, cache-hit check). With the #3270 line-ending pins both the persisted list and the cache-hit check route through the single _checkout_materialization_pins(), so the marker now appears in its definition and that list, and the rule also requires the helper at its definition, persistence and validation sites. Dropping the marker from the list still reports a violation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * perf(cache): persist shard pins on the clone instead of config spawns GitCache wrote each materialization pin with its own `git config` spawn after cloning a shard; with the line-ending and link pins that became four extra processes per new or rematerialized shard. The pins now ride on the existing clone as `-c key=value`, which git clone records in the new repository, as bare_cache already does. The checkout keeps the `-c` pins from _safe_git_args. Hermetic call counts drop to clone plus checkout. Mutation-break: dropping the pins from the clone fails test_cached_checkout_ignores_host_eol (shard no longer recognized). Addresses delta-panel follow-up (performance-expert). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(architecture): prove the shard pin-list count guard has teeth The sparse-symlink rule now requires git_cache.py to route persisted and required pins through _checkout_materialization_pins(). A fourth source-override case inlines the pin tuple at the persistence site and expects the rule to report it. Mutation-break: main's count-only rule misses it. Addresses delta-panel follow-up (test-coverage-expert). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(changelog): place #3599 entry away from the section head Avoids landing-order conflicts with other queued PRs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore(changelog): place #3599 entry away from the section head Avoids landing-order conflicts with other queued PRs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
âĶ3673) * fix(git): check out out-of-cone link targets with Git's own checkout A link whose target a sparse checkout left out was read with `git --attr-source=HEAD cat-file --filters`, falling back to plain `cat-file --filters` when that call failed. cat-file reads no index, so in a sparse checkout the fallback silently skipped the target's .gitattributes and wrote e.g. an eol=crlf file with LF. That is the post-merge red on macOS x86_64 since 90e76bf: the path: transport link to shared/notes.crlf returned LF while the checked-out target was CRLF. Out-of-worktree targets are now written by `checkout-index` into a throwaway work tree from a throwaway full index (`read-tree HEAD`), in one batch per materialization. Attributes come from that index exactly as `git checkout` reads them, on every Git version, with no version-gated option and no silent fallback. In-worktree targets still read the bytes Git already wrote. The line-ending pins (#3599) still apply; the content_hash domain is unchanged. Regression: test_out_of_sparse_eol_target_keeps_checkout_bytes_without_attr_source emulates a Git that rejects --attr-source; it fails on main with the CI assertion and passes here. Also re-points two #3603 downloader tests at the persistent-checkout seam that #3640 changed underneath them. Refs #3457 apm-spec-waiver: Git checkout byte fidelity for link targets; no OpenAPM statement or hash-domain change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(changelog): reference #3673 on the link materialization entry Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(links): report an unwritten out-of-cone target with the diagnostic A target that checkout-index leaves unwritten raised a bare FileNotFoundError; it now raises the 'did not check out ... as a regular file' diagnostic before any link is written. Panel follow-up (test-coverage-expert). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(links): rebuild shards from the cat-file materializer; honor APM_TEMP_DIR Review follow-ups (Copilot): - The shard marker is versioned to apm.linkMaterialization=git-objects-checkout. A sparse shard pinned by the first Git-object materializer could hold an out-of-cone eol=crlf target with LF and would otherwise be served as is; it is now rematerialized on its next cache hit. Regression: test_cache_hit_rematerializes_shard_without_link_materialization[pre-checkout-marker] (fails if the marker value is reverted). - The throwaway checkout scratch directory uses get_apm_temp_dir(), like the downloader's own scratch space, so APM_TEMP_DIR / temp_dir keep working on hosts with a restricted system temp directory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore: restore uv.lock swept in by a local uv rewrite Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix(tests): restore release-gate lanes for 0.36.0 Main's tag-gate lanes (build-release push + manual-build-release) were red on runs 43381609442, 43381682871 and 43394173235. PR CI never runs them. Product regressions (Windows/real installs), fixed: - Hook cleanup removed nothing on Windows: remove_stale_deployed_files looked up recorded hashes with POSIX keys while callers passed native backslash keys, so content proof failed closed (#3672, Refs #3649). - describe_path_obstruction returned None on Windows, which reports a directory-for-file as EACCES and a file-for-parent as ENOENT instead of EISDIR/ENOTDIR (#3709, Refs #3678). - Skill preflight refused claimed members of an unclaimed, populated skill directory that materialization leaves alone, failing a local bundle install (Refs #3625). Stale tests (behaviour changed intentionally), updated: - policy chain mocks now return real PolicyFetchResult (#2933, Refs #2753) - bare-cache/downloader/anonymous-https fixtures use real Git checkouts (#3603, Refs #3457) - mcp install help, registry error wording, deps why -g help (#3633 Refs #3588, #2803 Refs #2478, #3614 Refs #2884) - downloader stub carries carried_licenses (#3646, Refs #3556) - frozen-hash lifecycle uses a generic git host so -g URL validation uses git ls-remote instead of a live api.github.com 404 (#3672) New Windows-shaped regression tests are marked windows_compat. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore: spec waiver for release-gate repair apm-spec-waiver: restores pre-existing behaviour on Windows and in skill preflight; no OpenAPM-observable change Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


Description
TL;DR
Packages with in-repo symlinks now install identically on every host without OS symlink capability. APM pins
core.symlinks=falseso Git writes links as link text, then one owner resolves each tracked link from the locked commit's Git objects and writes the target's content. Linux, macOS and non-admin Windows produce the same bytes and the samecontent_hashas today's POSIX result, soapm install --frozenreplays a lockfile from any platform. Escaping, broken and cyclic links fail before hashing.Note
Builds on #3545 (merged), which pinned
core.symlinks=trueand failed with a Developer Mode diagnostic. That diagnostic is now unreachable and removed: no host needs symlink privileges.Problem (WHY)
apm installfor any package containing links â e.g. skills.sh-authored packages that linkAGENTS.md -> CLAUDE.md.../../.claude-plugin/plugin.json), socontent_hashdiffered from POSIX and--frozenfailed ([BUG] Package content hash differs on Windows for packages with in-repo symlinks, so --frozen fails #3457).shutil.copytreeand never validated containment, so an escaping link could copy a host file intoapm_modules; the no-cache clone path kept OS symlinks, whichcompute_package_hashskips.Why these matter: the confirmed decision (Atlas
decisions/2026-11-12-contained-symlinks-are-content.md, scope record on #3457) is that a contained link is package content: "The same commit yields the same bytes and hash on Linux, macOS and Windows."Approach (WHAT)
core.symlinks=falseon every dependency Git child (git_env.git_symlink_pin_*); Git never creates an OS link.git_sparse.materialize_tracked_symlinks:ls-treethe package scope for mode120000; if none, stop (one subprocess). Otherwise resolve each link component-by-component in the commit tree (POSIX..semantics), bounded at 40 hops, with directory-cycle detection.git cat-file --filters HEAD:<path>for out-of-cone targets, soeol=crlf/filters match a POSIX checkout) and executable bit; expand directory links recursively.validate_materialized_symlinksstrictly before copy. Direct clones materialize strictly.apm.linkMaterialization=git-objects; shards without it (OS symlinks or link text from older APM) are rematerialized on hit, as forcore.autocrlf.Implementation (HOW)
src/apm_cli/utils/git_sparse.pyâ_LinkResolver,materialize_tracked_symlinks(strict/lenient),validate_materialized_symlinks(read-only, same diagnostics). Replacesrepair_dangling_cone_symlinks; writes refuse linked parent directories (has_symlink_component).src/apm_cli/utils/git_env.pyâ pin valuefalse;GitSymlinkCapabilityError/raise_for_symlink_capabilityand thepackage=plumbing removed (no longer reachable).src/apm_cli/cache/git_cache.pyâ materialize beforeatomic_land; heal marker; hits do no link work.src/apm_cli/deps/bare_cache.py,deps/github_downloader.pyâ materialize after every checkout (shared bare, legacy sparse, legacy full clone, whole-repo clone); whole-repo cache copies validate first (a failure falls back to the strict clone path, which raises).transport_sparse_and_updates.py,transport_auth_platform.py, owner registry) re-pointed to the new owner and pin value.ci.yml, probe,test_windows_native_symlink.py) â the standard-user case now expects success with the POSIX hash; the negative control disables the link writer; the baseline also runs the real-binary--frozenlifecycle. Timeouts raised for that lifecycle.consumer/install-packages.md,apm-guidedependencies.md,contributing/integration-testing.md, CHANGELOG.Diagram
Legend: every checkout path writes link text, hands the package scope to the one owner, and the consumer validates before hashing; boxed nodes are new.
sequenceDiagram participant DL as GitHubPackageDownloader participant AR as AuthResolver participant GC as GitCache participant G as git (< 2.36) DL->>AR: try_with_fallback(unauth_first) AR->>GC: get_checkout(env without token) GC->>G: clone --local from cached __p bare GC->>G: checkout SHA (lazy blob fetch rejected) G-->>GC: exit 0, worktree empty rect rgb(230, 245, 255) GC->>G: ls-files --deleted -z G-->>GC: packages/pkg-shared/apm.yml ... GC-->>AR: raise IncompleteCheckoutError (stderr kept) end alt before this PR GC-->>DL: empty shard landed and cached Note over DL: validate fails - Not a valid APM package else after this PR AR->>AR: is_public_github_auth_failure -> true AR->>GC: get_checkout(env with token) GC-->>DL: complete shard, package validates endTrade-offs
copytreeresult and ecosystem practice (vercel-labs/skillscopyDirectory). Chose this over preserving links, which needs OS privilege and is excluded fromcontent_hash.apm marketplacereads.cat-file --filtersfallback.git cat-file --batch --filtersreports the unfiltered size (Git quirk observed on 2.56), so out-of-cone targets are read one call each; in-cone targets read the working tree.copytreeand escaping links copied host files.Benefits
content_hashper fixture across Linux/macOS/Windows (GOLDEN_PKG_HASH, lifecycle_GOLDEN_CONTENT_HASH).--frozenfrom a cold cache replays a lock recorded on any platform.ls-tree.Issue and approved scope
Issue: Fixes #3457
Human scope-approval comment: #3457 (comment) (extended scope, Daniel Meppiel (@danielmeppiel), 2026-11-12)
Does this PR complete the issue, or what remains? Completes the extended scope. Out of scope per the record: source-discovery link following (#2913, #2802), archive/proxy links (#3110), user Git config,
content_hashalgorithm, line endings (#3270, #3368).Type of change
Testing
Validation
Local host: macOS (arm64), Git 2.56.0. Targeted runs only, one process at a time; CI is the authority for Linux, Windows
windows_compatand the native standard-user job.ruff check/format --check, pylint R0801,lint-auth-signals.sh,lint_architecture_boundaries.py --root ., file length,relative_to)tests/unit/cache/test_git_cache_symlinks.py,test_git_symlink_config.py,test_git_symlink_pin.pytests/unit/utils/test_git_sparse.pytests/unit/deps/test_github_downloader_phase3.py,test_github_downloader_error_handling.py,tests/test_github_downloader.pytests/unit/cache/test_git_cache{,_recency,_sparse,_hardening}.py,test_proxy_compat.py,tests/unit/deps/test_{bare_cache_sparse,shared_clone_cache}.pytests/integration/test_sparse_cone_symlink_repair.pytests/integration/test_git_object_links_lifecycle.py(realapmbinary)--frozenreplay, escaping/broken/cyclic rejectiontests/unit/test_windows_native_symlink_{probe,workflow}.pytest_git_sparse.py(shared 100000-entry budget, Windows-unsafe entry names, ENOTDIR) +tests/unit/deps/test_git_file_transport_links.py(path:fetches) + transport suites (test_gitlab_git_transport.py, strategy selection, GitLab sparse contract)Cross-platform hash evidence. Fixture
tests/utils/git_link_fixture.pypinsGOLDEN_PKG_HASH = sha256:b735a61aâĶ587fb9: on macOS the POSIX baseline (real symlinks +copytree) and the new link-text + Git-object path both produce it, full and sparse. The lifecycle pinssha256:5dc1b14fâĶ62d63f, equal to an independently built expected tree. The same assertions run on Linux (unit lane) and Windows (windows_compat+ native standard-user job), so a green CI is the Linux/Windows half of the proof.Regression trap. The new
test_git_cache_symlinks.pyagainstorigin/main: 8 failed, 2 passed (the two passes are the POSIX baseline and a pin-shape check). Onmaina sparse checkout even rejects a contained directory link (Symlink 'pkg/shared' targets 'shared', which is not a tracked file).Scenario Evidence
tests/unit/cache/test_git_cache_symlinks.py::test_cached_checkout_materializes_links_with_golden_hash,tests/unit/utils/test_git_sparse.py::test_contained_links_become_posix_contentwindows_compat)test_git_cache_symlinks.py::test_posix_follow_links_baseline_is_the_golden_hashapm installthenapm install --frozenfrom a cold cache replays the locktests/integration/test_git_object_links_lifecycle.py::test_links_install_as_content_and_frozen_replays_the_cross_platform_hashtests/integration/test_windows_native_symlink.py::test_native_standard_user_materializes_links(+ lifecycle in baseline), Windows Native Symlink Acceptance jobtest_git_object_links_lifecycle.py::test_unresolvable_link_fails_before_hashing,test_git_sparse.py::test_unresolvable_links_are_rejected,::test_cycles_are_bounded,::test_never_writes_through_a_linked_directorytest_git_cache_symlinks.py::test_cache_hit_rematerializes_shard_without_link_materialization,test_git_sparse.py::test_package_without_links_costs_one_ls_treeHow to test
uv run --extra dev pytest tests/unit/utils/test_git_sparse.py tests/unit/cache/test_git_cache_symlinks.pyâ all pass; golden hash asserted.APM_BINARY_PATH=.venv/bin/apm uv run --extra dev pytest tests/integration/test_git_object_links_lifecycle.pyâ install, cold--frozenreplay, and three rejection cases pass.main, the new tests fail (link text / capability error instead of content).Spec conformance (OpenAPM v0.1)
CONFORMANCE.{md,json}regeneratedCo-authored-by: Copilot 223556219+Copilot@users.noreply.github.com