Skip to content

fix(git): materialize package links from Git objects on every host - #3603

Merged
Daniel Meppiel (danielmeppiel) merged 12 commits into
mainfrom
danielmeppiel-git-object-symlink-materialization
Nov 12, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 12 commits into
mainfrom
danielmeppiel-git-object-symlink-materialization

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

TL;DR

Packages with in-repo symlinks now install identically on every host without OS symlink capability. APM pins core.symlinks=false so Git writes links as link text, then one owner resolves each tracked link from the locked commit's Git objects and writes the target's content. Linux, macOS and non-admin Windows produce the same bytes and the same content_hash as today's POSIX result, so apm install --frozen replays a lockfile from any platform. Escaping, broken and cyclic links fail before hashing.

Note

Builds on #3545 (merged), which pinned core.symlinks=true and failed with a Developer Mode diagnostic. That diagnostic is now unreachable and removed: no host needs symlink privileges.

Problem (WHY)

Why these matter: the confirmed decision (Atlas decisions/2026-11-12-contained-symlinks-are-content.md, scope record on #3457) is that a contained link is package content: "The same commit yields the same bytes and hash on Linux, macOS and Windows."

Approach (WHAT)

# Fix
1 Pin core.symlinks=false on every dependency Git child (git_env.git_symlink_pin_*); Git never creates an OS link.
2 New owner git_sparse.materialize_tracked_symlinks: ls-tree the package scope for mode 120000; if none, stop (one subprocess). Otherwise resolve each link component-by-component in the commit tree (POSIX .. semantics), bounded at 40 hops, with directory-cycle detection.
3 Write the target's checkout bytes (the working-tree file Git wrote, else git cat-file --filters HEAD:<path> for out-of-cone targets, so eol=crlf/filters match a POSIX checkout) and executable bit; expand directory links recursively.
4 Shared cache shards materialize leniently (marketplace reads stay unaffected); the package consumer runs validate_materialized_symlinks strictly before copy. Direct clones materialize strictly.
5 Shards persist apm.linkMaterialization=git-objects; shards without it (OS symlinks or link text from older APM) are rematerialized on hit, as for core.autocrlf.

Implementation (HOW)

  • src/apm_cli/utils/git_sparse.py — _LinkResolver, materialize_tracked_symlinks (strict/lenient), validate_materialized_symlinks (read-only, same diagnostics). Replaces repair_dangling_cone_symlinks; writes refuse linked parent directories (has_symlink_component).
  • src/apm_cli/utils/git_env.py — pin value false; GitSymlinkCapabilityError / raise_for_symlink_capability and the package= plumbing removed (no longer reachable).
  • src/apm_cli/cache/git_cache.py — materialize before atomic_land; heal marker; hits do no link work.
  • src/apm_cli/deps/bare_cache.py, deps/github_downloader.py — materialize after every checkout (shared bare, legacy sparse, legacy full clone, whole-repo clone); whole-repo cache copies validate first (a failure falls back to the strict clone path, which raises).
  • Architecture guards (transport_sparse_and_updates.py, transport_auth_platform.py, owner registry) re-pointed to the new owner and pin value.
  • Windows native job (ci.yml, probe, test_windows_native_symlink.py) — the standard-user case now expects success with the POSIX hash; the negative control disables the link writer; the baseline also runs the real-binary --frozen lifecycle. Timeouts raised for that lifecycle.
  • Docs — consumer/install-packages.md, apm-guide dependencies.md, contributing/integration-testing.md, CHANGELOG.

Diagram

Legend: every checkout path writes link text, hands the package scope to the one owner, and the consumer validates before hashing; boxed nodes are new.

sequenceDiagram
    participant DL as GitHubPackageDownloader
    participant AR as AuthResolver
    participant GC as GitCache
    participant G as git (< 2.36)
    DL->>AR: try_with_fallback(unauth_first)
    AR->>GC: get_checkout(env without token)
    GC->>G: clone --local from cached __p bare
    GC->>G: checkout SHA (lazy blob fetch rejected)
    G-->>GC: exit 0, worktree empty
    rect rgb(230, 245, 255)
    GC->>G: ls-files --deleted -z
    G-->>GC: packages/pkg-shared/apm.yml ...
    GC-->>AR: raise IncompleteCheckoutError (stderr kept)
    end
    alt before this PR
        GC-->>DL: empty shard landed and cached
        Note over DL: validate fails - Not a valid APM package
    else after this PR
        AR->>AR: is_public_github_auth_failure -> true
        AR->>GC: get_checkout(env with token)
        GC-->>DL: complete shard, package validates
    end
Loading

Trade-offs

  • Copy, not link. Contained links install as copies (directory links duplicate bytes), matching today's POSIX copytree result and ecosystem practice (vercel-labs/skills copyDirectory). Chose this over preserving links, which needs OS privilege and is excluded from content_hash.
  • Lenient shard + strict consumer. A shard may serve a marketplace read or another package path, so an unresolvable link there stays link text; the package consumer rejects it. Rejected strict shards because one broken link anywhere in a marketplace repo would break apm marketplace reads.
  • Per-file cat-file --filters fallback. git cat-file --batch --filters reports the unfiltered size (Git quirk observed on 2.56), so out-of-cone targets are read one call each; in-cone targets read the working tree.
  • Whole-repo installs now validate links. A repo with an escaping or broken link anywhere now fails; before, broken links already failed copytree and escaping links copied host files.
  • Diagnostic from fix(git): pin core.symlinks=true so package hashes match across platforms #3545 removed rather than kept as dead code.

Benefits

  1. Non-admin Windows installs link-containing packages (native job asserts success, error 1314 host).
  2. One pinned content_hash per fixture across Linux/macOS/Windows (GOLDEN_PKG_HASH, lifecycle _GOLDEN_CONTENT_HASH).
  3. --frozen from a cold cache replays a lock recorded on any platform.
  4. Sparse cones no longer widen for out-of-cone links; packages without links cost one ls-tree.
  5. Escaping links can no longer copy host files in whole-repo installs.

Issue and approved scope

Issue: Fixes #3457

Human scope-approval comment: #3457 (comment) (extended scope, Daniel Meppiel (@danielmeppiel), 2026-11-12)

Does this PR complete the issue, or what remains? Completes the extended scope. Out of scope per the record: source-discovery link following (#2913, #2802), archive/proxy links (#3110), user Git config, content_hash algorithm, line endings (#3270, #3368).

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

Validation

Local host: macOS (arm64), Git 2.56.0. Targeted runs only, one process at a time; CI is the authority for Linux, Windows windows_compat and the native standard-user job.

Check Result
Lint mirror (ruff check/format --check, pylint R0801, lint-auth-signals.sh, lint_architecture_boundaries.py --root ., file length, relative_to) all clean
tests/unit/cache/test_git_cache_symlinks.py, test_git_symlink_config.py, test_git_symlink_pin.py 32 passed
tests/unit/utils/test_git_sparse.py 18 passed
tests/unit/deps/test_github_downloader_phase3.py, test_github_downloader_error_handling.py, tests/test_github_downloader.py 381 passed, 2 skipped
tests/unit/cache/test_git_cache{,_recency,_sparse,_hardening}.py, test_proxy_compat.py, tests/unit/deps/test_{bare_cache_sparse,shared_clone_cache}.py all passed
tests/integration/test_sparse_cone_symlink_repair.py 7 passed
tests/integration/test_git_object_links_lifecycle.py (real apm binary) 4 passed — install, cold-cache --frozen replay, escaping/broken/cyclic rejection
tests/unit/test_windows_native_symlink_{probe,workflow}.py 38 passed
Architecture mutation cases (sparse-symlink + git-child-environment guards) 6 passed
Review follow-ups: test_git_sparse.py (shared 100000-entry budget, Windows-unsafe entry names, ENOTDIR) + tests/unit/deps/test_git_file_transport_links.py (path: fetches) + transport suites (test_gitlab_git_transport.py, strategy selection, GitLab sparse contract) all passed; CI green incl. Windows Compatibility Gate

Cross-platform hash evidence. Fixture tests/utils/git_link_fixture.py pins GOLDEN_PKG_HASH = sha256:b735a61aâ€Ķ587fb9: on macOS the POSIX baseline (real symlinks + copytree) and the new link-text + Git-object path both produce it, full and sparse. The lifecycle pins sha256:5dc1b14fâ€Ķ62d63f, equal to an independently built expected tree. The same assertions run on Linux (unit lane) and Windows (windows_compat + native standard-user job), so a green CI is the Linux/Windows half of the proof.

Regression trap. The new test_git_cache_symlinks.py against origin/main: 8 failed, 2 passed (the two passes are the POSIX baseline and a pin-shape check). On main a sparse checkout even rejects a contained directory link (Symlink 'pkg/shared' targets 'shared', which is not a tracked file).

Scenario Evidence

# Scenario (user promise) Principle(s) Test(s) proving it Type
1 A package with file/dir/nested/chained links installs as content with the same hash on every OS Portability, Governed by policy tests/unit/cache/test_git_cache_symlinks.py::test_cached_checkout_materializes_links_with_golden_hash, tests/unit/utils/test_git_sparse.py::test_contained_links_become_posix_content integration (real git, windows_compat)
2 That golden hash equals today's POSIX follow-links result Governed by policy test_git_cache_symlinks.py::test_posix_follow_links_baseline_is_the_golden_hash integration
3 apm install then apm install --frozen from a cold cache replays the lock Governed by policy, DevX tests/integration/test_git_object_links_lifecycle.py::test_links_install_as_content_and_frozen_replays_the_cross_platform_hash e2e (real binary)
4 Non-admin Windows installs without symlink privilege Portability, DevX tests/integration/test_windows_native_symlink.py::test_native_standard_user_materializes_links (+ lifecycle in baseline), Windows Native Symlink Acceptance job e2e (native)
5 Escaping, broken, cyclic, oversized or host-unsafe links fail before hashing, no lock written Secure by default test_git_object_links_lifecycle.py::test_unresolvable_link_fails_before_hashing, test_git_sparse.py::test_unresolvable_links_are_rejected, ::test_cycles_are_bounded, ::test_never_writes_through_a_linked_directory e2e / integration
6 Old cache shards heal; packages without links unchanged DevX test_git_cache_symlinks.py::test_cache_hit_rematerializes_shard_without_link_materialization, test_git_sparse.py::test_package_without_links_costs_one_ls_tree integration

How to test

  • uv run --extra dev pytest tests/unit/utils/test_git_sparse.py tests/unit/cache/test_git_cache_symlinks.py → all pass; golden hash asserted.
  • APM_BINARY_PATH=.venv/bin/apm uv run --extra dev pytest tests/integration/test_git_object_links_lifecycle.py → install, cold --frozen replay, and three rejection cases pass.
  • Check the Windows Native Symlink Acceptance job: standard user, error 1314, native case passes, writer mutation fails, restored pass.
  • On main, the new tests fail (link text / capability error instead of content).

Spec conformance (OpenAPM v0.1)

  • Spec edit
  • Manifest edit
  • Test edit
  • CONFORMANCE.{md,json} regenerated
  • N/A -- this PR does not change OpenAPM-observable behaviour. Git checkout link handling is not an OpenAPM statement; materialized links reach [req-sc-015] as regular, scanned files (Mode B waiver in the commit message).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Dependency checkouts now pin core.symlinks=false, so Git writes each
in-repo link as link text and never needs OS symlink capability. One
owner, utils/git_sparse.materialize_tracked_symlinks, resolves every
tracked mode-120000 entry in the package scope against the checked-out
commit's tree (POSIX component-wise resolution, containment, 40-hop and
directory-cycle bounds) and writes the target's checkout bytes and
executable bit, or expands a directory link. The result is the tree a
POSIX checkout yields when its links are followed, so content_hash is
identical on Linux, macOS and Windows without Developer Mode, and
--frozen replays a lockfile from any platform.

Escaping, broken and cyclic links are rejected before hashing. Shared
cache shards materialize leniently and the package consumer validates
strictly; shards without the new apm.linkMaterialization marker heal
on hit. Out-of-cone targets resolve from objects, so sparse cones no
longer widen (#2707). The now-unreachable symlink capability diagnostic
from #3545 is removed.

Fixes #3457

apm-spec-waiver: Git checkout link handling is not an OpenAPM statement; materialized links reach req-sc-015 as regular, scanned files.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 12, 2026 12:06
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
â€Ķndows MAX_PATH

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ðŸŸĄ Changes recommended

Unresolved findings include Windows path traversal enabling host-file overwrites, unbounded symlink expansion risking memory exhaustion, and incorrect symlink handling in path resolution and file transport.

4 open issues
What changed in this PR

This PR replaces OS-dependent Git symlinks with copies resolved from commit objects, addressing cross-platform package hash differences in #3457 and sparse-checkout link failures in #2707.

Changes:

  • Pins Git to write link text, then centrally resolves and materializes file and directory links.
  • Integrates materialization and validation into cache and downloader paths, including migration of older cache entries.
  • Updates cross-platform tests, native Windows acceptance checks, architecture guards, and documentation.
File Description
tests/​utils/​git_link_fixture.py Adds portable Git-link fixtures and expected content hashes.
tests/​unit/​utils/​test_git_sparse.py Tests resolution, containment, cycles, and idempotence.
tests/​unit/​test_windows_native_symlink_workflow.py Updates native job names and timeout assertions.
tests/​unit/​test_windows_native_symlink_probe.py Updates lifecycle and writer-mutation assertions.
tests/​unit/​install/​test_symlink_capability_install.py Removes obsolete capability-error tests.
tests/​unit/​deps/​test_shared_clone_cache.py Mocks materialization for object-free checkout tests.
tests/​unit/​deps/​test_github_downloader_symlink_capability.py Removes obsolete capability-error routing tests.
tests/​unit/​deps/​test_github_downloader_phase3.py Verifies sparse materialization delegation.
tests/​unit/​deps/​test_github_downloader_error_handling.py Updates mocks and checkout signatures.
tests/​unit/​deps/​test_bare_cache_sparse.py Verifies out-of-cone content without widening.
tests/​unit/​cache/​test_proxy_compat.py Updates cache materialization markers.
tests/​unit/​cache/​test_git_symlink_pin.py Verifies the link-text pin.
tests/​unit/​cache/​test_git_symlink_config.py Tests pin precedence and materialized content hashes.
tests/​unit/​cache/​test_git_cache.py Updates cache mocks and marker fixtures.
tests/​unit/​cache/​test_git_cache_symlinks.py Tests golden hashes, cache healing, and privilege independence.
tests/​unit/​cache/​test_git_cache_sparse.py Tests narrow cones and materialization-free cache hits.
tests/​unit/​cache/​test_git_cache_recency.py Removes tests for deleted cache-hit validation.
tests/​unit/​cache/​test_git_cache_hardening.py Updates valid-cache fixtures.
tests/​test_github_downloader.py Updates materialization mocks and checkout assertions.
tests/​integration/​test_windows_native_symlink.py Requires successful materialization without symlink privileges.
tests/​integration/​test_sparse_cone_symlink_repair.py Updates sparse-link and invalid-target regressions.
tests/​integration/​test_git_object_links_lifecycle.py Adds install, cold-cache frozen replay, and rejection coverage.
tests/​integration/​test_git_cache_hermetic.py Updates cache-hit marker fixtures.
tests/​integration/​test_architecture_owner_rule_mutations.py Updates mutation intent wording.
src/​apm_cli/​utils/​git_sparse.py Implements Git-object link planning, writing, and validation.
src/​apm_cli/​utils/​git_env.py Pins link text and removes capability diagnostics.
src/​apm_cli/​deps/​github_downloader.py Integrates link materialization and consumer validation.
src/​apm_cli/​deps/​bare_cache.py Materializes links after shared-bare checkout.
src/​apm_cli/​cache/​git_cache.py Materializes staged checkouts and recognizes migration markers.
scripts/​windows_native_symlink_probe_entry.py Adds lifecycle coverage and writer mutation controls.
scripts/​windows_native_standard_user_symlink_gate.ps1 Extends the native acceptance timeout.
scripts/​architecture_linter/​checks/​transport_sparse_and_updates.py Guards canonical materialization and validation routing.
scripts/​architecture_linter/​checks/​transport_auth_platform.py Updates pin and materialization requirements.
packages/​apm-guide/​.apm/​skills/​apm-usage/​dependencies.md Documents links installed as content.
docs/​src/​content/​docs/​contributing/​integration-testing.md Describes updated native Windows acceptance checks.
docs/​src/​content/​docs/​consumer/​install-packages.md Documents privilege-free installs and invalid-link errors.
CHANGELOG.md Adds the cross-platform materialization release note.
.github/​workflows/​ci.yml Renames the native acceptance step and extends timeouts.
.apm/​architecture/​owners/​transport-auth-platform.json Updates the canonical link-handling responsibility.

Review effort: Balanced

Comment thread src/apm_cli/utils/git_sparse.py Resolved
Comment thread src/apm_cli/utils/git_sparse.py Resolved
Comment thread src/apm_cli/utils/git_env.py Resolved
Comment thread src/apm_cli/utils/git_sparse.py Resolved
â€Ķfetches

Review follow-ups on the Git-object link owner:
- a shared 100000-entry budget rejects acyclic fan-out before it grows
  exponentially;
- expanded source and destination entry names must be plain portable
  paths (no "..", backslash, colon or NUL), so a POSIX-authored
  "..\\..\\x" cannot escape the checkout on Windows;
- a component after a file, including "", "." and "..", is ENOTDIR;
- link blobs load on demand, and out-of-tree reads take attributes from
  the commit (--attr-source=HEAD, Git 2.40+, with fallback);
- the path: sparse file transport resolves a tracked link through the
  owner instead of returning its link text.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
â€Ķport contract

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
â€Ķo Windows

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
â€Ķ names

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
â€Ķfallback

Panel follow-ups (test-coverage-expert):
- the real-binary lifecycle now runs for a whole-repo and a subdirectory
  package whose links leave the package path; both pin a golden
  content_hash, replay with --frozen from a cold cache and reject
  escaping, broken and cyclic links before any lock is written;
- a real-Git test drives download_subdirectory_package through the
  full-clone fallback and asserts the installed tree is link content.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@danielmeppiel

Copy link
Copy Markdown
Collaborator Author

APM Review Panel: ship_now

Removes the last OS-privilege dependency in APM's install path by materializing Git symlinks from commit objects, so every host -- Windows non-admin, containers, CI -- records the identical content_hash.

panel-mode=lean; personas=supply-chain-security-expert,test-coverage-expert,python-architect,apm-ceo

cc Daniel Meppiel (@danielmeppiel) Sergio Sisternes (@sergio-sisternes-epam) -- a fresh advisory pass is ready for your review.

All three panelists converged with no dissent. Supply-chain security found no regressions and highlighted defense-in-depth containment: path guards, hop and write budgets, cycle detection, strict consumer rejection and Git-object provenance. The architect confirmed one owner (utils/git_sparse.py) for link materialization; every checkout path routes through it, the lenient-shard / strict-consumer split is coherent and enforced by the architecture linter, and the #3545 capability diagnostic is fully removed. Test coverage raised two recommended gaps: a subdirectory (monorepo) e2e with golden hash and --frozen replay, and real-Git coverage of the full-clone fallback. The author folded both in e7256c3; the fallback test is mutation-break verified.

Strategically this removes the last platform-specific requirement from install: Windows users no longer need admin elevation or Developer Mode for packages with links, and a lockfile recorded on any OS replays with --frozen everywhere. The four earlier Copilot reviewer findings (expansion budget, Windows-unsafe entry names, path: transport bypass, ENOTDIR) were fixed and resolved before this panel ran.

Dissent. None.

Aligned with: Portable by manifest -- links resolve from the manifest-locked commit's objects; the same content_hash on every OS without privilege escalation. Secure by default -- 40-hop bound, 100000-write budget, cycle detection, escape rejection, entry-name validation and no writes through linked parent directories, all before any disk write. Multi-harness / multi-host -- Windows non-admin (error 1314), containers and CI produce identical results.

Panel summary

Persona B R N Takeaway
Supply Chain Security Expert 0 0 0 Defense-in-depth containment: path guards, hop/write budgets, cycle detection, strict consumer rejection, Git-object provenance. No security regressions found.
Test Coverage Expert 0 2 0 Core logic and whole-repo lifecycle well covered; subdirectory lifecycle e2e and full-clone fallback were the gaps (both folded in e7256c3).
Python Architect 0 0 0 Single-owner link materialization through git_sparse.py; all checkout paths route through it; lenient/strict split linter-enforced; dead code removed.

B = blocking-severity findings, R = recommended, N = nits.
Counts are signal strength, not gates. The maintainer ships.

Recommendation

All three panelists converged and no findings remain open: both test-coverage items are folded with passing, mutation-checked tests. CI on the prior head was fully green, including Windows Compatibility Gate and Windows Native Symlink Acceptance. The fold commit adds tests only. Recommend merging once CI confirms green on the fold head. No follow-ups to track.


Full per-persona findings

Supply Chain Security Expert

No findings.

Test Coverage Expert

  • [recommended] No subdirectory lifecycle e2e test verifying golden content_hash or --frozen cross-platform replay at tests/integration/test_git_object_links_lifecycle.py
    Subdirectory (monorepo) installs take the sparse checkout -> lenient materialize -> strict validate path; only whole-repo had an e2e golden-hash and --frozen test.
    Suggested: Parametrize the lifecycle over a subdirectory package using the update-index --cacheinfo fixture approach.
    Proof (missing at): tests/integration/test_git_object_links_lifecycle.py::test_subdir_links_install_as_content_and_frozen_replays_the_cross_platform_hash -- proves: apm install of a monorepo subdirectory package with links produces the same content_hash on every platform and --frozen replays the lock from a cold cache [portability-by-manifest,devx]
    Folded: e7256c3 -- test_links_install_as_content_and_frozen_replays_the_cross_platform_hash[subdirectory] pins sha256:53a8b0e1..., and test_unresolvable_link_fails_before_hashing runs for both layouts.
  • [recommended] github_downloader subdir full-clone fallback path has mocked-only link materialization at src/apm_cli/deps/github_downloader.py:1591
    The fallback materialize_tracked_symlinks(strict=False) call had no real-Git test.
    Suggested: Drive download_subdirectory_package through the full-clone fallback with a real local bare repo.
    Proof (missing at): tests/integration/test_sparse_cone_symlink_repair.py::test_full_clone_fallback_materializes_out_of_cone_links -- proves: when sparse checkout fails for a monorepo package with links, the full-clone fallback still installs link content [portability-by-manifest,devx]
    Folded: e7256c3; removing the fallback call makes the test fail.

Python Architect

No findings.

This panel is advisory. It does not block merge. Re-apply the
panel-review label after addressing feedback to re-run.


Generated by autopilot-pr-review-worker. This comment is AI-generated and may contain errors.

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 90e76bf into main Nov 12, 2026
30 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-git-object-symlink-materialization branch November 12, 2026 15:12
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
Resolves conflicts with #3603 (links materialized from Git objects) by
keeping both intents: core.symlinks=false plus the
apm.linkMaterialization marker from #3603, and the core.autocrlf=false +
core.eol=lf line-ending pins from #3270. Cache shards now require all
four entries; the snapshot materializer replaces host symlink and
line-ending entries; the git-child-environment needles cover both the
link materializer and the line-ending owner.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
#3603 resolves a tracked link whose target a sparse cone left out with
`git cat-file --filters`, which applies checkout conversion. Callers such
as the downloader's sparse fallback and git_file_transport pass no `-c`
pins, so under a host `core.eol=crlf` (or the Windows native default) a
link to a `text=auto` file was written with CRLF and its content_hash
diverged across platforms, the #3270 failure on a new materialization
path. The resolver now appends git_line_ending_pin_args to its own Git
calls, so every caller gets the committed LF bytes.

Regression: test_link_target_outside_sparse_cone_ignores_host_eol (global
and env-count host core.eol=crlf; host-default control). Mutation-break:
without the pin both crlf cases fail.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
After #3603 the shard checkout runs materialize_tracked_symlinks, which
parses bytes; TestCreateCheckout's text-mode subprocess mocks fed it str
and three cases failed on main as well, alongside a stale config-call
count. The class fixture now stubs the resolver (covered by real-git
tests) and the counts derive from _checkout_materialization_pins, so
the line-ending and link pins are both accounted for. The bare spawn
test returns bytes when the caller did not ask for text.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
#3603's sparse-symlink rule required _LINK_MATERIALIZATION_PIN three
times in git_cache.py (definition, persisted pins, cache-hit check).
With the #3270 line-ending pins both the persisted list and the cache-hit
check route through the single _checkout_materialization_pins(), so the
marker now appears in its definition and that list, and the rule also
requires the helper at its definition, persistence and validation sites.
Dropping the marker from the list still reports a violation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
â€Ķorepo-de

Resolve git_cache.py against #3603: main replaced the sparse-widen
finalizer with materialize_tracked_symlinks. Keep the #3071 completeness
probe on cache hits and write-dedup, and run the post-checkout
unmaterialized-path probe before link materialization so an empty
promisor tree still raises the auth-classifiable IncompleteCheckoutError.
Drop the recency test for the removed finalizer, as main did.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
â€Ķatforms (#3599)

* fix(git): pin core.eol=lf so text=auto package hashes match across platforms

Under .gitattributes text / text=auto, Git writes the line ending that
core.eol selects. Its default (native) is CRLF on Windows, and the
#2971 core.autocrlf=false pin does not override it, so a package
committed with LF was materialized with CRLF on Windows, recorded a
different content_hash and failed `apm install --frozen` across
platforms.

- utils/git_env owns the line-ending pins (core.autocrlf=false and
  core.eol=lf) as -c args and in the network config snapshot, which
  replaces host autocrlf/eol entries like the core.symlinks pin.
- GitCache, shared-bare materialization and the uncached fallback
  clone/checkout carry the pins; cache shards record them and older
  shards without core.eol are rematerialized on hit.
- Explicit eol=lf/eol=crlf attributes, binary paths and committed CRLF
  blobs keep their repository-defined bytes.
- Real-git component tests and installed-CLI lifecycle tests run in the
  windows_compat lane; docs describe the guarantee and the reviewed
  `apm install --update` recovery for locks with pre-fix CRLF hashes.

Fixes #3270

apm-spec-waiver: remote package content_hash is reserved for OpenAPM v0.2; this fixes platform materialization only

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): note text=auto line-ending hash fix (#3599)

apm-spec-waiver: remote package content_hash is reserved for OpenAPM v0.2; this fixes platform materialization only

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs: point stale CRLF lock recovery at the troubleshooting steps

A plain `apm install --update` does not re-record a stale CRLF hash when
the installed copy still matches the locked SHA (install/phases/integrate.py
skips the re-download), and it re-resolves mutable refs. The packaged usage
guide and the CHANGELOG promised a same-pin in-place refresh; both now point
at the `Content hash mismatch` recovery steps, which require a fresh
consumer or removal of the installed copy and review of the SHA and hash
diff. install-packages.md links the lockfile content_hash definition
instead of restating it, and notes that older cached checkouts are
re-created on their next use.

Addresses Copilot inline on packages/apm-guide/.apm/skills/apm-usage/
dependencies.md:97 and panel follow-ups (doc-writer, supply-chain,
performance).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* perf(cache): persist bare consumer pins via clone --config

materialize_from_bare runs once per consumer and is not amortized across
installs. Writing the two line-ending pins with one `git config` spawn
each added a process per dependency on every install (Windows pays the
most per spawn). `git clone --config` records the same values in the new
repository's config with no extra process; the checkout still carries
the `-c` pins that outrank env snapshots. A comment notes why the symlink
pin travels only as `-c` there. The rematerialization of a cache shard
that lacks the pins now logs its elapsed time, so the one-time cost after
upgrading is observable.

The bare regression test asserts the consumer config records both pins;
mutation-break: dropping the `--config` args fails it.

Addresses panel follow-ups (performance-expert, python-architect).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(architecture): prove the line-ending pin guard has teeth

The PR extends transport-platform-git-child-environment with needles for
git_line_ending_pin_* in git_env.py, git_cache.py and bare_cache.py and a
forbid scan for core.eol literals outside git_env.py, but no test proved
those checks fire. Three source-override mutations now do: a core.eol
literal in bare_cache.py, git_cache.py bypassing git_line_ending_pin_args,
and the config snapshot no longer appending the line-ending pins. Each must
report a violation with the matching message.

Mutation-break: restoring the pre-PR linter check makes all three fail.

Addresses panel follow-up (test-coverage-expert).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(cache): trap extra git config spawns on the bare consumer path

A mocked materialize_from_bare must carry the line-ending pins as clone
--config arguments and spawn no separate `git config core.*` process, so
a later change that reintroduces per-key spawns on this non-amortized
path is a reviewed decision. Mutation-break: the previous per-key loop
fails it.

Addresses panel follow-up (performance-expert nit).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(git): pin line endings when resolving link targets from Git objects

#3603 resolves a tracked link whose target a sparse cone left out with
`git cat-file --filters`, which applies checkout conversion. Callers such
as the downloader's sparse fallback and git_file_transport pass no `-c`
pins, so under a host `core.eol=crlf` (or the Windows native default) a
link to a `text=auto` file was written with CRLF and its content_hash
diverged across platforms, the #3270 failure on a new materialization
path. The resolver now appends git_line_ending_pin_args to its own Git
calls, so every caller gets the committed LF bytes.

Regression: test_link_target_outside_sparse_cone_ignores_host_eol (global
and env-count host core.eol=crlf; host-default control). Mutation-break:
without the pin both crlf cases fail.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(cache): keep checkout mocks off the link resolver

After #3603 the shard checkout runs materialize_tracked_symlinks, which
parses bytes; TestCreateCheckout's text-mode subprocess mocks fed it str
and three cases failed on main as well, alongside a stale config-call
count. The class fixture now stubs the resolver (covered by real-git
tests) and the counts derive from _checkout_materialization_pins, so
the line-ending and link pins are both accounted for. The bare spawn
test returns bytes when the caller did not ask for text.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(architecture): count the link marker through the shared pin list

#3603's sparse-symlink rule required _LINK_MATERIALIZATION_PIN three
times in git_cache.py (definition, persisted pins, cache-hit check).
With the #3270 line-ending pins both the persisted list and the cache-hit
check route through the single _checkout_materialization_pins(), so the
marker now appears in its definition and that list, and the rule also
requires the helper at its definition, persistence and validation sites.
Dropping the marker from the list still reports a violation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* perf(cache): persist shard pins on the clone instead of config spawns

GitCache wrote each materialization pin with its own `git config` spawn
after cloning a shard; with the line-ending and link pins that became
four extra processes per new or rematerialized shard. The pins now ride
on the existing clone as `-c key=value`, which git clone records in the
new repository, as bare_cache already does. The checkout keeps the
`-c` pins from _safe_git_args. Hermetic call counts drop to clone plus
checkout.

Mutation-break: dropping the pins from the clone fails
test_cached_checkout_ignores_host_eol (shard no longer recognized).

Addresses delta-panel follow-up (performance-expert).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* test(architecture): prove the shard pin-list count guard has teeth

The sparse-symlink rule now requires git_cache.py to route persisted and
required pins through _checkout_materialization_pins(). A fourth
source-override case inlines the pin tuple at the persistence site and
expects the rule to report it. Mutation-break: main's count-only rule
misses it.

Addresses delta-panel follow-up (test-coverage-expert).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(changelog): place #3599 entry away from the section head

Avoids landing-order conflicts with other queued PRs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore(changelog): place #3599 entry away from the section head

Avoids landing-order conflicts with other queued PRs.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 12, 2026
â€Ķ3673)

* fix(git): check out out-of-cone link targets with Git's own checkout

A link whose target a sparse checkout left out was read with
`git --attr-source=HEAD cat-file --filters`, falling back to plain
`cat-file --filters` when that call failed. cat-file reads no index, so
in a sparse checkout the fallback silently skipped the target's
.gitattributes and wrote e.g. an eol=crlf file with LF. That is the
post-merge red on macOS x86_64 since 90e76bf: the path: transport link
to shared/notes.crlf returned LF while the checked-out target was CRLF.

Out-of-worktree targets are now written by `checkout-index` into a
throwaway work tree from a throwaway full index (`read-tree HEAD`), in
one batch per materialization. Attributes come from that index exactly
as `git checkout` reads them, on every Git version, with no
version-gated option and no silent fallback. In-worktree targets still
read the bytes Git already wrote. The line-ending pins (#3599) still
apply; the content_hash domain is unchanged.

Regression: test_out_of_sparse_eol_target_keeps_checkout_bytes_without_attr_source
emulates a Git that rejects --attr-source; it fails on main with the CI
assertion and passes here. Also re-points two #3603 downloader tests at
the persistent-checkout seam that #3640 changed underneath them.

Refs #3457

apm-spec-waiver: Git checkout byte fidelity for link targets; no OpenAPM statement or hash-domain change.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* docs(changelog): reference #3673 on the link materialization entry

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(links): report an unwritten out-of-cone target with the diagnostic

A target that checkout-index leaves unwritten raised a bare
FileNotFoundError; it now raises the 'did not check out ... as a regular
file' diagnostic before any link is written. Panel follow-up
(test-coverage-expert).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* fix(links): rebuild shards from the cat-file materializer; honor APM_TEMP_DIR

Review follow-ups (Copilot):
- The shard marker is versioned to apm.linkMaterialization=git-objects-checkout.
  A sparse shard pinned by the first Git-object materializer could hold
  an out-of-cone eol=crlf target with LF and would otherwise be served
  as is; it is now rematerialized on its next cache hit. Regression:
  test_cache_hit_rematerializes_shard_without_link_materialization[pre-checkout-marker]
  (fails if the marker value is reverted).
- The throwaway checkout scratch directory uses get_apm_temp_dir(), like
  the downloader's own scratch space, so APM_TEMP_DIR / temp_dir keep
  working on hosts with a restricted system temp directory.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore: restore uv.lock swept in by a local uv rewrite

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Daniel Meppiel (danielmeppiel) added a commit that referenced this pull request Nov 17, 2026
* fix(tests): restore release-gate lanes for 0.36.0

Main's tag-gate lanes (build-release push + manual-build-release) were red
on runs 43381609442, 43381682871 and 43394173235. PR CI never runs them.

Product regressions (Windows/real installs), fixed:
- Hook cleanup removed nothing on Windows: remove_stale_deployed_files
  looked up recorded hashes with POSIX keys while callers passed native
  backslash keys, so content proof failed closed (#3672, Refs #3649).
- describe_path_obstruction returned None on Windows, which reports a
  directory-for-file as EACCES and a file-for-parent as ENOENT instead of
  EISDIR/ENOTDIR (#3709, Refs #3678).
- Skill preflight refused claimed members of an unclaimed, populated skill
  directory that materialization leaves alone, failing a local bundle
  install (Refs #3625).

Stale tests (behaviour changed intentionally), updated:
- policy chain mocks now return real PolicyFetchResult (#2933, Refs #2753)
- bare-cache/downloader/anonymous-https fixtures use real Git checkouts
  (#3603, Refs #3457)
- mcp install help, registry error wording, deps why -g help
  (#3633 Refs #3588, #2803 Refs #2478, #3614 Refs #2884)
- downloader stub carries carried_licenses (#3646, Refs #3556)
- frozen-hash lifecycle uses a generic git host so -g URL validation
  uses git ls-remote instead of a live api.github.com 404 (#3672)

New Windows-shaped regression tests are marked windows_compat.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* chore: spec waiver for release-gate repair

apm-spec-waiver: restores pre-existing behaviour on Windows and in skill preflight; no OpenAPM-observable change

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: danielmeppiel <danielmeppiel@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Package content hash differs on Windows for packages with in-repo symlinks, so --frozen fails

2 participants