Skip to content

Honor Retry-After in Copilot harness HTTP 429 retries - #76807

Merged
pelikhan merged 8 commits into
mainfrom
copilot/fix-retry-after-http-429
Nov 11, 2026
Merged

pelikhan merged 8 commits into
mainfrom
copilot/fix-retry-after-http-429

Conversation

Copilot AI commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

The Copilot harness retried HTTP 429 failures on fixed backoff, exhausting attempts inside the upstream rate-limit window.

  • Retry timing: Parse Retry-After seconds or HTTP dates from terminal error events and stderr. Wait the requested duration within the configured retry budget.
  • Bounded failure: Stop immediately with rate_limit_wait_exceeded when the wait exceeds max-delay-ms or the remaining timeout budget, preserving shutdown time and structured telemetry.
  • Compatibility: Ignore assistant/tool events as retry evidence. Preserve existing behavior without a valid header, terminal policy guards, and HTTP 429 model-fallback behavior.

A 245-second window requires sufficient remaining execution time and:

engine:
  id: copilot
  harness:
    max-delay-ms: 245000

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Copilot harness to honor Retry-After on HTTP 429 Honor Retry-After in Copilot harness HTTP 429 retries Nov 11, 2026
Copilot AI requested a review from pelikhan November 11, 2026 08:52
@pelikhan
pelikhan marked this pull request as ready for review November 11, 2026 09:07
Copilot AI adaptive review requested due to automatic review settings November 11, 2026 09:07
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

PR Code Quality Reviewer

Outcome: reported_action

The agent completed and reported one or more actions. The no-op output is a completion signal, not a claim that no action was taken.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from PR Code Quality Reviewer · gpt56 · ~5 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

Commit pushed: 50a880d

Generated by PR Specification Reviewer for #76807

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗ïļ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗ïļ

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗ïļ Â· routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 8.5K · ◷

🏗ïļ ADR gate enforced by Design Decision Gate 🏗ïļ

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

🧊 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · routed: gpt61 medium · ~6 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 181 · ◷

Generated by Ponytail Reviewer for #76807

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-11-11T09:10:34Z
review_event: REQUEST_CHANGES
top_themes:
  - Unscoped stderr Retry-After evidence can trigger incorrect rate-limit waits
files_reviewed:
  - actions/setup/js/copilot_harness.cjs
  - actions/setup/js/copilot_retry_after.cjs
  - actions/setup/js/harness_retry_runner.cjs
  - actions/setup/js/harness_retry_config.cjs
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

if (NATIVE_RATE_LIMIT_ERROR_PATTERN.test(JSON.stringify(event))) nativeTerminalRateLimitError = true;
}
},
onStderrLine: line => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observeRetryAfter accepts every stderr line for the entire attempt, unlike stdout where non-terminal event types are filtered. This means unrelated diagnostic text can combine a 429 mention with a Retry-After header and override the normal failure classification, causing an unnecessary long sleep or rate_limit_wait_exceeded stop even when the terminal failure is not rate limiting. Please restrict stderr evidence to a recognized terminal error/report (or correlate the header with the same error) before applying it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: stderr Retry-After evidence is now accepted only alongside a recognized response status, and assistant/tool events remain excluded. Header evidence is retained independently of the output tail.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found one blocking correctness issue: stderr Retry-After evidence is not scoped to terminal rate-limit errors, so unrelated diagnostic output can trigger an incorrect wait or stop. Inline comment added. Static review only; tests were not run per review instructions.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~5 AIC · ⌖ ~1 AIC · ⊞ 1.6K
Comment /review to run again

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head d4fb266 against the applicable formal contract. The implementation and tests intentionally add bounded Copilot terminal HTTP 429 Retry-After handling, while preserving invalid/missing/expired-header behavior and excluding assistant/tool events from retry evidence. This behavior is covered by the synchronized Unified Agent Session Specification v2.9.0, Section 6.4, T-UAS-090: numeric-seconds and HTTP-date parsing, max-delay/soft-timeout bounds, and allowlisted rate_limit_wait_exceeded classification. The Copilot SDK Driver Specification v1.0.2 is out of scope. Specification commit e0a6f87 was queued for publication to the PR branch. No blocking contract violations found.

Generated by PR Specification Reviewer for #76807 · copilot · gpt50 · ~5 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review-specs to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ðŸŸĄ Changes recommended

Retry-After handling has correctness gaps: early retry branches bypass delay limits, valid HTTP-date formats are rejected, and rate-limit detection can misclassify responses.

3 open issues
What changed in this PR

This PR adds bounded Retry-After handling to the Copilot harness, addressing retries that otherwise occur inside an upstream HTTP 429 rate-limit window.

Changes:

  • Parse retry delays from terminal error events and stderr, excluding assistant/tool events.
  • Apply delay and timeout limits, recording rate_limit_wait_exceeded when the wait cannot fit.
  • Add parser and integration tests, extend telemetry types, and document configuration.
File Description
docs/​src/​content/​docs/​reference/​engines.md Documents retry timing and configured bounds.
docs/​public/​schemas/​unified-session.schema.json Adds the new failure class to the schema.
actions/​setup/​js/​types/​unified_session.d.ts Extends the failure-class type.
actions/​setup/​js/​harness_retry_runner.cjs Documents retry-delay metadata on attempt results.
actions/​setup/​js/​harness_retry_record.test.cjs Covers normalization of the new failure class.
actions/​setup/​js/​harness_retry_record.cjs Recognizes the new failure class.
actions/​setup/​js/​copilot_retry_after.test.cjs Adds parser and harness integration coverage.
actions/​setup/​js/​copilot_retry_after.cjs Implements Retry-After parsing.
actions/​setup/​js/​copilot_harness.cjs Collects retry evidence and applies bounded waits.

Review effort: Balanced (auto)

Comment thread actions/setup/js/copilot_harness.cjs Outdated
Comment on lines +2524 to +2527
if (result.retryAfterMs != null) {
// Node timers overflow above 2^31-1 ms and would otherwise retry immediately.
const exceedsDelayCap = result.retryAfterMs > Math.min(maxDelayMs, 2147483647);
const exceedsTimeout = softTimeoutGuard && Date.now() + result.retryAfterMs >= softTimeoutGuard.softDeadlineMs;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: every retry decision now passes through the same Retry-After delay and budget checks; the existing fresh-versus-continue choice is preserved.

*/
function parseCopilotRetryAfterMs(output, now = Date.now()) {
const text = output.replace(/\\[rn]/g, "\n").replace(/\\"/g, '"');
if (!/\b429\b/.test(text)) return null;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: Retry-After is associated with its explicit response status or recognized Copilot error signature. CAPIError: Too Many Requests is supported, while a 503 with a numeric Retry-After value is not treated as a 429.

Comment on lines +25 to +26
} else if (/^[A-Za-z]{3}, /.test(value)) {
ms = Date.parse(value) - now;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: the parser now supports IMF-fixdate, RFC 850, and asctime forms, with UTC interpretation and the RFC two-digit-year rule; regression tests cover both obsolete formats.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs to the Retry-After branch in copilot_harness.cjs. The design is sound: the parser ignores assistant and tool events, the max-delay-ms cap and soft-timeout guard prevent unbounded waits, and exhaustion stops like the existing paths.

📋 Key Themes & Highlights

Key Themes

  • Quota ordering: the Retry-After branch runs before the quota stop, so a 429 quota exceeded with retry-after is retried. Confirmed with node.
  • Output truncation: the 64 KB tail buffer can drop an early Retry-After header.

Positive Highlights

  • ✅ Timer overflow above 2^31-1 ms is handled, with tests.
  • ✅ Assistant text cannot trigger a wait, since only error events and stderr are observed.
  • ✅ Exhaustion returns stop, consistent with the existing paths.

Limitations

  • The vitest suites could not run here: npm ci failed on a registry TLS certificate error, so the test results are unverified.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · routed: haiku55 xhigh · ~7 AIC · ⌖ ~2 AIC · ⊞ 6.6K
Comment /matt to run again

log(`attempt ${attempt + 1}: startup interruption detected (driver-handoff Turns=0) but retry budget exhausted — no attempts remain`);
}

if (result.retryAfterMs != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] This Retry-After branch runs before the isQuotaExceeded stop (line 2550). A hard quota error such as CAPIError: 429 429 quota exceeded with a retry-after header parses to a delay, and isCAPIQuotaExceededError also matches it, so it is classified as rate_limit_error and retried instead of stopped. I confirmed this with node: the parser returns 2000 and the quota matcher returns true for that input.

ðŸ’Ą Suggested fix or test

If a quota error should stop, gate the branch on !isQuotaExceeded. If retrying is intended, add a regression test that pins the quota-plus-Retry-After case so the ordering is explicit.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: explicit hard quota-exceeded errors remain terminal even with Retry-After, while retryable CAPIError: Too Many Requests responses honor the header.

Comment thread actions/setup/js/copilot_harness.cjs Outdated
} catch {
// Ignore JSON parsing failures: native CLI errors and header dumps are plain text.
}
retryAfterOutput = (retryAfterOutput + "\n" + line).slice(-64 * 1024);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The 64 KB buffer keeps only the tail of the output (slice(-64 * 1024)). A Retry-After header printed early in a long run is dropped, and the harness silently falls back to exponential backoff.

ðŸ’Ą Suggested change

Keep header-bearing lines separately, or retain the first N KB along with the tail. Add a test that puts the header before a more-than-64 KB prefix so the truncation is covered.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 61477796: Retry-After evidence is retained separately from the output tail, so an early header is not lost after long stderr output. Added a >64 KB regression test.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

ðŸģ A little prep to help this PR along! Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/copilot_harness.cjs:2144): address unresolved feedback - Review comment
  3. Review (actions/setup/js/copilot_harness.cjs:2524): address unresolved feedback - Review comment
  4. Review (actions/setup/js/copilot_harness.cjs:2117): address unresolved feedback - Review comment

Keeping the changes focused; human review and CI get the final taste test. No merging or CI triggering.

Structured data
{
  "completed_task": 0,
  "head": "50a880d0a0f8f8fa6460d0d02db24c9d5348434e",
  "kind": "requested",
  "schema_version": "1",
  "state": "0331cbd9b8b4ba0dfd13630e4b14dc5c00611b506d2b0db80bc81be6ac6d089f",
  "work": [
    "49be6728adf5406903178a22e1d8e990267ed84176ca555f2587afda3d483a59",
    "867e2f680ea9b4e83bba85be20ab1d3b4351ca6a4e2642c8a266c5be76c16be2",
    "facb9d97420d5c78f5ceda7e09fb82b05dd669a3254219d0a33bd1f49caa8baf"
  ],
  "workflow": "pr-sous-chef"
}

Generated by ðŸ‘ĻðŸģ PR Sous Chef · pi · routed: haiku55 xhigh · ~3 AIC · ⌖ ~2 AIC · ⊞ 6.9K · ◷
Comment /souschef to run again

â€Ķer-http-429

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot

ðŸģ A little prep to help this PR along! Copilot assignment requested for the following outstanding work:...

Completed in 61477796: merged the latest main, addressed the assigned Retry-After feedback, and passed the final validation gate (545 focused tests passed). CodeQL timed out and its tool instructed me not to retry; no CI was triggered.

Comment thread actions/setup/js/copilot_harness.cjs Outdated
}
},
onStderrLine: line => {
observeStderrRetryAfter(line);
@github-actions

Copy link
Copy Markdown
Contributor

Category: bug | Risk: medium | Score: 83/100 (impact 41 + urgency 27 + quality 15). Recommended action: fast_track. Batch: pr-batch:copilot-runtime. Reason: fixes a real runtime failure in Copilot 429 handling and is already green on CI.

Generated by 🔧 PR Triage Agent · copilot · mai11 · ~5 AIC · ⌖ ~2 AIC · ⊞ 15.9K · ◷

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot generalize to other agentic engines or any engine using GitHub inference
update debug playbook

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot generalize to other agentic engines or any engine using GitHub inference
update debug playbook

Implemented in e60b9a59 for Claude and Codex, including GitHub inference, and updated the debug playbook. Engines without a retrying harness still need engine-specific integration. Focused tests and the final validation gate passed; CodeQL did not analyze any changes.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot only enable this check if the provider of the model is GitHub

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot only enable this check if the provider of the model is GitHub

Implemented in a04913cb: Retry-After timing applies only to GitHub-provided models; other providers retain their existing retry behavior. The final validation gate passed (666 tests).

}
},
onStderrLine: line => {
if (honorRetryAfter) observeStderrRetryAfter(line);
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot run pr-finisher skill

@pelikhan
pelikhan merged commit bc5dec5 into main Nov 11, 2026
34 checks passed
@pelikhan
pelikhan deleted the copilot/fix-retry-after-http-429 branch November 11, 2026 19:54
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Copilot harness: honour Retry-After on HTTP 429 instead of retrying inside the window

5 participants