Skip to content

Instrument work queue state and health with OpenTelemetry - #74634

Merged
pelikhan merged 4 commits into
mainfrom
pelikhan/work-queue-telemetry
Nov 3, 2026
Merged

pelikhan merged 4 commits into
mainfrom
pelikhan/work-queue-telemetry

Conversation

@pelikhan

@pelikhan pelikhan commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator

Operators need visibility into work queue backlog, retained capacity, and recovery failures without relying on debug logs. This adds correlated OpenTelemetry spans for runtime queue reads, publication, dispatch, reconciliation, and compaction.

Approach

  • Record verified queue snapshots under gh-aw.work_queue.*, including Work counts and backlog age, delivery barriers, open Claims, retained reservations, capacity saturation, and paused controls.
  • Capture publication attempts, CAS conflicts, ambiguous writes, and reconciliation outcomes. Count only acknowledged new commits; recovered requests and uncommitted candidates do not inflate transition counts.
  • Reuse existing workflow trace context, OTLP fan-out, and sanitized local mirrors. Nested ledger operations contribute to the outer operation span, and telemetry failures remain non-fatal.
  • Document operator-facing attributes, freshness semantics, and dashboard aggregation guidance, and add a patch changeset.

Operational considerations

Health measurements are event-driven span snapshots, not continuously sampled metrics. Dashboards should select the latest observation per queue and alert separately on stale observations. Missing snapshots mean unknown state, not an empty queue. Queue-specific telemetry excludes task payloads, result bodies, Work/Claim/dispatch/request IDs, and raw errors.

Validation

  • JavaScript type-check passed.
  • Formatting, lint, and the required make agent-report-progress gate passed.
  • Targeted queue lifecycle, compaction, expiration, summary, documentation, exporter, and telemetry regression suites passed: 466 tests across 11 test files.

pelikhan and others added 2 commits November 3, 2026 12:05
@pelikhan
pelikhan marked this pull request as ready for review November 3, 2026 22:13
Copilot AI adaptive review requested due to automatic review settings November 3, 2026 22:13
@pelikhan

pelikhan commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot fix ci failures

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · routed: opus55 medium · ~12 AIC · ⌖ ~2 AIC · ⊞ 7.7K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · ~15 AIC · ⌖ ~21 AIC · ⊞ 12.9K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #74634

@pelikhan

pelikhan commented Nov 3, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot fix ci failures

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What this change does: This PR wraps queue reads, publications, dispatch, reconciliation, and compaction in correlated OpenTelemetry spans. It adds metadata-only queue snapshots and tests for publication recovery, lifecycle outcomes, and exporter behavior.

Should fix:

  1. Telemetry maintains a second request registry (actions/setup/js/work_queue_telemetry.cjs:L29)

    • What this is: The new REQUESTS set repeats the request-kind vocabulary already defined by REQUEST_KINDS in actions/setup/js/work_queue_replay.cjs.
    • Problem: A future request kind can be accepted by replay but silently omitted from gh-aw.work_queue.request.kind until someone edits both lists. That is extra code and a protocol/telemetry drift path.
    • Fix: Export and reuse Object.keys(REQUEST_KINDS) (or put the vocabulary in one shared module), keeping only telemetry-specific filters local.
    • If we skip it: Queue behavior and its operator telemetry can disagree without a failing validation.
  2. The export guard duplicates the OTLP failure boundary (actions/setup/js/work_queue_telemetry.cjs:L200-L203)

    • What this is: otlp.logSpan already catches build and exporter errors and always resolves.
    • Problem: The outer catch is not needed for exporter failures and also labels errors from snapshot aggregation or queue-ID construction as exporter failures. It makes two unrelated failure policies harder to reason about.
    • Fix: Guard local snapshot assembly separately if it must remain non-fatal, then call logSpan directly and let the existing OTLP boundary handle export errors.
    • If we skip it: A telemetry implementation bug is reported as a collector outage, while the duplicate warning path remains dead for normal export failures.

Verdict: fix 1 and 2 first.
Lean: -8 lines possible.

Generated by ✂️ Ponytail Reviewer for #74634 · codex · routed: gpt56 max · ~9 AIC · ⌖ ~1 AIC · ⊞ 118
Comment /ponytail to run again

"operation_budget_blocked",
"prefix_complete",
]);
const REQUESTS = new Set(["policy", "pool_install", "deployment", "control", "submit", "dispatch_next", "finish", "observe", "dispatch", "release", "result", "delivery_failure", "cancel_work", "cancel_claim", "checkpoint", "issue_link"]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should fix: REQUESTS duplicates the authoritative request-kind table in actions/setup/js/work_queue_replay.cjs. A new valid request can then be accepted by replay but silently omitted from telemetry until a second list is edited; derive this set from REQUEST_KINDS (or move the vocabulary to one shared module) so protocol and telemetry cannot drift.

for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value;
}
await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError });
} catch {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should fix: otlp.logSpan already catches build and exporter errors and always resolves, so this outer catch is not an exporter safety net. It also labels failures from snapshotAttributes or queue-id construction as exporter failures; guard local snapshot assembly separately and let the existing OTLP boundary own export errors.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The telemetry integration can break isolated queue consumers because of missing bundled dependencies and expose credential-bearing URLs through unsanitized exporter errors.

8 open issues
What changed in this PR

This PR adds OpenTelemetry visibility to gh-aw’s runtime work queue, helping operators observe backlog, retained capacity, and recovery health through existing workflow traces.

Changes:

  • Add queue-health snapshots and publication, dispatch, reconciliation, and compaction instrumentation.
  • Reuse workflow trace context and shared OTLP export infrastructure.
  • Add telemetry regression coverage, operator documentation, and a patch changeset.
File Description
docs/​src/​content/​docs/​reference/​work-queue.md Documents queue-health attributes, snapshot freshness, and dashboard guidance.
docs/​src/​content/​docs/​reference/​open-telemetry-attributes.mdx Links the telemetry reference to the queue-health inventory.
actions/​setup/​js/​work_queue_telemetry.test.cjs Adds coverage for snapshots, publication accounting, lifecycle outcomes, and telemetry failures.
actions/​setup/​js/​work_queue_telemetry.cjs Implements operation spans, health snapshots, and telemetry accounting.
actions/​setup/​js/​work_queue_store.cjs Instruments ledger reads, publications, and the compaction helper.
actions/​setup/​js/​work_queue_reconciler.cjs Wraps dispatch and queue reconciliation with telemetry.
actions/​setup/​js/​work_queue_dispatch.cjs Wraps assignment launches with telemetry.
.changeset/​work-queue-telemetry.md Adds a patch release note for queue observability.

Review effort: Max (auto)

const { queueAuthenticationScope } = require("./work_queue_auth_cache.cjs");
const { queueGitScope, queueLedgerClient } = require("./work_queue_git_transport.cjs");
const { expirationView } = require("./work_queue_expiration.cjs");
const { observeQueueOperation, publicationAttempt, publicationRecovery } = require("./work_queue_telemetry.cjs");
if (current.snapshot) {
for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value;
}
await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError });
/** Checkpoint validated history against the current Git HEAD and publish by fast-forward CAS. */
async function compactWorkQueue({ githubClient, owner, repo, context, branch = WORK_QUEUE_BRANCH, maxRetries = DEFAULT_MAX_RETRIES, now = Date.now }) {
async function compactWorkQueue(options) {
return observeQueueOperation("compact", options, () => compactWorkQueueImpl(options));
if (expired) attributes["work.expired"]++;
if (ownership === "available") {
if (work.retry_not_before > time) attributes["work.retry_delayed"]++;
attributes["work.oldest_available_age_ms"] = Math.max(Number(attributes["work.oldest_available_age_ms"]), Math.max(0, time - work.enqueued));
Comment on lines +77 to +78
// keep_open completions have Claim-local barriers, independent of Work.
if (claim.keep_open && ["pending", "failed"].includes(claim.barrier)) attributes[`delivery.${claim.barrier}`]++;
if (current.snapshot) {
for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value;
}
await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError });
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

@copilot fix ci failures

Fixed the source-backed CJS and Docs Build failures in d55ec490 (bundle dependencies and absolute docs links); focused tests and the final gate pass. The skills-review job’s remaining failure was a post-job OTLP collector HTTP 401.

@copilot fix ci failures

Fixed the source-backed CJS and Docs Build failures in d55ec490 (bundle dependencies and absolute docs links); focused tests and the final gate pass. The skills-review job’s remaining failure was a post-job OTLP collector HTTP 401.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan
pelikhan merged commit 5932217 into main Nov 3, 2026
49 checks passed
@pelikhan
pelikhan deleted the pelikhan/work-queue-telemetry branch November 3, 2026 23:20
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Warning

The push_to_pull_request_branch operation was skipped: Pull request #74634 is closed; refusing to push to its branch. The code changes were not applied.

Submitted one commit for publication (not confirmed pushed) addressing review items 2, 3, 5, 6, 7, 8, 9: derived request kinds from REQUEST_KINDS; split attribute assembly from export; sanitized export errors in the OTLP helpers; instrumented work_queue_compaction_apply; admission-time backlog age; no double-counted mirrored keep_open barriers (docs updated); export bounded to 2s.

Not done:

  • Item 1: make merge-main failed — no git credentials/fetch; local origin/main is 10 commits ahead, and the merge needed a promisor fetch.
  • Item 4 (bundle lists): the exporter has a large transitive closure missing from the isolated bundles, so exportSpan no-ops if send_otlp_span.cjs is absent. Adding the full closure to setup.sh/session_parsers.go is left for a maintainer.

Validation: the JS test runner could not install dependencies (certificate error), so vitest did not run. go test ./actions/setup, a node smoke test, and prettier passed. I did not run make agent-report-progress. CI needs a maintainer re-trigger. Review replies and resolutions are deferred until the commit is published.

Sous-chef head: 16575ee
Sous-chef work: 0504060a177b077799c356baf020e0e36b7f230d85ed39df02997d23dfd79979 08105e91563129e60993c5e48b1f2d8f44fd7c9ea2f24810718663a12c8f4608 67bfe77762174a9d09ea2a10809efcef2de05cc79dab7c40649bda4d26871070 684d1fd0e16bf114bc1bd411e4e036c2f0298e4c0943bb992ca4f5461764bcbe 7591bf5c5bc6d8ad40f2168dca24b330f95c8367bab75053185185e0d4fb029c a0ef1e64cec48bd98588510fd20e62cfb02b8455a7c42c13a340ea779fea4512 a5c048fe42b61b2dd6b083c826d6d6e0c79d66db5c302b5752346e9e52a0774b f12f93ae8d28dd481b4bab65f1fe0c9a4a139c2e12040683ce9061739a32b8af
Sous-chef state: f5e728f0260828e2ffc06f19aff5224e3feb37db2956b9585ea31777ed6d5253

Generated by 🏁 PR Finisher · copilot · auto · ~90 AIC · ⌖ ~2 AIC · ⊞ 10.7K · ◷

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants