Repository navigation
Instrument work queue state and health with OpenTelemetry - #74634
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot fix ci failures |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Design Decision Gate 🏗️Outcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality SentinelOutcome: The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible. Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.
|
|
🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills... |
|
✅ Ponytail Reviewer completed successfully!
|
|
@copilot fix ci failures |
There was a problem hiding this comment.
What this change does: This PR wraps queue reads, publications, dispatch, reconciliation, and compaction in correlated OpenTelemetry spans. It adds metadata-only queue snapshots and tests for publication recovery, lifecycle outcomes, and exporter behavior.
Should fix:
-
Telemetry maintains a second request registry (
actions/setup/js/work_queue_telemetry.cjs:L29)- What this is: The new
REQUESTSset repeats the request-kind vocabulary already defined byREQUEST_KINDSinactions/setup/js/work_queue_replay.cjs. - Problem: A future request kind can be accepted by replay but silently omitted from
gh-aw.work_queue.request.kinduntil someone edits both lists. That is extra code and a protocol/telemetry drift path. - Fix: Export and reuse
Object.keys(REQUEST_KINDS)(or put the vocabulary in one shared module), keeping only telemetry-specific filters local. - If we skip it: Queue behavior and its operator telemetry can disagree without a failing validation.
- What this is: The new
-
The export guard duplicates the OTLP failure boundary (
actions/setup/js/work_queue_telemetry.cjs:L200-L203)- What this is:
otlp.logSpanalready catches build and exporter errors and always resolves. - Problem: The outer catch is not needed for exporter failures and also labels errors from snapshot aggregation or queue-ID construction as exporter failures. It makes two unrelated failure policies harder to reason about.
- Fix: Guard local snapshot assembly separately if it must remain non-fatal, then call
logSpandirectly and let the existing OTLP boundary handle export errors. - If we skip it: A telemetry implementation bug is reported as a collector outage, while the duplicate warning path remains dead for normal export failures.
- What this is:
Verdict: fix 1 and 2 first.
Lean: -8 lines possible.
Generated by ✂️ Ponytail Reviewer for #74634 · codex · routed: gpt56 max · ~9 AIC · ⌖ ~1 AIC · ⊞ 118
Comment /ponytail to run again
| "operation_budget_blocked", | ||
| "prefix_complete", | ||
| ]); | ||
| const REQUESTS = new Set(["policy", "pool_install", "deployment", "control", "submit", "dispatch_next", "finish", "observe", "dispatch", "release", "result", "delivery_failure", "cancel_work", "cancel_claim", "checkpoint", "issue_link"]); |
There was a problem hiding this comment.
Should fix: REQUESTS duplicates the authoritative request-kind table in actions/setup/js/work_queue_replay.cjs. A new valid request can then be accepted by replay but silently omitted from telemetry until a second list is edited; derive this set from REQUEST_KINDS (or move the vocabulary to one shared module) so protocol and telemetry cannot drift.
| for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value; | ||
| } | ||
| await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError }); | ||
| } catch { |
There was a problem hiding this comment.
Should fix: otlp.logSpan already catches build and exporter errors and always resolves, so this outer catch is not an exporter safety net. It also labels failures from snapshotAttributes or queue-id construction as exporter failures; guard local snapshot assembly separately and let the existing OTLP boundary own export errors.
There was a problem hiding this comment.
🟡 Changes recommended
The telemetry integration can break isolated queue consumers because of missing bundled dependencies and expose credential-bearing URLs through unsanitized exporter errors.
8 open issues
Bundle queue telemetry in isolated setup artifacts · New Sanitize errors logged by OTLP export helpers · New Instrument the production compaction entry point · New Measure backlog age from admission time · New Avoid double-counting mirrored delivery barriers · New Keep remote span export off the queue critical path · New Fix the broken work-queue inventory link · New Fix the relative link to the OTLP guide · New
What changed in this PR
This PR adds OpenTelemetry visibility to gh-aw’s runtime work queue, helping operators observe backlog, retained capacity, and recovery health through existing workflow traces.
Changes:
- Add queue-health snapshots and publication, dispatch, reconciliation, and compaction instrumentation.
- Reuse workflow trace context and shared OTLP export infrastructure.
- Add telemetry regression coverage, operator documentation, and a patch changeset.
| File | Description |
|---|---|
docs/src/content/docs/reference/work-queue.md |
Documents queue-health attributes, snapshot freshness, and dashboard guidance. |
docs/src/content/docs/reference/open-telemetry-attributes.mdx |
Links the telemetry reference to the queue-health inventory. |
actions/setup/js/work_queue_telemetry.test.cjs |
Adds coverage for snapshots, publication accounting, lifecycle outcomes, and telemetry failures. |
actions/setup/js/work_queue_telemetry.cjs |
Implements operation spans, health snapshots, and telemetry accounting. |
actions/setup/js/work_queue_store.cjs |
Instruments ledger reads, publications, and the compaction helper. |
actions/setup/js/work_queue_reconciler.cjs |
Wraps dispatch and queue reconciliation with telemetry. |
actions/setup/js/work_queue_dispatch.cjs |
Wraps assignment launches with telemetry. |
.changeset/work-queue-telemetry.md |
Adds a patch release note for queue observability. |
Review effort: Max (auto)
| const { queueAuthenticationScope } = require("./work_queue_auth_cache.cjs"); | ||
| const { queueGitScope, queueLedgerClient } = require("./work_queue_git_transport.cjs"); | ||
| const { expirationView } = require("./work_queue_expiration.cjs"); | ||
| const { observeQueueOperation, publicationAttempt, publicationRecovery } = require("./work_queue_telemetry.cjs"); |
| if (current.snapshot) { | ||
| for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value; | ||
| } | ||
| await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError }); |
| /** Checkpoint validated history against the current Git HEAD and publish by fast-forward CAS. */ | ||
| async function compactWorkQueue({ githubClient, owner, repo, context, branch = WORK_QUEUE_BRANCH, maxRetries = DEFAULT_MAX_RETRIES, now = Date.now }) { | ||
| async function compactWorkQueue(options) { | ||
| return observeQueueOperation("compact", options, () => compactWorkQueueImpl(options)); |
| if (expired) attributes["work.expired"]++; | ||
| if (ownership === "available") { | ||
| if (work.retry_not_before > time) attributes["work.retry_delayed"]++; | ||
| attributes["work.oldest_available_age_ms"] = Math.max(Number(attributes["work.oldest_available_age_ms"]), Math.max(0, time - work.enqueued)); |
| // keep_open completions have Claim-local barriers, independent of Work. | ||
| if (claim.keep_open && ["pending", "failed"].includes(claim.barrier)) attributes[`delivery.${claim.barrier}`]++; |
| if (current.snapshot) { | ||
| for (const [key, value] of Object.entries(snapshotAttributes(current.snapshot.state, current.snapshot.at))) current.attributes[`${PREFIX}${key}`] = value; | ||
| } | ||
| await require("./otlp.cjs").logSpan(`work_queue.${operation}`, current.attributes, { startMs, endMs, isError }); |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Fixed the source-backed CJS and Docs Build failures in
Fixed the source-backed CJS and Docs Build failures in |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Warning The Submitted one commit for publication (not confirmed pushed) addressing review items 2, 3, 5, 6, 7, 8, 9: derived request kinds from Not done:
Validation: the JS test runner could not install dependencies (certificate error), so vitest did not run. Sous-chef head: 16575ee
|
|
🎉 This pull request is included in a new release. Release: |



Operators need visibility into work queue backlog, retained capacity, and recovery failures without relying on debug logs. This adds correlated OpenTelemetry spans for runtime queue reads, publication, dispatch, reconciliation, and compaction.
Approach
gh-aw.work_queue.*, including Work counts and backlog age, delivery barriers, open Claims, retained reservations, capacity saturation, and paused controls.Operational considerations
Health measurements are event-driven span snapshots, not continuously sampled metrics. Dashboards should select the latest observation per queue and alert separately on stale observations. Missing snapshots mean unknown state, not an empty queue. Queue-specific telemetry excludes task payloads, result bodies, Work/Claim/dispatch/request IDs, and raw errors.
Validation
make agent-report-progressgate passed.