Skip to content

Let a model expression choose the routing level at run time - #77588

Merged
rabeckett merged 4 commits into
mainfrom
routing/dynamic-router-values
Nov 14, 2026
Merged

rabeckett merged 4 commits into
mainfrom
routing/dynamic-router-values

Conversation

@rabeckett

Copy link
Copy Markdown
Collaborator

This lets a workflow choose its routing level when the run starts, through the model: field it already has. When
the main agent's model is a single expression, such as model: ${{ inputs.level || 'router/balanced' }}, and the
workflow would otherwise be eligible for default routing on Copilot, Claude or Codex, the value may now be router,
router/economy, router/balanced or router/robust. Any other value runs as before.

The activation job resolves the expression once, in a new branch of resolve_model_routing.sh, and publishes the
value, whether routing applies, and the mode. The compiler points the main model at that output, the same way
experiment variants already reach later jobs, so the agent, threat detection and aw_info all see one decision. This
also fixes a model chosen by a pre-activation step being empty in the agent job, which today compiles with
needs-consistency warnings. The agent job writes the routed or plain AWF config at run time, as default routing
already does, with the mode filled in from activation. A routed run never passes the router value to the engine, and
threat detection falls back to its own default model.

Only expressions that activation resolves exactly as later jobs would are eligible: ones that read inputs, vars,
run-wide github properties, experiments or pre-activation outputs. An expression that reads a step's or another
job's outputs, a job-local property such as github.job, or calls hashFiles() keeps compiling exactly as it does
today. Malformed router values such as router/fast, Router/Economy or router/economy?effort=high, and values
containing line breaks, stop the run in activation.

Router values in the GH_AW_MODEL_AGENT_<ENGINE> and GH_AW_DEFAULT_MODEL_<ENGINE> variables, and pi, are left for
follow-ups, as the issue describes.

The model-routing reference gains a short "Choosing the level at run time" section, the router value restrictions
and the model alias specification (now 1.4.0) describe the new rule, and the cost-management note on model variables
is narrowed to the variables.

Tests in model_routing_dynamic_test.go run the generated activation decision step and evaluate the agent,
detection, aw_info and AWF config branches for router, concrete, empty, whitespace and malformed values. They also
cover eligibility on each engine, experiment variants including an experiment named model, the pre-activation case
without needs-consistency warnings, and the expression contexts that activation can and cannot resolve. Two
existing tests that pinned the old non-routed compilation of a whole-expression Copilot model were updated.
Recompiling the committed workflows produces no lock-file changes from this branch.

Fixes #77587

When the main agent's model is a single expression on a workflow eligible
for default routing (Copilot, Claude or Codex), it may resolve to router,
router/economy, router/balanced or router/robust. The activation job
resolves the expression once and publishes the value, routing decision and
mode; later jobs read that output, which also fixes models chosen in
pre-activation being empty in the agent job. Expressions that activation
cannot resolve as later jobs would keep the existing compilation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ðŸŸĄ Changes recommended

The routing validation checks the trimmed selector, allowing leading or trailing line breaks to bypass rejection and incorrectly enable routing.

1 open issue
What changed in this PR

This PR lets eligible Copilot, Claude, and Codex workflows choose a routing level through a model expression. Activation resolves the expression once and shares the decision with downstream jobs, including models selected by pre-activation steps.

Changes:

  • Add runtime routing selection, validation, and activation outputs.
  • Apply the shared decision to engine models, threat detection, metadata, and firewall configuration.
  • Add regression tests and document supported expressions and router-value restrictions.
File Description
pkg/​workflow/​threat_detection_helpers.go Use detection defaults when dynamic routing applies.
pkg/​workflow/​pi_engine_config_compilation_test.go Update Copilot expression expectations.
pkg/​workflow/​model_routing.go Track dynamic selectors and expose routing mode.
pkg/​workflow/​model_routing_test.go Remove the obsolete non-routed expression expectation.
pkg/​workflow/​model_routing_dynamic.go Implement eligibility checks and dynamic routing generation.
pkg/​workflow/​model_routing_dynamic_test.go Cover runtime decisions and downstream behavior.
pkg/​workflow/​model_routing_default.go Share runtime guards with dynamic routing.
pkg/​workflow/​copilot_engine_execution.go Guard Copilot’s model during routed runs.
pkg/​workflow/​compiler_yaml_step_lifecycle.go Record activation’s resolved model and routing metadata.
pkg/​workflow/​compiler_orchestrator_workflow.go Apply dynamic routing during compilation.
pkg/​workflow/​compiler_activation_context.go Publish selector and routing-mode outputs.
pkg/​workflow/​codex_engine.go Support routed and plain runtime model branches.
pkg/​workflow/​claude_engine.go Guard Claude model variables for dynamic routing.
pkg/​workflow/​awf_command_builder.go Validate and inject the runtime firewall routing mode.
docs/​src/​content/​docs/​specs/​model-alias-specification.md Specify expression-based routing in version 1.4.0.
docs/​src/​content/​docs/​reference/​model-routing.md Document runtime selection and eligibility.
docs/​src/​content/​docs/​reference/​cost-management.md Clarify restrictions on model override variables.
actions/​setup/​sh/​resolve_model_routing.sh Resolve and validate dynamic model selectors.
actions/​setup/​js/​generate_aw_info.cjs Update comments and errors for expression-based routing.

Review effort: Balanced

Leading or trailing line breaks were trimmed as whitespace before the
check, so a value such as router/robust followed by a newline still routed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rabeckett
rabeckett marked this pull request as ready for review November 14, 2026 00:06
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

🧊 Test quality analysis by Test Quality Sentinel · copilot · routed: haiku55 xhigh · ~1 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 7.2K · ◷

@github-actions

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt56 · ~4 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.5K · ◷

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated by Ponytail Reviewer for #77588 · codex · routed: gpt61 medium · ~8 AIC · ⌖ ~1 AIC · ◇ ~1 AIC · ⊞ 182 · ◷

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Specification Reviewer completed successfully!

Generated by PR Specification Reviewer for #77588 · copilot · gpt56 · ~8 AIC · ⌖ ~2 AIC · ◇ ~1 AIC · ⊞ 1.6K · ◷

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the current diff and existing review threads. I found no additional actionable correctness, security, performance, or maintainability issues on changed lines. Review was static; tests were not run per review instructions.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~4 AIC · ⌖ ~2 AIC · ⊞ 1.5K
Comment /review to run again

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (review-history)
{"id":"ldg-de40113a-2a15-46ab-ade8-b4e18d7272d9","parents":[],"payload":{"id":"ldg-b983973e-be7f-4941-aed0-1df4c2461274","operation":"append","value":{"finding_count":0,"head_sha":"f68e0798a97e95f39028142400ff3667eb05d2d5","outcome":"no_findings"}},"sha":"sha256:0307bb3a5a814df569c884b47dc6817811d61d11d2f273e3358e46066e0b62b1","timestamp":"2026-11-14T00:15:00.865Z","type":"ledger_append","version":1}
History

@github-actions

Copy link
Copy Markdown
Contributor

🏗ïļ Design Decision Gate — ADR Required

This PR makes significant changes to core business logic (674 added lines under pkg/, over the 100-line threshold) and has no implementation label. It does not have a linked Architecture Decision Record (ADR). The linked issue #77587 has a detailed design but no ADR sections, so it does not count.

📄 Draft ADR committed: docs/adr/77588-let-model-expression-choose-routing-level.md. Review and complete it before merging.

🔒 This PR cannot merge until an ADR is linked in the PR body.

📋 What to do next
  1. Review the draft ADR committed to your branch. It was generated from the PR diff and issue Let model: expressions choose the routing level at run time #77587.

  2. Complete the missing sections: confirm the alternatives (the issue's out-of-scope items are drafted as alternatives), add any others you considered, and resolve the [TODO: verify] deciders. Each Michael Nygard section (Context, Decision, Alternatives Considered, Consequences) must be present.

  3. Commit the finalized ADR to docs/adr/ on your branch.

  4. Reference the ADR in this PR body by adding a line such as:

    ADR: ADR-77588: Let a Model Expression Choose the Routing Level at Run Time (link unavailable: invalid HTTPS URL)

Once an ADR is linked in the PR body, this gate will re-run and verify that the implementation matches the decision.

🔍 Evidence used for the draft
  • pkg/workflow/model_routing_dynamic.go (new): the activation decision and the published model-routing, model-routing-mode and model-selector outputs.
  • actions/setup/sh/resolve_model_routing.sh: the run-time router-value check.
  • pkg/workflow/threat_detection_helpers.go: on routed runs, threat detection uses its own default model.
  • docs/src/content/docs/reference/model-routing.md: the new "Choosing the level at run time" section.

🏗ïļ ADR gate enforced by Design Decision Gate 🏗ïļ Â· pi · routed: haiku55 xhigh · ~3 AIC · ⌖ ~2 AIC · ⊞ 8.6K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed PR head f68e079 against Model Alias Format Specification v1.4.0 §9.4 and the model-routing reference. The implementation and tests support the intended activation-time routing behavior; no blocking conformance regression found. Synchronized docs/specs/model-alias-specification.md in commit dc16477 to make provider/eligibility limits, non-routed fallback, activation failure cases, activation-output propagation, threat-detection isolation, and model-variable restrictions normative. Publication of that specification-only commit is queued on the same-repository PR branch. Coverage was static; tests were not run per review policy.

Generated by PR Specification Reviewer for #77588 · copilot · gpt50 · ~8 AIC · ⌖ ~2 AIC · ⊞ 1.6K
Comment /review-specs to run again

@github-actions

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (spec-review-history)
{"id":"ldg-12318484-847e-4eaf-9527-d25737ab739c","parents":[],"payload":{"id":"ldg-7357025c-8216-4720-93c7-0889830a830a","operation":"append","value":{"finding_count":1,"head_sha":"f68e0798a97e95f39028142400ff3667eb05d2d5","outcome":"updated"}},"sha":"sha256:57beadcf3e5ffa469330a87356567f6c1772f8c479324e944e7a7de653e1c567","timestamp":"2026-11-14T00:18:45.777Z","type":"ledger_append","version":1}
History

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rabeckett
rabeckett requested a review from pelikhan November 14, 2026 00:42
@rabeckett

Copy link
Copy Markdown
Collaborator Author

@pelikhan this is ready for review at e314d2e. CI is green (50 checks passed, none failing), the branch is clean against main, there are no open review threads, and the Design Decision Gate ADR is finalized as Proposed.

Validation

  • Local: the full pkg/workflow suite passes apart from tests that fail the same way on main (Cloud Hypervisor, KVM, NVX, npm cooldown, conclude threat detection). Recompiling all committed workflows produces no lock-file changes from this branch.
  • Sandbox (githubnext/gh-aw-routing-sandbox, compiled with --gh-aw-ref at 212b5f0; later commits only changed the line-break check, which has a unit test, and the ADR):
    • router/economy, an empty value (falling back to router/balanced), and router/robust set by a pre-activation step each routed at that level.
    • gpt-5.6-luna, given directly and through a pre-activation step, ran unrouted on that model, which confirms the pre-activation fix.
    • Claude on Copilot routed at economy within models.allowed.
    • router/fast stopped in activation with ERR_CONFIG.
    • aw_info and usage/aw_session.jsonl record the requested value and mode, and threat detection used its own default model on routed runs.
  • Copilot review (balanced): its one finding, that line breaks were trimmed before the check, is fixed in f68e079.

The Matt Pocock Skills Reviewer run failed with report_incomplete, the same reviewer failure already tracked in #77578 for another PR.

Could you merge if it looks good?

set +o histexpand
set -euo pipefail

# Dynamic routing: the main agent's model is an expression, resolved here once and

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rabeckett consider moving to javascript in the next iteration

@rabeckett
rabeckett merged commit fad4ade into main Nov 14, 2026
77 checks passed
@rabeckett
rabeckett deleted the routing/dynamic-router-values branch November 14, 2026 01:56
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v1.3.0

pelikhan pushed a commit that referenced this pull request Nov 16, 2026
code-scanning-fixer (Copilot), static-analysis-report (Claude) and
schema-consistency-checker (Codex) now choose their routing level from
their model expression instead of an engine.model-routing block, so the
run-time routing path from #77588 runs on each engine. Scheduled runs keep
their current level; a manual dispatch defaults to it and can pick another.
code-scanning-fixer resolves the level in its pre-activation job.

TestRecompileDefaultModelRouting_EligibleLocksOnly now tells the dynamic
decision step apart from the implicit routing default.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Let model: expressions choose the routing level at run time

3 participants