Nimbus
Back to Glossary
Architecture

Eval Loops for Enterprise Agent Harnesses

Coding agents can be scored on SWE-bench and Terminal-Bench. An enterprise harness is scored on whether the executed write matched the signed payload — independent sensors, not the model’s own claim that it was done.

Nimbus Research

An eval loop for an agent harness is an independent check that the job is actually done — tests, schemas, read-backs, humans — that does not take the model’s word.

Coding harnesses already have a public language for this. SWE-bench gives an agent a GitHub issue and grades a patch with the repo’s tests. Terminal-Bench (Stanford / Laude Institute) gives an agent a machine and grades the end state of a container, not the transcript. Leaderboards even report agent + model as a pair, which is the right unit: Agent = Model + Harness. Steal that honesty. Do not steal the benchmark as your control for Salesforce.

Enterprise eval is: did the quoted CRM write match the signed payload, and can you replay who signed. A 40% Terminal-Bench score does not tell you whether Opportunity.Amount was authorised. How to evaluate an agent harness is the buying sheet. This page is the architecture of the sensor loop harness engineering keeps tightening.

Words you’ll hear

  • Oracle / verifier. The independent test. SWE-bench: FAIL_TO_PASS tests. Terminal-Bench: pytest-style assertions on container state. Enterprise: SoR read-back and payload hash.
  • Transcript eval. Grading the chain-of-thought. Useful for debugging. Insufficient as a release gate. Models claim victory; Anthropic’s long-running harness names premature victory as a failure mode.
  • Computational vs inferential sensors. Böckeler / Thoughtworks. Compiler vs LLM-as-judge. Prefer computational for invariants (schema, identity, hash). Use inferential for taste (narrative quality), never as the only SoR gate.
  • LLM-as-judge. Another stochastic component. Fine as a critic specialist. Not the signer. HITL architecture.
  • Offline vs online eval. Offline: golden jobs, replay. Online: shadow reads, canary writes, production sensors. You need both; most teams only have a demo recording.
  • Harness eval vs model eval. Changing Claude vs GPT on the same tools is model eval. Changing hooks, grants, or wiki and keeping the model is harness eval. Report them separately or you will buy a new model for a missing schema check.

Nimbus’s production sensor for writes is the quote-and-gate plus graph: governance and Lifecycle Graph. That is computational. Wiki playbooks are guides. Do not confuse a fluent Conflux draft with a passed eval.

Why coding benchmarks are the wrong outer score

They are the right inner score. Inner vs outer. Terminal-Bench’s design is even a lesson: grade the environment, not the story. The environment for RevOps is Salesforce, not a Docker VM with a hidden oracle.

Problems when you import SWE-bench into an enterprise RFP:

  • Wrong workspace. Patch quality ≠ payload authorisation.
  • Saturation and leakage. Public coding benches get gamed; your CRM schema is not a public task.
  • No identity. Benchmarks do not have a Finance signer.
  • No replay duty. A leaderboard row is not ISO 42001 evidence.
  • Wrong “done.” Tests pass on a fixture; production field still wrong.

McKinsey 2025 is about scaling work, not about bash tasks. NIST AI RMF Measure is: did the control work in your context of use. EU AI Act wants interrupt and record, not a percentile on Terminal-Bench 2.1.

Use coding benches to pick an inner harness for engineering. Use quote/replay to pick an enterprise harness. How to choose.

What an enterprise eval loop actually runs

Design it like Terminal-Bench in spirit: end state of the systems that matter, plus process constraints the company cannot waive.

Precondition sensors (feed-forward that is checkable). Required connectors attached. Roster includes the signer role. Wiki revision pinned. Spend quote accepted. If any fail, the run does not start. That is a harness eval of configuration, not of eloquence. Agent teams declaring required systems belong here.

Step sensors. Retrieval logged and in-scope (no confused-deputy dump). Tool errors do not silently retry a write. Routing used compact on extract if that is policy. Connector architecture.

Release sensors (the outer oracle).

  1. Quote is structured: object, fields, values, cardinality, hash.
  2. Named human with the right role signed that hash (write-back).
  3. Adapter executed only that payload.
  4. SoR read-back equals quote (or a documented, signed delta).
  5. Graph (or equivalent ledger) contains brief, team, policy version, signer, payload, result. Export works without the vendor. Lifecycle graph.

Negative tests. Reject path: SoR unchanged. Detached grant: write impossible. Wrong role: Hard/Critical cannot complete. These are the analogue of tests that must stay red. If your PoV never fails, you did not eval the harness. You evaluated a happy path. Proof of value.

Inferential sensors (optional, never sole). A legal specialist flags language. A critic agent scores a narrative. Useful. If they can waive a Hard gate, you added a second stochastic writer.

Human as sensor, not as folklore. HITL is a step with identity. A Slack emoji is transport. A six-month zero-reject rate is a finding: either perfect or unread.

Air Canada and the ChatGPT brief sanctions are eval-loop absences: no independent check before a system of record (policy page, court docket) changed.

Offline suites you can actually keep

You will not publish a public “CRM-bench.” You can keep a private suite:

  • Golden jobs. Anonymised or sandbox SoR. Expected quote. Expected refuse.
  • Replay. Last month’s signed write: same hash, same graph nodes.
  • Policy diffs. Change wiki cap; next run must quote the new cap or refuse.
  • Model swap. Same harness, new weights: tools still dispatch; sensors still fire. That isolates model eval. Model routing.
  • Chaos. Kill the interceptor; writes must not fail open.

Version the suite with the harness. What is an agentic workflow: the definition that ran is an input. A golden job that still “passes” after you removed the Hard gate is a broken eval, not a better model.

LangSmith, Phoenix, and similar tracing tools help observe inner and framework loops. They are not the SoR oracle. How to evaluate AI audit and observability. Tracing without a hash match is a nicer transcript.

Nimbus should be scored on whether you can automate those golden jobs on a sandbox org: attach, refuse, sign, read-back, export. Workstreams are the fixture runner. If we cannot show a red refuse, we fail this architecture too.

Building a private suite without a public CRM-bench

You do not need 2,294 GitHub issues. You need a dozen jobs that hurt when they are wrong.

Pick three families. (1) A write that must refuse (wrong role, missing field, detached grant). (2) A write that must match a fixture after sign-off. (3) A read-only job that must not call a write tool at all. Encode each as a workstream template or a scripted PoV. Run weekly. When a wiki cap changes, family (2) must fail until the quote updates — that is harness eval, not flaky CI.

Grade environment state. Terminal-Bench does not score the agent’s diary. Copy that. After the run, query the sandbox SoR. Compare to the signed hash. If you only grade the canvas prose, you are back to transcript eval. Write-back.

Keep model and harness scores apart. Swap GPT vs Claude on the same golden job: if sensors still fire and hashes still match, the harness held. If a new model skips a field and the schema sensor catches it, that is a pass for the harness and a note for the model. If the sensor does not catch it, you do not need a larger model. You need a sensor. Harness engineering.

Report agent + model. SWE-bench leaderboards already do this. Your internal dashboard should too: “Nimbus + routed compact/frontier” or “LangGraph + GPT + our interceptor.” Hiding the harness is how you buy a new model for a missing hook.

Budget the eval itself. Inferential judges on every step will cost more than the job. Thoughtworks’ advice: deterministic checks on every transaction; probabilistic judges on critical paths. Schema and identity are every-transaction. Narrative quality is not.

What you can cite externally. You can say you run refuse tests and read-backs. You cannot honestly say “we scored 83% on Terminal-Bench therefore Finance is safe.” Stanford / Laude’s paper is a CLI benchmark. Use it for CLI harnesses.

Air Canada needed a sensor on “did we emit a policy commitment.” The court docket needed a sensor on “do these citations exist.” Your suite is that instinct with fixtures. CBC; Reuters.

Online eval is the part teams skip. Offline goldens rot when the wiki moves. Shadow mode — agent quotes, human still writes, compare payloads — is an eval loop that does not need production write permission. Canary — one workstream, one object type, Hard gate, weekly refuse report — is how you learn whether operators rubber-stamp. A six-month zero-reject chart is not a quality medal. It is a sensor that may be dead. HITL.

Compare this to CI for software. You would not ship because the developer said the tests passed on their laptop. You would not replace CI with an LLM that reads the diff and scores “looks good.” You might add that LLM as a critic. Enterprise write eval is CI for mutations. Nimbus’s gate is the required check; your SoR read-back is the assertion file. If we only store the transcript, we are the laptop. Demand the assertion.

LangSmith and similar are the right place to debug traces for framework and inner loops. Export those traces into your golden runner; do not let the tracing UI become the only evidence for audit. Auditors will ask for the hash and the signer. How to evaluate AI audit.

Do not wait for a consortium bench. Your suite is a competitive advantage if it encodes your caps and objects. Share the method (refuse, read-back, replay) in the RFP. Keep the fixtures. Vendors who cannot run against your sandbox are not ready for your SoR, however they score on Terminal-Bench.

How this shows up in Nimbus

The product’s eval spine is: NTU quote before the run, scoped retrieval, canvas artefacts, write quotes, tiered gates, graph commit. Sensors you should still add: your own SoR read-back in the sandbox, your own golden files (the analogue of pytest). The platform cannot know your “correct Amount” without your oracle. Terminal-Bench ships oracles per task. You must ship oracles per job. That is harness engineering, not a missing model.

Questions people actually ask

Can we use an LLM-as-judge on the quote?

As a critic, yes. As the only signer, no. Computational match of fields is cheap and stable.

Do we wait for an industry “enterprise SWE-bench”?

You would still need private oracles. Start this quarter with sandbox read-backs.

Our vendor only shares SWE-bench.

File as inner evidence. Demand refuse/replay for outer. Evaluate the harness.

Is tracing enough for ISO 42001?

Traces help Measure. You still need Manage: a control that fired. A pretty trace of an unsigned write is a better incident report.

How does this relate to agent teams vs single agents?

Teams add hand-off evals (typed artefacts). They do not replace the write oracle. How to evaluate agent teams.

Agent harness architecture. How to evaluate write-back governance. What is harness engineering.

How to evaluate AI audit and observability and Write-back governance for systems of record.

Sources

Related guides

Next

Agent Harness Architecture

See what governed AI looks like on your stack.

Connect your tools, run a workstream, and keep every decision on your ledger - free for 7 days.

Get started for freeTalk to our team