How to Evaluate Loop Engineering

Evaluating loop engineering means testing whether a vendor can run standing orders the way operators actually work — skip when nothing changed, record quiet outcomes, reuse recipes, notify a named roster — not whether a demo chat looked fluent. NIST’s AI Risk Management Framework Measure and Manage functions assume you can observe outcomes and tighten controls after harm. A loop without a run page is not observable. It is email archaeology.

This sheet is vendor-agnostic. Paste it into an RFP. Run it on an incumbent automation suite, a new “agentic” platform, or a homegrown scheduler. The object under test is the standing order: a compiled recipe that starts on a signal and leaves a record. If the vendor cannot show that object, stop scoring adjectives.

Three nouns vendors conflate on slide one:

• Loop. A compiled standing order with triggers and a run page. See what is loop engineering and six things that start a loop.
• Eval loop. Independent verification that a job finished — tests, read-back, signer — not a trigger. See eval loops for enterprise agent harnesses.
• Agentic workflow. A designed interactive sequence with stops. See what is an agentic workflow. Score it with how to evaluate an agent harness. That sheet scores the interactive harness. This sheet scores repeat automation.

If your RFP only asks context-window size, you will buy a model for a Monday cron job. ISO/IEC 42001 wants named actors and records around AI systems. “The bot ran” is not a record. “The assistant said it was done” is not a record.

Why loop evaluation fails

Teams reuse copilot scorecards because those cards are already in the drawer. They fail in predictable ways:

1. The demo pass. A model summarises a file beautifully. No trigger, no skip, no version. You have scored reading, not standing orders.
2. The integration pass. “We connect to Salesforce.” No run page, no quiet outcome, no roster. You have scored a connector, not an operating object.
3. The personal-automation pass. A chain fires. Outcomes scatter across personal inboxes. No organisational recipe. You have scored glue, not loop engineering.

McKinsey’s 2025 State of AI reports wide experimentation and narrower scale. Loop engineering is how repeat work scales: compile what worked, trigger it reliably, record what happened. High-performing organisations in that survey are more likely to redesign workflows, not merely sprinkle assistants on the current process. This RFP is a redesign test.

Red flags you can mark in the room:

• Outcomes live only in chat transcripts
• “Success” when there was nothing to do — but Finance still got paged
• Reuse means “find Sarah’s Slack thread from Q2”
• Every write goes through a model — even fixed journal templates
• No pause — cancel kills state without a resumable run
• Notifications go to channels, not to a roster attached to the job

Stanford HAI’s AI Index is a useful external reminder that capability is not the scarce input. You are not scoring whether the model can write a polite email. You are scoring whether Tuesday’s job exists when the author is on leave.

The RFP sheet — eight tests

Each test has a why, a when, a thing to do, and a thing to refuse. Run them in order if you are short on time: skip semantics first, then run page, then reuse. A fluent demo that fails test 1 is not a standing order.

1. Can it skip when nothing changed?

Run the loop on unchanged inputs. The run page should say skipped or no op — with timestamp and recipe version — not green-check spam.

Why it matters: month-end with zero exceptions is success. Bots that “succeed” on empty tables train operators to ignore alerts. See six things that start a loop for triggers that should dedupe.

When to insist: any scheduled or event-driven job that will run in unattended hours.

What to do: show ten consecutive skipped runs. Show alert volume — ideally zero.

What to refuse: a “success” email on an empty extract, or a skip that exists only as the absence of mail. Absence is not a record.

2. Can it record quiet outcomes?

Quiet success is a first-class outcome: ran, nothing to write, roster optionally notified at digest frequency.

Why it matters: auditors and controllers ask what happened on the twelfth — including days nothing moved. Silence without a record is indistinguishable from failure. NIST’s Measure function is not optional because the week was quiet.

When to insist: regulated work, shared-service work, anything a second person will reconstruct.

What to do: export skipped runs for a month without vendor engineering.

What to refuse: a professional-services quote to produce last month’s quiet days. If export is a project, observation is not a product feature.

3. Can you parameterise a write without a model?

Show a loop that posts or quotes a fixed-shape payload — accounts, amounts, CRM fields — from structured input, with no language model in the path.

Why it matters: many operational writes are templates, not essays. If the vendor routes everything through chat, you are paying inference tax on deterministic work and importing non-determinism into close. Compare loop vs workflow vs agent and a loop is not an agent.

When to insist: journals, stage updates, status writes, any payload a controller could have typed from a spreadsheet.

What to do: disable the model. Does the loop still quote the write and wait on sign?

What to refuse: “the model is more flexible” as an answer to a fixed schema. Flexibility on a journal line is a defect.

4. Can it pause and resume cleanly?

A loop waiting on a file, a signer, or an external system should pause with visible state — not vanish into a thread.

Why it matters: close week spans days. Operations must distinguish “waiting on a person” from “broken.” What is human-in-the-loop AI applies to standing orders too.

When to insist: any recipe that crosses a night, a weekend, or a named approver.

What to do: pause mid-run. Attach the missing file. Resume without restarting from scratch unless you choose to. Keep the same run identifier.

What to refuse: cancel-as-pause. If state dies, you do not have a pause. You have a restart with extra steps.

5. Can it notify the roster — not a copied channel?

Notifications should target the people on the job — controller, RevOps, counsel — with a link to the run page. See what is an AI workstream for the job-object idea, and how to evaluate collaborative AI for the roster questions.

Why it matters: Slack channels rot when people leave. Rosters follow the job. A copied channel is how a departed contractor keeps getting close packs, and how the new controller never does.

When to insist: any loop that another department will act on.

What to do: remove one person from the roster. Prove they stop receiving loop notifications without creating a new automation.

What to refuse: “we’ll update the webhook.” If membership is not data, notification is folklore.

6. Can you reuse a recipe without copying the old Slack channel?

Clone the loop — triggers, steps, gates, notify rules — into a new team or region without re-prompting from memory.

Why it matters: loop engineering is an organisational capability, not hero prompts. If reuse requires export to JSON and a services quote, note the tax. Thoughtworks’ operating-system framing is useful here: ownership and durable state belong to the job, not to the person who first described it.

When to insist: any recipe you will need in a second business unit within a year.

What to do: stand up the same loop for a second team in under one hour — operator-led.

What to refuse: a clone that copies the prompt but drops the skip rules, the signer, or the run-page contract. That is a new folklore, not reuse.

7. Does every trigger land on the same run page?

Schedule, file, data change, drop, ping, run now — several doors, one outcome surface. See six things that start a loop.

Why it matters: operators should not learn six UIs. Audit should not merge six log formats. If the scheduled close and the emergency rerun do not look like the same object, you will get two classes of evidence.

When to insist: as soon as a team has more than one start condition for the same recipe.

What to do: fire two different triggers against the same recipe. Show both run pages side by side.

What to refuse: a “manual” path that writes with weaker gates than the scheduled path. Urgency is not a policy exception.

8. Can you refuse a write and prove the system of record unchanged?

Even loops that only draft should demonstrate fail-closed behaviour when signers reject. Loops that write must show quote → sign → execute → read-back. See what is write-back governance.

Why it matters: this is where loop evaluation meets harness evaluation. How to evaluate an agent harness test one — stop an action — applies to automated writes too. What is AI governance is the category language.

When to insist: before any production write, including “just a status field.”

What to do: show a rejected payload. Show the CRM or ERP unchanged. Show the rejection on the run page, with who rejected and which policy version applied.

What to refuse: a write that cannot be shown in a quoted form. A paragraph the model later “applies” is not a payload.

RFP questions — paste these

These are the short versions you can drop into a vendor questionnaire. They are not vendor-specific. They are not even AI-specific. They are standing-order tests.

1. Show ten skipped runs with timestamps and recipe version.
2. Show a write path with no model call — structured in, quoted out.
3. Pause a run for 48 hours; resume; show a continuous run identifier.
4. Clone a loop to a second team without re-entering prompts.
5. Trigger the same recipe via schedule and via file drop; compare run pages.
6. Remove a roster member; prove notifications stop.
7. Reject a quoted write; prove the system of record unchanged; show who rejected.
8. Where do outputs live if email is down — still on the run page?

Add the harness sheet when loops call agents or share connectors with interactive work. Add how to evaluate collaborative AI when the output is a signed pack rather than a silent write.

Proof of value — one week

Do not spend the week watching a prepared demo. Spend it on one real job.

Day 1–2: Pick one repeat job — weekly pipeline summary, bank file intake, redline folder watch. Name the trigger your team already watches. Write the skip condition in a sentence a controller would accept.

Day 3: Run ten times with empty or stale inputs. Demand skipped run pages. If you cannot get them, the rest of the week is theatre.

Day 4: Run once with a real change. Confirm roster notification links to the run — not a pasted screenshot. Confirm an independent reader can open the run without the author.

Day 5: Clone to a second roster or region. Time it. Attempt a refused write. Confirm the system of record did not move.

Pass criteria: skip semantics, run page, roster notify, reuse, and one refused or unsigned write that did not land. Fail any one and you do not have loop engineering. You have a demo that will not survive the first quiet week.

How this differs from harness evaluation

You need both sheets if the four pillars describe your stack — Automate (loops) plus Collaboration (agents on workstreams) under governance. Loop engineering vs harness engineering explains the crafts. This page and the harness page are how you buy them separately.

Loop vs RPA is the adjacent comparison if incumbents sell bots. Do not let an RPA success-email farm satisfy test 1. “The script finished” is not a skipped run.

Department lenses

Use the same eight tests. Change the job you bring to the POV.

Finance — scheduled close packs, parameterised journals, controller on the roster. Skip on a week with no exceptions. See loops for finance and planning.

RevOps — stage-triggered hygiene, skip when fields are unchanged, notify the people who own the stage definition. See loops for revenue operations.

Legal — inbound redlines, notify counsel, no send without a workflow gate. The loop starts the job; the workflow owns the stop. See loops for legal and compliance.

Related reading: what is collaborative AI, a loop is not an agent, and what is an enterprise AI operating system when you need the kernel metaphor rather than the RFP sheet.

How this shows up in Nimbus

Nimbus Loops are one implementation of the standing-order object this sheet scores: triggers, recipes, run pages, roster notify, and governance gates on a workstream. You can run the eight tests there. You should also run them on whoever else claims “unattended agents” or “intelligent automation.”

Nimbus is not loop engineering. Loop engineering is whether your organisation can compile repeat work, skip quietly, and leave a record a second person can open. The product should make those tests boring. If a Nimbus demo — or any demo — cannot show ten skipped runs and one refused write, treat it as a copilot evaluation and score it on the harness sheet instead.