Why agents need skills: doing the job the way you want, unsupervised

Autonomy is conformance plus success, with no one watching

We measured what it takes for an agent to follow a Standard Operating Procedure unsupervised: five agent configurations, three public benchmarks with deterministic oracles, 255 runs per configuration, all on claude-haiku-4-5. The short version: ship the SOP as a skill the agent can discover by itself, and single-turn task success rises by +0.35 (95% CI [+0.26, +0.45], p<0.001) while multi-turn procedure conformance on τ²-bench climbs from roughly 50 to 59% up to about 90%. The delivery channel (skill in context versus tool retrieval) is a wash. And bolting heavyweight run-tracking on top buys nothing except a 2 to 4× tool-call bill.

Three properties frame the result. Conformance is whether the agent does the job the way you want: it follows the approved SOP, the right steps in the right order, honoring the constraints. Success is whether it produces the outcome you want, the correct result. Autonomy is achieving both with no person in the loop, and that last part is stricter than it sounds. An agent that merely runs unattended while conforming poorly or landing on a different outcome is automatic; an agent that conforms and succeeds unsupervised is autonomous. Real autonomy requires the agent to obtain and apply the right SOP by itself.

The study exists because we kept hitting the same wall in our own work. Wire an agent up to every tool it needs, point it at a real workflow, and it will do a job, just not always yours. It skips the approval step your compliance team insists on. It refunds a customer who was outside the policy window. It reaches a plausible-looking outcome by a path nobody signed off on, and since nobody was watching, nobody caught it. The fix, we suspected, was already sitting in most companies: the Standard Operating Procedure. The open question was how the right SOP reaches the agent's context at the right moment with no human choosing it, and whether the delivery mechanism (a skill, a tool, or an MCP prompt) changes the outcome.

First we show why delivery decides autonomy, then how we ran the test, then what supplying the SOP buys, then two negative results that save you money. One thing up front, because it shapes how you should read the rest: this study was sized for budget and compute, not for statistical power. The single-turn success gain and the conformance jump on τ² are solid; the multi-turn magnitudes are directional. We flag which is which as we go, and there's a whole section on it before the numbers.

Why delivery decides autonomy

An LLM with tool access can already do a lot. What it doesn't come with is your order of operations: which calls, in what sequence, under which constraints. For a one-off task that's fine. The agentic loop investigates, tries things, and usually lands somewhere reasonable. Repeated operational work is different. If the agent has to rediscover the plan by trial and error on every run, you get a different path each time, and some of those paths skip steps you care about.

So the binding constraint is that the right SOP has to reach the agent's context at the right moment, with no human choosing it. That is where the delivery mechanism decides the outcome, and where the difference between a skill and an MCP prompt stops being academic.

A skill carries auto-discovery. The runtime surfaces each skill's description; when a task arrives, the agent itself matches it to the relevant skill and loads the full procedure on demand. No human picks it. Standing context stays small, since only the descriptions are resident until something matches, so the approach keeps working as the library grows: a design that holds at ten SOPs still holds at a thousand. An MCP prompt has no auto-discovery. A prompt is a user-invoked template that a person selects from a menu. With no user present, the procedure never enters context and the agent improvises.

This isn't only our framing. An active working group is adding skills to the Model Context Protocol precisely because tools and prompts alone don't let an agent discover and apply the correct procedure by itself. A tool can carry a procedure; nothing about a tool or a prompt makes the agent reach for the right one unprompted. That gap is the reason skills exist as a distinct primitive, and we come back to where the spec stands at the close.

How we tested it

We built a ladder of five arms, climbing from no procedure to an auto-discovered procedure with run-tracking. Each arm is one agent per domain, scoped to that domain's tools, graded by the benchmark's own deterministic oracle.

The skills arm is delivered as a filesystem SKILL.md the runtime auto-discovers, the same mechanism a production agent would use. The two planning arms retrieve a hand-authored, completeness-checked stepwise procedure through an in-process MCP tool. Critically, the procedure text is identical across both planning arms and is track-agnostic, so the planning-tracking − planning contrast isolates the run-tracking protocol alone. Nothing else moves.

We ran this across three public benchmarks, each with a deterministic oracle that checks the agent's actual behavior rather than asking a model to grade it. They span single-turn and multi-turn, with a simulated user driving the conversation in the multi-turn cases.

Scope, sampling, and confidence: read this before the numbers

This is an engineering study, not a peer-reviewed benchmark, and the design was sized for budget and compute. Treat the numbers as directional evidence, not definitive measurement. Four things constrain what we can claim, and we'd rather you hear them from us than infer them later.

We subsampled hard. Each benchmark ships far more tasks than we ran: SOP-Bench has ~2,154, SOPBench ~830, τ² ~278. We evaluated a deterministic 10-tasks-per-domain subsample (τ² at 15/domain), which is 45 to 140 runs per arm. Multi-turn cells cost roughly 50 to 200K tokens each, because every task runs an agent plus an LLM user-simulator loop, and they share an org rate limit. A full-pool × 5-arm × repeats sweep simply wasn't affordable. Ten per domain is a deliberate coverage/cost trade-off, not a power calculation.

We ran each cell once (R=1). Cells weren't repeated, so cross-run reliability (pass^k) is untested, and each point estimate carries run-to-run stochastic noise on top of sampling noise. One model tier. Everything is claude-haiku-4-5, with no cross-model comparison. I'd expect the absolute effects, and especially the externalized-state trade-offs, to move with stronger models.

The statistics are paired. Every contrast is computed within (benchmark, domain, task), and we report paired-bootstrap 95% CIs (≥10k resamples) plus McNemar's test on discordant pairs. We only call a result significant when the CI excludes zero. At these sample sizes, single-task differences below ~10 percentage points sit inside the noise. When we quote a per-arm mean gap that looks bigger than the paired contrast, trust the paired number; it is the honest one.

The headline: the skill takes conformance to ~90%

This is the metric the whole study exists for. Conformance is the fraction of the prescribed procedure the agent actually follows. Higher means closer to the SOP.

Look at τ². The skill arm follows roughly 90% of the prescribed policy actions, against ~59% with domain context only and ~50% with nothing. Put differently: without the SOP the agent skips roughly every other prescribed action; with it, one in ten. That's a paired, statistically significant conformance gain of +0.16 over baseline (CI [+0.06, +0.28]) and +0.28 over null (CI [+0.16, +0.40]). The raw per-arm-mean gap looks larger (closer to +0.32/+0.40) because the arms don't run a byte-identical task set. The paired contrast, computed the same way as our success contrasts, is roughly half that, and it's the number to quote.

On SOPBench the direction is identical but the effect is small and not significant (paired +0.04, CI [−0.03, +0.11]; most tasks tie). So the honest summary is: giving the agent the SOP measurably improves how faithfully it follows the procedure, clearly on τ² and inconclusively on SOPBench, and it degraded conformance nowhere in our runs.

Conformance also means not acting when you shouldn't. SOPBench labels each task as should-act or should-decline, and the SOP improves both sides. Correct declines on should-decline tasks rise from 0.13 (null) to 0.26 (skills), and execution on should-act tasks from 0.03 to 0.06. The absolute rates are low and the subsets small (n≈32 act / 38 decline, SOPBench only), so read this as directional. But it points the right way: an agent given the SOP is better both at doing the job and at correctly refusing out-of-policy requests. That second half is exactly what you need before you trust it unattended.

† SOP-Bench is single-turn and has no action-sequence oracle, so its only conformance proxy is tool coverage: did the agent call the prescribed domain tools. Single-turn tasks force those calls regardless of the SOP, so the proxy is uninformative here; it even reads backward (baseline 0.75 > skills 0.52, because the skill sometimes decides with fewer tool calls). Real conformance is measured on the multi-turn benchmarks, whose oracles actually check whether the agent followed the procedure. There, the SOP clearly makes the agent conform.

The outcome: success moves with conformance

Task success rate (TSR) is the benchmark's native measure of whether the agent produced the correct final result.

The key contrast is skills − baseline = +0.35 on SOP-Bench (CI [+0.26, +0.45], p<0.001, significant), +0.04 on SOPBench (not significant), and +0.16 on τ² (not significant). Read that as: supplying the SOP drives the outcome too, large and significant in the clean single-turn regime, directional under the noisier multi-turn one. Conformance and success move together: with the skill, the agent both follows the procedure and lands on the right answer more often.

Be clear-eyed about the multi-turn magnitudes, though. The τ² success lift is real in direction, but at n=45 its size sits inside the noise band. We'd need more tasks per domain, repeats, and a second model tier to move that from "directional" to "measured." The single-turn success gain, by contrast, is one of the two claims I'd stand behind without hedging.

Two negative results that save you money

The most useful findings here are the ones that tell you what not to build.

The delivery channel is a wash

Compare planning (SOP pulled as a tool) against skills (SOP auto-discovered in context): +0.03 on SOP-Bench, −0.01 on SOPBench, −0.04 on τ². Every one of those sits inside the noise, on both conformance and success. Once the procedure is available to the agent, the channel shows no detectable difference in outcome at this power. So the case for skills rests on discoverability rather than higher scores: the agent selecting and applying the right procedure with no human invocation. A skill has that property built in; a user-invoked prompt has no path to it. We didn't run a user-invoked prompt arm, because in an unsupervised run there is no user to invoke it, which is exactly the point.

Externalized run-tracking is net-negative

Now the cost side. Here are the mean tool calls per run. Lower is leaner.

The auto-discovered skill is the leanest delivery: it applies the procedure with the fewest tool calls and keeps standing context small, since only descriptions are resident until a skill matches. Bolting an externalized run-tracking protocol on top, the planning-tracking arm, buys no additional conformance or success (the planning-tracking − planning success contrast is −0.06 / +0.01 / +0.09, all not significant) while costing ~2 to 4× the tool calls. Discovery plus on-demand delivery did the same job leaner; the state machinery added cost and nothing else at these horizons.

The caveat worth naming: our tasks are short, 4 to 15 steps. The strongest theoretical case for externalized tracking is the long composite session where in-context self-tracking degrades, and that regime is untested here. If your workflows are genuinely long-horizon, the honest reading is narrower: don't pay for tracking until you can show the horizon demands it.

What we'd instrument next

Two honest gaps sit at the center of the skills argument, and we're naming them because they're where the next round of evidence has to come from. This experiment shows that the SOP helps and that the delivery channel among agent-driven options is a wash. It does not empirically show skills beat MCP prompts (we ran no user-invoked prompt arm), nor does it measure auto-discovery accuracy. The case for skills specifically rests on the auto-discovery design property, argued rather than measured. We'd rather say that plainly than let it be overread.

The highest-value addition is routing / discovery accuracy: when a task arrives, did the agent auto-discover and load the correct SOP from the library, with no user? That measures auto-discovery directly, isolating a real failure mode of scaling the library: mis-routing. We don't log selected-versus-correct skill today; we should. Second is reliability (pass^k, R≥5). We ran one repeat per cell, so consistency across stochastic runs is untested, and reliability is, I suspect, both the natural place for externalized state to finally pay off and the thing that matters most for unsupervised operation, where a single rare wrong run is expensive.

Package the SOP as a skill

Pulling it together. Autonomy means the agent does the job your way and gets your outcome, unsupervised, so the two metrics worth tracking are conformance and success rather than uptime or turn count. The SOP is what delivers both: with the SOP as a skill, conformance improves significantly on τ² (the skill arm follows ~90% of prescribed actions versus 50 to 59% without) and success rises with it, with the honest caveat that the SOPBench conformance gain is small and the multi-turn success magnitudes are directional. In our runs, the limit was whether the right procedure reached the agent's context at all. Raw model capability rarely came up.

So the "do this Monday" version: take the procedures that define how your operators actually work and package them as auto-discoverable skills rather than prompts a human has to pick or bare tools buried in a server. Measure your agents on whether they follow those procedures and reach the right outcome, both, unsupervised. Resist the urge to build a run-state machine until you can show the horizon demands one. As for which work an agent can own in the first place and which stays human, that depends on which stage of the work you're looking at, the companion argument in The Tideline.

Corollary: don't optimize for fewer turns

One result is tempting to misread. On SOPBench, the SOP cuts the follow-up turns a user has to supply from about 6 down to 0.5 to 1.8, so the agent runs with far less hand-holding. That is a side effect, though, and a dangerous target: τ² tasks require eliciting information from the user, so the more capable arms actually take more turns, and an agent that needs zero human turns while doing the job the wrong way is exactly the failure this study exists to catch. A team optimizing for fewer turns would flag its best agents as its worst ones.

Corollary: the MCP spec is converging on this mechanism

The skills primitive is being standardized in the open. The Skills Over MCP working group (formed as an interest group in February 2026, a full working group that April, co-led by maintainers from Anthropic and Nordstrom, with participants from Google, GitHub, AWS, Databricks, Bloomberg, and Saxo Bank) is specifying exactly the mechanism our skills arm relied on. Its current direction, SEP-2640, serves skills over MCP's existing Resources primitive: the host loads only each skill's name and description into context at startup, and the model pulls the full instructions only when a task matches. As of July 2026 the draft and its reference implementation were both still in review, with skills/list and skills/get methods and per-file integrity checks added through community review. None of it is final, but the direction matches what our benchmark points at: a discoverable, self-applied procedure. If you're packaging SOPs now, follow the SEP-2640 thread so your skill library lands on the standard rather than beside it.