Why Agents Need Skills: Doing the Job the Way You Want, Unsupervised

Repeated work, rediscovered every run

An LLM with tool access can already do a lot. What it doesn't come with is your order of operations: which calls, in what sequence, under which constraints. For a one-off task that's fine. The agentic loop investigates, tries things, and usually lands somewhere reasonable. Repeated operational work is different. If the agent has to rediscover the plan by trial and error on every run, you get a different path each time, and some of those paths skip steps you care about.

We kept hitting this wall in our own work. Wire an agent up to every tool it needs, point it at a real workflow, and it will do a job—just not always yours. It skips the approval step your compliance team insists on. It refunds a customer who was outside the policy window. It reaches a plausible-looking outcome by a path nobody signed off on, and since nobody was watching, nobody caught it. The fix, we suspected, was already sitting in most companies: the Standard Operating Procedure. The open question was how to get the right SOP into the agent's context at the right moment with no human choosing it, and whether the delivery mechanism (a skill, a tool, or an MCP prompt) changes the outcome.

So we tested it. Three benchmarks, deterministic oracles, five arms from no SOP at all to an auto-discovered SOP with externalized run-tracking, all on claude-haiku-4-5. The short version: giving the agent the SOP raises single-turn task success by +0.35 (95% CI [+0.26, +0.45], p<0.001) and pushes multi-turn procedure conformance from ~50–59% up to ~90%. The delivery channel (skill in context versus tool retrieval) is a wash. And bolting heavyweight run-tracking on top buys nothing except a 2–4× tool-call bill. The rest of this post is the evidence: how we ran it, what supplying the SOP buys, two negative results that save you money, and how far to trust each claim.

One thing up front, because it shapes how you should read the rest: this study was sized for budget and compute, not for statistical power. The single-turn success gain and the conformance jump on τ² are solid. The multi-turn magnitudes are directional. We'll flag which is which as we go, and there's a whole section on it before the numbers.

Autonomy is conformance plus success, with no one watching

Four properties matter here, and they're exactly what we set out to measure. Conformance is whether the agent does the job the way you want: it follows the approved SOP, the right steps in the right order, honoring the constraints. Success is whether it produces the outcome you want, the correct result. Autonomy is achieving both with no person in the loop. That last word is stricter than it sounds, and it's worth one definition we'll use for the rest of the piece: an agent that merely runs unattended while conforming poorly or landing on a different outcome is automatic; an agent that conforms and succeeds unsupervised is autonomous. Real autonomy requires the agent to obtain and apply the right SOP by itself.

And then there is efficiency: doing all of that without wasteful overhead, in a way that keeps scaling as your SOP library grows. Auto-discovery keeps only the skill descriptions resident and loads a full procedure into context only when it matches, rather than carrying every SOP at once. A benchmark that ignored efficiency would miss the difference between a design that works with ten SOPs and one that still works with a thousand. As the tool-call counts later show, it would also miss the difference between a lean skill and a heavyweight run-tracking rig that buys nothing.

The binding constraint is that last requirement: the right SOP has to reach the agent's context at the right moment, with no human choosing it. That is where the delivery mechanism decides the outcome, and where the difference between a skill and an MCP prompt stops being academic.

A skill carries auto-discovery. The runtime surfaces each skill's description; when a task arrives, the agent itself matches it to the relevant skill and loads the full procedure on demand. No human picks it. Standing context stays small, since only the descriptions are resident until something matches, so the approach scales as the library grows. An MCP prompt has no auto-discovery. A prompt is a user-invoked template that a person selects from a menu. With no user present, the procedure never enters context and the agent improvises.

This isn't only our framing. It's why there's an active working group to add skills to the Model Context Protocol: tools and prompts alone don't let an agent discover and apply the correct procedure by itself. A tool can carry a procedure, sure. But nothing about a tool or a prompt makes the agent reach for the right one unprompted. That gap is the reason skills exist as a distinct primitive.

How we tested it

We built a ladder of five arms, climbing from no procedure to an auto-discovered procedure with run-tracking. Each arm is one agent per domain, scoped to that domain's tools, graded by the benchmark's own deterministic oracle.

The skills arm is delivered as a filesystem SKILL.md the runtime auto-discovers, the same mechanism a production agent would use. The two planning arms retrieve a hand-authored, completeness-checked stepwise procedure through an in-process MCP tool. Critically, the procedure text is identical across both planning arms and is track-agnostic, so the planning-tracking − planning contrast isolates the run-tracking protocol alone. Nothing else moves.

We ran this across three public benchmarks, each with a deterministic oracle that checks the agent's actual behavior rather than asking a model to grade it. They span single-turn and multi-turn, with a simulated user driving the conversation in the multi-turn cases.

Scope, sampling, and confidence—read this before the numbers

This is an engineering study, not a peer-reviewed benchmark, and the design was sized for budget and compute. Treat the numbers as directional evidence, not definitive measurement. Four things constrain what we can claim, and we'd rather you hear them from us than infer them later.

We subsampled hard. Each benchmark ships far more tasks than we ran—SOP-Bench has ~2,154, SOPBench ~830, τ² ~278. We evaluated a deterministic 10-tasks-per-domain subsample (τ² at 15/domain), which is 45–140 runs per arm. Multi-turn cells cost roughly 50–200K tokens each, because every task runs an agent plus an LLM user-simulator loop, and they share an org rate limit. A full-pool × 5-arm × repeats sweep simply wasn't affordable. Ten per domain is a deliberate coverage/cost trade-off, not a power calculation.

We ran each cell once (R=1). Cells weren't repeated, so cross-run reliability (pass^k) is untested, and each point estimate carries run-to-run stochastic noise on top of sampling noise. One model tier. Everything is claude-haiku-4-5—no cross-model comparison. I'd expect the absolute effects, and especially the externalized-state trade-offs, to move with stronger models.

The statistics are paired. Every contrast is computed within (benchmark, domain, task), and we report paired-bootstrap 95% CIs (≥10k resamples) plus McNemar's test on discordant pairs. We only call a result significant when the CI excludes zero. At these sample sizes, single-task differences below ~10 percentage points sit inside the noise. When we quote a per-arm mean gap that looks bigger than the paired contrast, trust the paired number—it's the honest one.

The headline: the skill takes conformance to ~90%

This is the metric the whole study exists for. Conformance is the fraction of the prescribed procedure the agent actually follows. Higher means closer to the SOP.

Look at τ². The skill arm follows roughly 90% of the prescribed policy actions, against ~59% with domain context only and ~50% with nothing. Put differently: without the SOP the agent skips roughly every other prescribed action; with it, one in ten. That's a paired, statistically significant conformance gain of +0.16 over baseline (CI [+0.06, +0.28]) and +0.28 over null (CI [+0.16, +0.40]). The raw per-arm-mean gap looks larger (closer to +0.32/+0.40) because the arms don't run a byte-identical task set. The paired contrast, computed the same way as our success contrasts, is roughly half that, and it's the number to quote.

On SOPBench the direction is identical but the effect is small and not significant (paired +0.04, CI [−0.03, +0.11]; most tasks tie). So the honest summary is: giving the agent the SOP measurably improves how faithfully it follows the procedure, clearly on τ² and inconclusively on SOPBench, and it never degrades conformance anywhere.

Conformance also means not acting when you shouldn't. SOPBench labels each task as should-act or should-decline, and the SOP improves both sides. Correct declines on should-decline tasks rise from 0.13 (null) to 0.26 (skills), and execution on should-act tasks from 0.03 to 0.06. The absolute rates are low and the subsets small (n≈32 act / 38 decline, SOPBench only), so read this as directional. But it points the right way: an agent given the SOP is better both at doing the job and at correctly refusing out-of-policy requests. That second half is exactly what you need before you trust it unattended.

† SOP-Bench is single-turn and has no action-sequence oracle, so its only conformance proxy is tool coverage: did the agent call the prescribed domain tools. Single-turn tasks force those calls regardless of the SOP, so the proxy is uninformative here; it even reads backward (baseline 0.75 > skills 0.52, because the skill sometimes decides with fewer tool calls). Real conformance is measured on the multi-turn benchmarks, whose oracles actually check whether the agent followed the procedure. There, the SOP clearly makes the agent conform.

The outcome: success moves with conformance

Task success rate (TSR) is the benchmark's native measure of whether the agent produced the correct final result.

The key contrast is skills − baseline = +0.35 on SOP-Bench (CI [+0.26, +0.45], p<0.001, significant), +0.04 on SOPBench (not significant), and +0.16 on τ² (not significant). Read that as: supplying the SOP drives the outcome too, large and significant in the clean single-turn regime, directional under the noisier multi-turn one. Conformance and success move together. With the skill, the agent does the job the right way and lands on the right answer. Without it, it reliably does neither.

Be clear-eyed about the multi-turn magnitudes, though. The τ² success lift is real in direction, but at n=45 its size sits inside the noise band. We'd need more tasks per domain, repeats, and a second model tier to move that from "directional" to "measured." The single-turn success gain, by contrast, is one of the two claims I'd stand behind without hedging.

Two negative results that save you money

The most useful findings here are the ones that tell you what not to build.

The delivery channel is a wash

Compare planning (SOP pulled as a tool) against skills (SOP auto-discovered in context): +0.03 on SOP-Bench, −0.01 on SOPBench, −0.04 on τ². Every one of those sits inside the noise, on both conformance and success. Once the procedure is available to the agent, the channel shows no detectable difference in outcome at this power. So the case for skills isn't higher scores. It's discoverability: the agent selecting and applying the right procedure with no human invocation. A skill has that property built in; a user-invoked prompt has no path to it. We didn't run a user-invoked prompt arm, because in an unsupervised run there is no user to invoke it, which is exactly the point.

Externalized run-tracking is net-negative

Now the cost side. Here are the mean tool calls per run. Lower is leaner.

The auto-discovered skill is the leanest delivery: it applies the procedure with the fewest tool calls and keeps standing context small, since only descriptions are resident until a skill matches. Bolting an externalized run-tracking protocol on top, the planning-tracking arm, buys no additional conformance or success (the planning-tracking − planning success contrast is −0.06 / +0.01 / +0.09, all not significant) while costing ~2–4× the tool calls. Discovery plus on-demand delivery is the efficient design. Heavyweight state machinery is pure overhead at these horizons.

The caveat worth naming: our tasks are short, 4–15 steps. The strongest theoretical case for externalized tracking is the long composite session where in-context self-tracking degrades, and that regime is untested here. If your workflows are genuinely long-horizon, don't read this as "never track." Read it as "don't pay for tracking you can't yet show you need."

Don't optimize for fewer turns

One result is tempting to misread, so let's handle it head-on. On SOPBench, the SOP cuts the follow-up turns a user has to supply from ~6 (no SOP) down to ~0.5–1.8. The agent runs with far less hand-holding. That's nice, and it's also a side-effect rather than the goal.

An agent that needs zero human turns yet does the job the wrong way, or gets the wrong answer, is exactly the failure this study exists to catch. τ² is the reminder that keeps you honest here: its tasks require eliciting information from the user, so the more capable arms actually take more turns. If you optimized for "fewer turns," you'd wrongly flag the best agents as the worst ones.

The MCP spec is converging on exactly this mechanism

We've leaned on the phrase "an active working group is adding skills to MCP" a couple of times now. Here is what that group is actually building: a spec converging on precisely the mechanism our skills arm relied on (description-first discovery, full instructions loaded on demand) rather than on the prompts primitive.

The Skills Over MCP working group formed as an interest group in February 2026 and became a full working group that April, co-led by a maintainer from Anthropic and one from Nordstrom, with participants from Google, GitHub, AWS, Databricks, Bloomberg, Saxo Bank and others. It grew out of one sharp question, raised in SEP-2076 ("Agent Skills as a First-Class MCP Primitive"): do MCP's existing primitives (tools, prompts, resources) already suffice for skills, or do skills need a primitive of their own?

Two answers are on the table. SEP-2076 proposes skills as a brand-new first-class primitive, with its own skills/list and skills/get methods, a skills server capability, and a notifications/skills/list_changed signal, the same shape tools and prompts already have. The working group's current direction, though, is the leaner SEP-2640 ("Skills Extension," on the Extensions Track), which builds skills on the existing Resources primitive rather than adding a new one. Reusing Resources inherits URI addressability, resources/read, subscriptions and templates for free; a server declares the capability as io.modelcontextprotocol/skills and can then serve its tools and the instructions for using them as a single atomic unit.

A skill, in the draft, is a directory with a SKILL.md at its root carrying YAML frontmatter (at minimum a name and a description) plus any supporting files, and no nested SKILL.md beneath it. Each skill gets a skill:// URI (for example skill://acme/billing/refunds/SKILL.md), whose final path segment must equal the skill's name. An optional skill://index.json lists what's available, but a skill's URI is directly readable via resources/read whether or not it's indexed.

Here's the part that matters for our purposes: how it's meant to be consumed. The host loads only the frontmatter, the name and description, into the model's context at startup; the model then calls resources/read to pull the full instructions only when a task matches. That is progressive disclosure, and it is exactly the property that let the skills arm keep standing context small while still applying the right procedure on demand, the efficiency result the tool-call counts bear out. Prompts work differently: a prompt is still something a user picks. Skills get their own discovery path precisely so the agent can route to them with no human in the loop, which is the autonomy property this whole piece is about.

None of this is final. SEP-2640 and its reference implementation were both still in review as of writing, with related work underway on registry-level discovery (a skills.json proposal) and coordination with the external Agent Skills content spec. But the direction is unambiguous, and it matches what our benchmark points at: give the agent a discoverable, self-applied procedure, and it conforms and succeeds on its own.

What we'd instrument next

Two honest gaps sit at the center of the skills argument, and we're naming them because they're where the next round of evidence has to come from. This experiment shows that the SOP helps and that the delivery channel among agent-driven options is a wash. It does not empirically show skills beat MCP prompts (we ran no user-invoked prompt arm), nor does it measure auto-discovery accuracy. The case for skills specifically rests on the auto-discovery design property, argued rather than measured. We'd rather say that plainly than let it be overread.

The highest-value addition is routing / discovery accuracy: when a task arrives, did the agent auto-discover and load the correct SOP from the library, with no user? That measures auto-discovery directly, isolating a real failure mode of scaling the library: mis-routing. We don't log selected-versus-correct skill today; we should. Second is reliability (pass^k, R≥5). We ran one repeat per cell, so consistency across stochastic runs is untested, and reliability is, I suspect, both the natural place for externalized state to finally pay off and the thing that matters most for unsupervised operation, where a single rare wrong run is expensive.

Package the SOP as a skill

Pulling it together. Autonomy means the agent does the job your way and gets your outcome, unsupervised, so the two metrics worth tracking are conformance and success rather than uptime or turn count. The SOP is what delivers both: without it the agent under-conforms and under-succeeds; with the SOP as a skill, conformance improves significantly on τ² (the skill arm follows ~90% of prescribed actions versus ~50–59% without) and success rises with it, with the honest caveat that the SOPBench conformance gain is small and the multi-turn success magnitudes are directional. In our runs, the limit was whether the right procedure reached the agent's context at all. Raw model capability rarely came up.

The value of a skill is auto-discovery. The agent has to select and apply the right SOP with no user present; skills do that, user-invoked MCP prompts have no way to, and that's why the MCP working group is adding skills to the protocol. And keep it lean: discovery plus on-demand delivery wins, while externalized run-tracking is cost without benefit at these horizons.

So the "do this Monday" version: take the procedures that define how your operators actually work and package them as auto-discoverable skills rather than prompts a human has to pick or bare tools buried in a server. Measure your agents on whether they follow those procedures and reach the right outcome, both, unsupervised. Resist the urge to build a run-state machine until you can show the horizon demands one. As for which work an agent can own in the first place and which stays human, that's a question of position in the arc of work, the companion argument in The Tideline.