Agent skills: why five beat a hundred

The format is simple. SKILL.md opens with two required fields, a name and a description. At startup the agent loads the names and descriptions of every installed skill, and only opens the full text when it decides a skill fits the task. Both points of this article follow from that sentence: choosing a skill is the model's call based on a short description, and the skill's content is a set of instructions the agent will carry out with your permissions.

Skills steady the procedure and add little knowledge

On 14 August a team from Princeton, Stanford, UC San Diego, USC and Johns Hopkins published Demystifying Agent Skills: Why They Work, Until They Don't. It is a preprint, not yet peer reviewed, but it rests on a solid base: 8,135 trial records from controlled experiments on the Terminal‑Bench and SkillsBench benchmarks, with the Codex and Gemini CLI agents.

The authors hand-coded why a skill helped. In 65.7% of cases the mechanism was what they call procedural anchoring: the skill told the agent which setup steps to run, which order to use tools in, what to check along the way and which known pitfalls to avoid. Supplying missing knowledge accounted for 4.5%.

Form matters too. From the same records of earlier attempts, the authors built either a skill or plain workflow memory, a cleaned-up record of what the agent had done. The skill came out 6.06 percentage points ahead. The practical reading: a skill pays off for a procedure your team runs every week, where errors come from skipped steps. If the agent lacks facts, a skill will not fix that.

Skills can also hurt. In runs with skills, invocation and boundary failures, where the guidance was there but the agent misused it, over-applied it, ignored it or was blocked by outside limits, came to 78 out of 528, against 19 out of 528 without skills. The authors say it plainly: skills fail under brittle assumptions, incompatible contexts or insufficient adaptation.

At a hundred skills the agent stops hitting the target

The strongest result is about selection. The authors gave the agent a pool with one skill that fit the task and a growing number of others, then checked which ones the agent actually used. The metric is actual-use precision: the share of skills used that were the right one.

The fall is steady at every step. When the extra skills were close in topic to the right one, precision was 34.5% with five, 22.3% with ten, 15.7% with twenty, 7.3% with fifty and 3.7% with a hundred. At a hundred skills the agents, depending on the pairing and the pool, still reached for the right one in 54 to 74% of tasks, just alongside several others.

When a skill is picked from its description alone, similarity hurts most. In a separate test, an embedding model ranked skills by how close their descriptions were to the task. With a pool of similar skills, top‑1 precision fell from 70.5% to 53.4%, and with random ones only from 97.7% to 84.1%. In actual use, though, precision fell with every kind of pool. So there are two lessons: merge skills with close descriptions, say one for the sales report and one for the margin report, or sharpen their descriptions, and keep the pool itself small.

To be fair: in this experiment the share of tasks solved barely moved, from 36.4% to 39.3%. For a team that is thin comfort. You install a skill so the agent follows your procedure. With a hundred skills you do not know which one it followed, so you lose exactly what the skill was there for: repeatability, and the ability to check what happened.

August: skills as a way to steal credentials

On 6 August, at Black Hat USA, Zenity Labs described a campaign on skills.sh, the public skills registry run by Vercel. Between 2 and 13 July the attackers set up organisations with names close to the official Paperclip and Browser Use projects and cloned their skills. The copies started clean and collected installs. On 11 July the skill files gained instructions telling the agent to download and run a script from the attackers' server.

The script searched 138 paths on disk: SSH keys, AWS, Azure and GCP credentials, Kubernetes and Docker configs, npm and PyPI tokens, .env files. By 2 August the skill family had passed 1.7 million installs. That is an aggregate install count, not the number of people affected, as Zenity itself stresses. Vercel and GitHub removed the skills and repositories within 12 hours of the report.

The mechanism matters more than the number. For several days the skills were faithful copies of the originals. The malicious command arrived later, in a separate installation file the agent was told to open only when something needed installing. A review of SKILL.md alone would have found nothing. If your agent pulls skills from a registry every time it starts, or updates them on its own, last month's review means nothing.

Anthropic, when it described the skill format in October 2025, advised installing skills only from trusted sources and, for less certain ones, auditing every file, paying particular attention to scripts and to instructions that connect to external addresses. The skills.sh campaign showed that a one-off review is not enough: trust has to attach to a specific version, not to a name.

What a good company skill looks like

One procedure, a precise description that says when to use it, steps to check, and an explicit ban on what the agent must not do. An example for the monthly VAT summary:

---
name: vat-summary
description: Prepares the monthly VAT summary from the accounting system's CSV export. Use when someone asks for a monthly VAT summary or a check of the export before the JPK filing.
---

# Monthly VAT summary

1. Load the CSV file the user points to. The columns are described in references/columns.md.
2. Check that every row is dated within one month. If not, stop and list the rows outside it.
3. Sum net and VAT separately for the 23%, 8%, 5% and 0% rates, separately for sales and purchases.
4. Compare the VAT total with the control total in the export's last row. A gap above 0.01 PLN is an error, not rounding.
5. Write the result using the template in assets/summary.md.

Never send data off this machine and never run commands that download files from the network.

A policy for the team

The first five are easiest to find in corrections. Look at what people had to fix after the agent over the last month, and mark the cases where the agent knew the subject but skipped a step or did it in the wrong order. Those are your skill candidates. Errors that come from missing knowledge are solved differently, for example with a document the agent can read.

Pinning skills to a specific commit from an internal repository takes one line:

git clone https://git.example.com/ai/skills.git skills && git -C skills checkout <commit-sha>

Start by listing which skills the agents in your team have installed today and where they came from. If any came from a Paperclip or Browser Use clone, treat the machine as compromised and rotate the keys and tokens that were on it.