Do SKILL.md Linters Matter? An Empirical Study

This report evaluates whether recently proposed SKILL.md best practices translate into measurable improvements in agent performance. After implementing a linter based on these recommendations and applying it to Garry Tan's GBrain repository, I observed consistent improvements in skill-routing accuracy across all 18 benchmark runs, ranging from +1.3 to +7.6 percentage points.

A recent paper, From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills (Hong, Imani & Ahmed, UC Irvine), proposes a set of best practices for writing SKILL.md files — the documentation files that tell an agent what a skill does and when to invoke it. Does following them actually change agent behavior, or does it just make the files look tidier? To find out, I implemented a linter that enforces the paper's recommendations and tested it on real-world data rather than an idealized lab setup.

The test subject is Garry Tan's GBrain repository, which contains 53 skills as of July 2026.

Method

Fully evaluating agent performance is difficult and often subjective, so I narrowed the focus to routing — whether the model picks the right skill for a prompt — rather than full execution.

To start, I ran the linter nikhail against the GBrain repository, checking all 53 skills, and got this report:

$ uvx nikhail path/to/gbrain/skills --html gb_report.html
rules violated:
  S005  body-size-budget: SKILL.md body over 500 lines defeats
        progressive disclosure
  S006  referenced-files-exist: Files referenced from SKILL.md must
        exist in the package
  S007  no-orphan-files: Bundled files never mentioned in SKILL.md are
        dead weight the agent will never use
  S008  parseable-frontmatter: SKILL.md must begin with valid YAML
        frontmatter
  S009  trigger-guidance: Description should state when to use the
        skill, not only what it does
  S011  no-markup-in-description: Description must not contain XML/HTML
        tags (prompt-injection surface)
  S014  third-person-description: Description should be written in
        third person, not first person
  S016  no-rigid-command-sequence: Long fixed command sequences
        over-prescribe execution; state the goal or ship a script
  S018  has-example: A substantial skill should include at least one
        concrete example

path/to/gbrain/skills/academic-verify/SKILL.md
     -  warning  S007  bundled file 'routing-eval.jsonl' is never
        referenced in SKILL.md — the agent has no instructions for when
        or how to use it
   205  info     S006  referenced file 'skills/perplexity-research/SKILL.md'
        resolves only from 'gbrain/' (2 level(s) above the skill package)
        — the reference breaks if the skill is distributed standalone
   ...

The linter runs a series of deterministic checks and reports every rule violation it detects. For example, S005 flags oversized skill bodies, while S009 flags descriptions that lack trigger guidance. This guidance is critical because the router depends on the description to determine when a skill should be invoked.

What was fixed

Because at routing time most models look only at the title and the description to decide which skill to invoke, I applied fixes to the description strings alone, leaving the rest of each skill package unchanged.

For each flagged skill, a fresh agent session (Claude Fable 5) was spawned with that skill's full original text plus its linter findings, and asked to apply the fixes to the description string only.

The routing test bench

Each test case presents a user prompt alongside the catalog of all 52 skill descriptions and checks whether the model routes to the expected skill — or correctly declines to pick one. Grading is exact-match and binary. The frozen intent set has 323 cases from four sources:

• 110 routing-eval cases — the human-written routing-eval.jsonl fixtures that ship inside the GBrain repo itself.
• 104 generated easy cases — paraphrases that stay close to each skill's own vocabulary.
• 104 generated hard cases — paraphrases that deliberately avoid the skill's vocabulary, the way a user who has never read the skill would phrase the ask.
• 5 null cases — prompts that should route to no skill at all, so the bench also penalizes over-eager selection.

The generated cases were authored by Claude Fable 5. Each configuration ran k = 5 trials per case, comparing the original descriptions (Arm B) against the linter-fixed ones (Arm C) across multiple models and reasoning-effort levels.

Results

Routing accuracy improved in every configuration tested, by 1.3–4.9 percentage points depending on the model.

All raw test data is in bench/results, and a sample per-run summary shows the level of detail each run records. Across all 12 benchmark runs, Arm C beat Arm B every single time. Three patterns stand out:

1. Weaker routers gain more. The cheapest model tested (GPT-5.4 nano) gained 4.9 points. A strong model can often infer intent through a weak description on its own.
2. The gain lives where users don't speak the skill's vocabulary. Cases that already contain the skill's trigger words barely move. A description fix helps exactly when the prompt and the original description share no vocabulary.
3. The improvements are consistent across different LLMs. 79 of the 323 cases net-improved, 57 declined, and 187 tied (sign test p ≈ 0.085).

An honest caveat: the tiny null-intent class (n = 5) shows that richer descriptions can also attract off-topic asks.

One design note: Claude Fable 5 and Claude Opus 5 do not appear as routers because Fable 5 wrote the Arm C descriptions and the generated test intents, while Opus 5 authored the linter's rules and produced the findings. I didn't want a model grading text it wrote itself.

Another interesting question is whether giving the router more thinking time reduces the benefit of better skill descriptions. The effort sweep (Chart 2) says otherwise: at every effort level on both Claude Sonnet 5 and GPT-5.6 Terra, the fixed descriptions stay ahead. Extra reasoning does not fully compensate for information missing from a thin description.

Additional testing: three model generations

Full-bench runs are expensive, and most of the cost goes to intents both arms always get right. So I distilled a half-cost subset: 26 of the 52 skills and 150 of the 323 intents, keeping every intent that any prior run ever missed (plus a few always-correct anchors per skill, and an upweighted null class), with five frozen paraphrase variants per intent so k = 5. On this subset I swept three generations of OpenAI reasoning models at low and high effort.

Because the subset deliberately over-samples hard cases, absolute accuracies run lower than on the full bench, but the fix effect is sharper: +6.1 to +7.6 percentage points in all six runs. The effect also holds across two years of model progress: o1 with fixed descriptions (80–82%) routes better than GPT-5.6 Terra with the originals (75–77%) — fixing the descriptions was worth more than two model generations of router.

Conclusion

Following the SKILL.md best practices from From Anatomy to Smells: An Empirical Study of SKILL.md in Agent Skills is not just cosmetic. Applying linter-guided, description-level fixes to a real 53-skill repository improved skill-routing accuracy in all 18 benchmark runs. The largest gains came from prompts that did not mirror the skill's own terminology and from smaller routing models that were less able to infer missing context from sparse descriptions.

Routing is only the first question; whether the fixes also change execution quality once a skill is invoked is the next experiment.