Toward User Comprehension Supports for LLM Agent Skill Specifications #

§1 TL;DR #

878 cybersecurity SKILL.md files coded for four user-comprehension anchors: operational basis (92.1%), output contract (63.0%), boundary disclosure (51.4%), example demonstration (19.0%); only 2.3% provide all four. A 6-skill DNS/C2 subset shows examples make first local checks constructible from spec alone.

§2 痛点 / 方法 / 结果 #

Q1 痛点 #

Users choose and trust LLM agent skills based on SKILL.md specifications, but existing audits focus solely on adversarial behavior — prompt injection, unsafe tool use, secret exfiltration. Whether specifications actually help users form bounded expectations about what a skill consumes, produces, and covers remains unexamined. A skill can pass every safety audit yet still leave users unable to verify what it does or construct a first check. The stakes compound in cybersecurity, where users install skills for tasks they cannot independently verify.

Q2 方法 #

The authors adapt the description-to-permission fidelity paradigm from mobile app ecosystems (WHYPER, AutoCog, TKPERM) to agent skill specifications. They operationalize user comprehension into four textual anchors:

1. Operational basis — does the spec name inputs, prerequisites, evidence, or tools the skill relies on?
2. Output contract — can the reader anticipate the result shape and judge whether output is on-task?
3. Boundary disclosure — does the spec state where the skill does or does not apply?
4. Example capability demonstration — does the spec give a concrete example, sample output, or test case that seeds a first local check?

These anchors are measured via deterministic rule-based coding (case-insensitive regex on headings and prose) across 878 cybersecurity SKILL.md files from five GitHub sources (2 institutional, 3 community). A focused qualitative subset (n=6 DNS/C2 telemetry skills) tests whether the example anchor reduces reliance on helper-code inspection for first-check construction.

核心技术壁垒: The four-anchor comprehension framework — decomposing "does the spec support user understanding" into four independently measurable textual signals adapted from the description-to-permission fidelity paradigm. No prior work measured agent skill specifications as user-facing capability disclosures rather than instruction containers.

Q3 结果 #

Anchors are steeply uneven: operational basis appears in 92.1% of skills, output contract drops to 63.0%, boundary disclosure to 51.4%, and example demonstration to 19.0%. Full coverage (all four anchors) occurs in only 2.3% (20/878). Skill authors treat "what the skill needs" as obligatory but "what success looks like" as optional.

In the DNS/C2 subset, the single example-bearing skill (dns-exfil-zeek) required no helper-code inspection for first-check construction. The five no-example skills left varying implementation details — command arguments, output field names, parser expectations — unrecoverable from the spec alone.

§3 架构 / 方法图 #

The paper contains zero figures; the study design is a two-component measurement pipeline. The following diagram reconstructs the method flow:

flowchart TD subgraph Corpus["Component 1: Corpus Coding (n=878)"] A[878 cybersecurity SKILL.md files\n5 GitHub sources] --> B[Rule-based coding\ncase-insensitive regex\nheading + prose matching] B --> C1["Anchor 1: Operational basis"] B --> C2["Anchor 2: Output contract"] B --> C3["Anchor 3: Boundary disclosure"] B --> C4["Anchor 4: Example demonstration"] C1 & C2 & C3 & C4 --> D[Prevalence & co-occurrence\nquantification] end subgraph Subset["Component 2: DNS/C2 Subset (n=6)"] E[6 DNS/C2 telemetry skills\nfiltered by domain + local executability] --> F[First-check construction test\nspec-only readthrough → helper-code inspection] F --> G[Compare: example-bearing vs\nno-example skills] end D --> H[Finding: steep anchor dropout\n92.1% → 19.0%\n2.3% all-four] G --> I[Finding: examples reduce\nhelper-code dependency] H & I --> J[Recommendation: audit specs\nfor comprehension support]

The two components are complementary: the corpus coding establishes prevalence at scale (quantitative), while the DNS/C2 subset illustrates the practical consequence of missing examples for one anchor (qualitative). No interaction or feedback loop exists between the two components.

§4 作者证明 #

无形式化作者证明 — 仅实证

The paper contains no formal mathematical model, no convergence guarantees, and no proofs. All claims rest on corpus-scale prevalence measurement and a small qualitative subset. This is standard for empirical measurement studies in the usable-security tradition.

六项检查 #

1. Arithmetic consistency: 809/878 = 92.14% ≈ 92.1%; 553/878 = 62.98% ≈ 63.0%; 451/878 = 51.37% ≈ 51.4%; 167/878 = 19.02% ≈ 19.0%; 20/878 = 2.28% ≈ 2.3%. All reported percentages match stated numerators to rounding precision.
1. Co-occurrence bound: 2.3% all-four ≤ 19.0% minimum single anchor — mathematically necessary and consistent. The large gap between individual prevalence and joint prevalence indicates anchors are not bundled by authors as a coordinated package.
1. Corpus composition: Five GitHub sources spanning institutional (Transilience, Trail of Bits) and community (mukul975, alirezarezvani, Eyadkelleh) origins. Cybersecurity-only; generalizability is explicitly bounded to this domain.
1. Coding validity: Deterministic rule-based coding (regex) is reproducible but not validated against manual annotation. The paper acknowledges reported percentages are prevalence estimates — false positives and negatives from regex under/over-matching are unquantified.
1. Subset representativeness: The DNS/C2 subset (n=6) is explicitly illustrative, not designed for statistical inference. Inclusion criteria are documented: domain match, local executability, helper-code presence.
1. Base rate interpretation: The 73-point gap between operational basis (92.1%) and example demonstration (19.0%) is robust to reasonable measurement error — even 10-point swings in either direction would not close it.

Agent-specific assessment #

No success-rate model, no latency budget, no formal failure-mode classification. The implicit failure mode is comprehension failure from missing anchors — users cannot form bounded expectations about skill behavior. A metric that could have been bounded: task completion accuracy (or first-check construction time) conditioned on anchor availability. This would require a controlled user study the paper does not conduct but explicitly recommends (§5 Q1, §6 Limitation 1).

§5 实验与数据 #

Coding instrument #



Paper Table 1 — Coding rule families for corpus analysis.

The four variables are measured via keyword and heading pattern families. Operational basis cues are the broadest (input-oriented headings, action verbs like "requires"/"expects"/"accepts"). Example capability demonstration cues are the narrowest (headings like "Example"/"Quickstart"/"Test Case", phrases like "expected output"/"fixture"/"try this"). The asymmetry in cue breadth partially explains the prevalence gradient — operational basis has more lexical surface area to match than example demonstration.

Prevalence results #



Paper Table 2 — Corpus-coded comprehension anchors (n=878).

The headline distribution: operational basis 92.1% → output contract 63.0% → boundary disclosure 51.4% → example demonstration 19.0%. Each transition drops roughly 15–30 percentage points. The all-four co-occurrence rate (2.3%) is an order of magnitude below any single anchor, indicating that the anchors are effectively independent in authoring practice. Skill authors treat stating prerequisites as near-obligatory but providing concrete examples as rare.

DNS/C2 first-check subset #



Paper Table 3 — Information that still required code inspection before constructing a first local check in the DNS/C2 subset (n=6).

dns-exfil-zeek, the only skill with a package-level example (concrete JSON output with summary fields, flagged domains, evidence indicators), required no substantial recovery from helper code. The five no-example skills each left different implementation details unclear: exact JSON fields (ransomware-net), runnable commands and output fields (dns-query-analysis, dns-tunnel-zeek), minimal connection log format (beaconing-freq.), and header-first TSV parser expectations (data-exfil-ind.). The variety of "what remained unclear" entries shows that missing examples create diverse comprehension gaps, not a single uniform failure.

Corpus sources breakdown #

The 878 skills come from five GitHub sources: Transilience (institutional), Trail of Bits (institutional), mukul975/Jangra (community), alirezarezvani/Rezvani (community), Eyadkelleh/Kelleh (community). The paper does not report per-source prevalence breakdowns, leaving open whether institutional vs community authoring practices differ in comprehension support coverage.

§6 论证链 #

#	Claim	Evidence	§Ref
1	SKILL.md serves dual function: agent instruction and user-facing capability disclosure	Analogy to Android permission prompts (Felt et al., 2012) and privacy nutrition labels (Kelley et al., 2009); dual-use framing established	§1, §2.2
2	Safety audits do not address comprehension support	Agent Audit, skill-audit, and DynAuditClaw check adversarial behavior; passing safety audit does not ensure users can form bounded expectations	§2.1
3	Description-to-permission fidelity can be adapted to skill specifications	WHYPER, AutoCog, TKPERM measured app descriptions vs permissions; the same logic applies to spec text vs operational capability	§2.3
4	Four comprehension anchors are measurable via textual cues	Rule-based coding instrument with deterministic regex patterns for operational basis, output contract, boundary disclosure, example demonstration	§3.1
5	Anchors are steeply uneven across 878 cybersecurity skills	92.1% operational basis → 63.0% output contract → 51.4% boundary → 19.0% example; only 2.3% all-four	§4.1
6	Examples make first local checks constructible from spec alone	dns-exfil-zeek (with example) needed no code inspection; 5 no-example skills required helper-code inspection to recover implementation details	§4.2
7	Skill specifications should be evaluated as user-facing disclosures, not just instruction containers	Synthesis of steps 2–6; three open research questions on when examples help (Q1), mislead (Q2), and where comprehension support should live (Q3)	§5, §7

§7 实现 cross-reference #

Source code and data: https://github.com/zikaiwen/cyber-skill-comprehension (§3.1)

The repository contains the rule-based coding implementation and the 878-skill corpus. The paper does not provide file-level or line-level code citations; the regex pattern families in Table 1 constitute the primary specification of the coding instrument. Specific implementation details (pattern compilation, multiline handling, per-source file parsing) require repository inspection.

核心技术壁垒: The hardest-to-replicate contribution is not computational but conceptual: recognizing that the same description-to-permission fidelity paradigm used for Android apps applies to agent skill specifications, and that "supports comprehension" decomposes into four independently measurable anchors. Replication of the corpus coding itself is straightforward (deterministic regex), but arriving at the four-anchor decomposition requires the cross-domain connection between usable-security disclosure research and agent-skill ecosystems — a framing absent from prior skill-audit work (Agent Audit, skill-audit, DynAuditClaw) which targeted adversarial behavior exclusively.

关键実現细節:

1. Coding uses case-insensitive regex with multiline heading matching — this design choice means anchors expressed only in prose body text (without matching headings or keywords) will be missed, introducing a systematic false-negative risk for unconventionally written specs.
2. The four anchors are measured independently; the 2.3% all-four co-occurrence is computed post-hoc from per-anchor binary flags. This means the co-occurrence figure compounds per-anchor measurement noise — if each anchor has even 5% false-negative rate, the true all-four rate could be meaningfully higher than 2.3%.