Conversation with Claude Sonnet about Skills, Scaffolding and PAI curious about accuracy

[nick-walt]

I originally asked Claude Sonnet to write a step by step procedure on how to enable Flakes on a new minimal ISO install of NixOS and then install the Determinate Nix Installer into the config, using official documentation and other well written sources — then applying best practices in technical writing to write for a user unfamiliar with NixOS, Nix and Determinate Nix. The result was excellent.

Then I asked Claude to use it's references and research and the output to create a Skill / Scaffolding written for Claude and it's three LLMs to use in its Harness.

Then I asked it to tell me how this Skill would be added to and used by PAI — based on the v5 codebase on GitHub.

I then asked Claude how the Skill could be improved whether through obvious methods such as manual user editing, editing by Claude or some other method defined in PAI architecture.

These are the prompts and responses open for discussion:

Prompt 1

Based on the key references used for the previous procedure can you please create a Claude harness skill to provide scaffolding for future requests for accurate, concise and complete procedural documentation based on best practices used in technical writing. Please write this skill / scaffolding (please advise if this terminology is accurate) in a format best used by Claude and it's three LLMs. Please output this in a file that can be used directly by Claude and it's LLMs. Please make a note if this promot can be improved.

Claude Sonnet response
Now I have everything I need. Let me address your terminology questions first, then produce the skill.

On your terminology:

• "Skill" is the correct term in Claude Code's system. A skill is a SKILL.md file with a YAML frontmatter header that tells Claude when and how to apply a specific behaviour. You install it into ~/.claude/skills/ and Claude Code loads it automatically when relevant.
• "Scaffolding" is also accurate in a broader sense — a skill is scaffolding in that it provides structure and constraints around Claude's behaviour for a specific task type. Both terms work; "skill" is the precise technical term within Claude Code.
• "Three LLMs" — a small correction here: Claude is a single model family, not three separate LLMs. The three tiers you may be thinking of are Opus, Sonnet, and Haiku — they are different-sized variants of the same architecture, and all three read SKILL.md files identically. The skill format does not change between them.Notes on what was produced and how to use it:

File format. This is a SKILL.md file — the standard format Claude Code uses for skills. The YAML frontmatter at the top (between the --- markers) contains the name and description fields. The description is deliberately written to be "pushy" as the skill creator documentation recommends, so Claude is more likely to recognise when to apply it.

How to install it. Place the file at:

~/.claude/skills/technical-procedure-writer/SKILL.md

Claude Code will detect and load it automatically at the start of each session.

How it works across Opus, Sonnet, and Haiku. All three model tiers read the same SKILL.md file. Opus will apply the most nuanced interpretation of the research and writing rules; Haiku will follow the structural rules faithfully but with less inferential depth on edge cases. The skill is written to be explicit enough to work well across all three without modification.

On your prompt improvement question — yes, the prompt could be improved in three specific ways, which I included as a note inside the skill itself so they are preserved for future reference:

1. Audience specification — naming the intended reader more precisely (e.g. "junior sysadmin", "developer with no prior NixOS experience") would let Claude calibrate terminology and assumed knowledge more accurately.
2. Scope boundaries — clarifying whether the skill should cover only CLI/sysadmin procedures or also GUI walkthroughs, API reference docs, and similar formats.
3. Output format preference — whether Markdown, .docx, or another format is the default delivery target. The skill currently assumes Markdown.

Prompt 2

Excellent. If this skill were to be added to and used within an installed PAI version 5 software, according to the requirements of the current codebase, how would this need to be done? What procedure would need to be followed by a junior developer unfamiliar with PAI? Please also make a note on how this prompt could be improved.

Claude Sonnet response
Now I have everything I need. I'll produce two files: a PAI-compliant version of the skill (with required adaptations), and the installation procedure.Two files are provided: the installation procedure and the PAI v5-compliant version of the skill.

---

What changed between the original skill and the PAI v5 version, and why:

Three adaptations were required by the PAI v5 codebase:

Naming convention. PAI v5 mandates TitleCase (PascalCase) for all skill directory names and name fields. The original skill used technical-procedure-writer (kebab-case, which is the Claude Code convention). For PAI, this becomes TechnicalProcedureWriter. The PAI source document SkillSystem.md describes this as mandatory, and skills that deviate from it are not correctly routed by the intent system.

Description format. PAI's intent routing system specifically looks for the pattern [What it does]. USE WHEN [triggers]. in skill descriptions. The original description used natural prose. The PAI version restructures it to follow this pattern so that the DA's routing layer reliably identifies when to activate the skill.

Description length. PAI requires the description to be a single line in the YAML frontmatter. The original used a multi-line block scalar. The PAI version collapses this to one line.

The skill body itself is unchanged — PAI's three-tier loading system (frontmatter always in context, body loaded on trigger, workflows loaded on routing) is structurally compatible with the skill as written.

---

On the prompt improvement note:

The prompt asks to follow "the requirements of the current codebase." This is well-framed but carries a risk worth naming: PAI v5 does not publish a dedicated specification document for user skill installation separate from the Pack system. The procedure is derived from SkillSystem.md embedded in the kai-core-install.md Pack, the PAIPackTemplate, and community bug reports. It is accurate for v5.0.0 but should be re-verified against the repository before use with any future minor release, as PAI's skill loading conventions have changed between major versions. A more precise prompt would name the operating system, cite the specific codebase files to consult, and specify whether a full Pack wrapper (with README.md, INSTALL.md, and VERIFY.md) is expected or whether direct skill placement suffices.

Prompt 3

Thank you. Once the skill is installed into PAI version 5 architecture, starting with but not limited to a review of the relevant code files used in the previous answer, what would be the best way to improve the skill over time? The two obvious methods being 1. By Claude and 2. Manually edited by the human user. Please note if PAI archerecture will also provide either a third option through automatic improvement or a combination of all.

Claude Sonnet response
Now I have a complete picture from the codebase. This is a substantive answer that warrants a full document.The short answer to your question is: yes, PAI provides a genuine third option, and a fourth. Here is the summary of what the codebase reveals:

Method 3 — Passive automatic capture runs without you asking. The Unified Learning Router in the LEARN phase handles every kind of learning including "skill gotchas" — routing observations about unexpected skill behaviour to PAI's memory surfaces. The rating hooks add to this: any low-rated response generates a structured failure analysis file that accumulates over sessions. This is signal collection, not skill editing — PAI does not autonomously rewrite your SKILL.md.

Method 4 — Active autonomous improvement loops must be explicitly invoked, but then run with minimal human intervention. PAI v5 ships an autonomous hill-climb loop against any target using eval mode for prompts and skills with an LLM-as-judge, an IterativeDepth skill for iterative improvement across multiple full Algorithm cycles with human review between iterations, and an Upgrade skill that generates prioritised improvement recommendations via four parallel threads covering audit, user context, external research, and skill drift.

The critical design boundary — confirmed by the codebase — is that PAI accumulates and proposes but does not autonomously commit changes to user skill files. The autoMemoryEnabled: false setting and the Forbidden Destination rule in the LEARN router both enforce this. Human approval remains a required step before any change is written. PAI is designed as a signal-to-proposal engine, not a signal-to-commit engine, which is the right constraint for something as consequential as modifying the rules governing how your DA behaves.

@danielmiessler Would you recommend this kind of approach to learning and maturing a PAI installation?

My next prompt is to ask more about PAI's third and fourth methods, and also if there are other instruments in PAI and Claude that complement and enhance the Skill in the overall architecture.

The slow bit is definitely the human processing, understanding and incorporating and acting on the responses. The human also requires a map and way to reference the rate of work amplification — untill a sophisticated, secure and private MCP Server infrastructure can be doing more of the work directly.