esmirnov.aiUse this to test a specific claim about an LLM: whether it follows supplied evidence, applies an unfamiliar rule, or judges responses consistently when their order changes. You get an experiment you can run, or an analysis of responses you already collected.
Why use it?
Getting the baseline answer right leaves several possible explanations. The model might follow the evidence, pick the first option, or keep returning a familiar answer. This skill builds matched cases where those behaviors predict different results.
Change a decisive fact and check that the answer changes correctly. Independently change an irrelevant cue, such as row order, and check that the answer stays the same. Add missing-evidence and repeated-prompt controls when useful. Report correctness alongside stability, since a constant wrong answer can be perfectly stable.
ChatGPT, Claude and Codex can design experiments from a detailed prompt. The skill saves the protocol, scoring rules and interpretation limits for repeated use. Its results describe behavior on the tested tasks; they do not establish an internal reasoning algorithm or prove memorization.
Installation
Clone the repository and keep the whole llm-counterfactual-lab folder, including references/:
git clone --depth 1 https://github.com/eduard-s/skills.git ai-skills
For a fresh local Codex installation:
mkdir -p ~/.agents/skills
cp -R ai-skills/skills/llm-counterfactual-lab ~/.agents/skills/
Claude Code: Put the complete folder in ~/.claude/skills/. For Claude's other interfaces, follow the official installation guide.
ChatGPT desktop: Use @skill-creator with the folder's Markdown files and ask it to preserve the references. See the official ChatGPT and Codex skill guide for installation and invocation options. This is a standalone skill, without a plugin package.
Native skill imports and live target-model execution have not been tested for this skill. Its structure, example calculations and synthetic scoring fixtures have been checked.
Use it
Invoke $llm-counterfactual-lab in Codex, /llm-counterfactual-lab in Claude Code, or select the installed skill with @ in ChatGPT. Then supply a claim, an experiment budget, or existing prompts and raw responses.
Designing a protocol and analyzing supplied responses need no model API. Running one requires an available target interface. Without one, the skill gives you an explicitly unrun protocol and a separate answer key.
Try it
Use llm-counterfactual-lab.
Design a six-call experiment to test whether a document assistant follows
supplied lab measurements. Change one decisive measurement and row order
independently. Include missing-evidence and exact-repeat controls.
I do not have a target model interface here.
Sample answer
Illustrative response based on the repository's worked example. This is not a captured assistant response or a measured model result.
Protocol, not run. Ask which specimen has the greater corrected signal, defined as raw signal minus its own blank. Use four cases: two versions of a decisive blank measurement, each in two row orders. The correct specimen changes when that blank changes and stays the same when only row order changes.
Add one case with the decisive blank missing and one exact repeat. Score correct answers, correct evidence transitions, order stability and the two controls separately. Stop after six attempts and retain errors as well as responses. There are no target observations yet.
Open the skill source · Read the full worked protocol · Research basis · Development checks