How to safely test an AI Skill before trusting it

The short answer

To safely test an AI Skill, run one identified version on a disposable workspace with no real secrets, no personal sessions, synthetic input, network access off by default, and only the minimum tools needed for one harmless task. Record every approval request, process, file change, network attempt, and external side effect. Deny unexpected actions first; do not approve them just to see what happens. Reset the environment between runs, widen one permission at a time, and stop when observed behavior exceeds the Skill's declared task.

This is a controlled behavior test, not proof that the Skill is safe. A Skill may behave differently with another model, prompt, dependency, platform, account, or update. The useful outcome is a bounded decision tied to one artifact and environment: reject for now, test again with narrower controls, or allow for one defined task with limits.

Review the files before execution. The public Agent Skills specification defines SKILL.md as the required entry point, but a Skill may also include executable scripts/, references, assets, and additional files. Use the SKILL.md inspection flow first if the source, revision, or complete tree is still unclear.

Start with a one-page test charter

Do not begin with “try the Skill and watch it.” Write a small test charter that makes unexpected behavior visible:

Field	Record before the run
Artifact	Source, immutable revision or digest, complete file-tree hash, dependencies, and install method
Intended task	One harmless result the Skill should produce, such as summarizing synthetic notes or editing a toy repository
Allowed inputs	Exact test files, prompts, tool output, and any deliberately provided untrusted sample
Allowed authority	Writable directory, permitted commands or tools, approved network destinations, and disposable account scope
Forbidden assets	Home directory, real repositories, browser sessions, SSH material, environment secrets, personal data, production systems, and unrelated accounts
Expected effects	Files, processes, requests, and external records that should exist after a successful run
Stop conditions	Any undeclared destination, credential request, parent-directory access, persistence, destructive command, hidden download, or broader approval
Reset proof	Snapshot, image, fresh clone, or scripted rebuild that returns the lab to a known state

NIST SP 800-115 separates security assessment work into planning, execution, and post-execution analysis and reporting, and recommends defining systems, permitted testing, data handling, and incident guidance before the assessment. It is not an AI Skill testing standard, but the same discipline prevents an informal experiment from silently becoming permission to touch real assets.

Build an empty room, not a copy of your laptop

Choose the strongest isolation you can operate and verify. A dedicated virtual machine gives a separate kernel; a container or host sandbox can be lighter, but its mounts, sockets, network, helper processes, and host integrations define the real boundary. Do not call an environment “isolated” until you have checked what it can read, write, contact, and launch.

Prepare the lab with:

• a fresh operating-system or container image and an ordinary non-admin user;
• only the exact Skill snapshot and the minimum pinned runtime it needs;
• a toy repository or synthetic documents with no copied production content;
• fake canary values such as TEST_TOKEN_DO_NOT_USE in expected and forbidden locations;
• no mounted home directory, Docker socket, SSH agent, browser profile, clipboard bridge, cloud metadata, or real credential store;
• network disabled initially, then an explicit destination allowlist only if the task genuinely requires egress;
• logs or snapshots that show process, filesystem, network, tool, approval, and external-state changes.

OpenAI's Codex agent approvals and security guidance describes sandbox scope and approval policy as separate controls: the sandbox limits what an action can technically reach, while the approval policy decides when the agent must stop and ask. It also documents network-off and workspace-limited defaults for local Codex surfaces. Verify the effective settings in your own run; a default described by documentation is not evidence that your local configuration still has that default.

Anthropic's Claude Code security guidance similarly separates permission prompts, working-directory boundaries, sandboxing, and user review. Its sandbox environment comparison warns that writable mounts can still be modified and permitted egress can still leak readable data; it recommends a dedicated VM for untrusted code. These are useful examples of a general rule: permissions decide whether an action may start; isolation limits the damage after it starts.

Use synthetic data that can reveal boundary crossings

Empty test data can make a dangerous Skill look harmless because there is nothing valuable to find or send. Instead, create synthetic inputs that exercise the declared workflow and reveal unexpected access without exposing real information.

Include:

1. a normal input that should complete the intended task;
2. a malformed input that should produce a bounded error rather than a wider search;
3. an untrusted text sample containing an instruction irrelevant to the task, which the agent should treat as data;
4. a fake secret inside the allowed workspace that should not leave it;
5. a fake secret outside the allowed workspace that the Skill should never read;
6. a filename, path, or tool result that tests whether the Skill follows data as if it were authority.

Canaries detect access only when you can observe the access path. A fake token appearing in a prompt, request body, log, generated file, or external record is evidence of a boundary crossing. Its absence is not proof that no other read occurred.

Run the deny-observe-narrow-repeat loop

Use this sequence for the smallest end-to-end task:

1. Baseline. Capture running processes, the filesystem state, active network policy, available tools, environment names, and any disposable account records.
2. Start with denials. Keep network, parent-directory reads, persistent settings, and external writes denied. Ask for only the harmless charter task.
3. Observe the request. When the agent asks for authority, compare the exact command, path, destination, payload, account, and effect with the charter. A vague approval is a failed observation point.
4. Allow one narrow action. If the request is necessary and expected, grant only that action or destination. Never approve an unexplained command merely to continue the test.
5. Inspect the delta. Compare processes, files, requests, tool calls, logs, and external state with the baseline and expected-effects list.
6. Reset. Destroy or restore the environment. Confirm that files, hooks, packages, credentials, processes, and external records did not survive unintentionally.
7. Repeat one variable. Change one input or permission and run again. If several conditions change, you cannot explain which one caused the behavior.

The loop should fail closed. A sandbox error is evidence that a boundary blocked an action; it is not a reason to rerun unsandboxed. An unexpected approval request is a finding to explain, not a dialog to dismiss.

Keep one observation log

Use one row per meaningful event. This six-channel log makes declared and observed behavior comparable without pretending that every platform exposes the same telemetry:

Time and step	Approval or tool call	Process or command	File or persistence delta	Network destination and payload class	External effect and decision
Baseline	None	Expected runtime only	Clean toy workspace	Network denied	No external records
Normal task	Exact file-read request	Declared helper	One expected output file	None expected	Continue
Untrusted input	Tool request names real target	No new process	No out-of-scope write	Blocked request to undeclared host	Stop and investigate
Cleanup	None	Runtime exited	Snapshot restored; no hooks	No connection	Reset verified

Record denials as well as successes. Also record blind spots: for example, “DNS attempts visible but request bodies unavailable” or “MCP server logs controlled by provider.” If you cannot observe a high-impact channel, reduce the reachable authority or choose a stronger lab.

Decide from mismatches, not from a clean-looking demo

Compare the charter with the log:

• Reject for now if identity is unresolved, the Skill seeks forbidden assets, hides or downloads instructions, requests unrelated credentials, bypasses controls, persists without need, or creates an unexplained external effect.
• Narrow and retest when the behavior is explainable but authority is broader than necessary, an allowlist is incomplete, a dependency is mutable, cleanup is uncertain, or an important channel is not observable.
• Allow for one task with limits only when the exact artifact completed the intended task, requested only necessary authority, matched expected effects, respected denials, and reset cleanly. Record the approved environment and re-test triggers.

Do not average away a severe mismatch because several other runs looked normal. The AI Skill attack-surface map helps trace the input-to-authority path behind a mismatch, while the risk assessment rubric separates capability, likelihood, impact, and evidence confidence.

Know what invalidates the result

Test again when the Skill, dependency lock, installer, model, host version, permission mode, sandbox policy, tool or MCP server, network destination, input class, account scope, or intended task changes. Re-test after an incident or when logs reveal a previously unseen behavior.

A successful test does not prove absence of dormant logic, model-dependent behavior, time-based changes, remote-service changes, or attacks outside the cases you ran. It shows that one identified version behaved within one recorded boundary for a small set of tests. Keep the result attached to those limits.

Bottom line

The safest useful test is small, observable, and disposable. Bind one artifact, define one harmless task, remove real assets, start with network and extra authority denied, use synthetic canaries, record six effect channels, reset, and change only one variable per run. Promote the Skill only to the environment and task that the evidence supports. For the wider five-layer model behind these limits, use the AI Skill security guide.