Skills pinchbench
📦
pinchbench
v1.0.0 Content revision r1 Critical ⚙️ External commands🌐 Network access📁 Filesystem access⚡ Contains scripts🔑 Env variables
Benchmark OpenClaw Agents With PinchBench
Agent teams need repeatable evidence before comparing models. PinchBench runs real-world OpenClaw tasks, grades outcomes, and can publish results to a leaderboard.
Supports: Claude Codex Code(CC)
Do not auto-install this skill.
The canonical policy requires operator review before any installation action.
Install with my Agent
Copy this request to your Agent. It includes the canonical Skill page and manifest.
Agent request
Review the Skillstore skill "pinchbench" from https://skillstore.io/skills/pinchbench-pinchbench.md and its manifest at https://skillstore.io/api/skills/pinchbench-pinchbench/manifest. Verify the artifact. Do not auto-install. Inspect the skill and report your findings, then wait for an operator or manual installation decision.Your Agent should still show its plan and request any confirmation required by the security policy.
Agent-readable resources
Use these links when an AI agent, crawler, or script needs clean context instead of reading the full page.
Test it
Using "pinchbench". Run the automated-only suite for openrouter/anthropic/claude-sonnet-4 without uploading results.
Expected outcome:
PinchBench completed the automated suite. The report showed the overall score, failed tasks, timing, and upload was skipped.
Using "pinchbench". Compare two model runs using the same task suite.
Expected outcome:
- Model A had stronger file and coding task scores.
- Model B used fewer tokens and finished faster.
- The summary highlighted score, cost, and timeout differences.
Using "pinchbench". Upload a saved benchmark result after registration.
Expected outcome:
The saved run was submitted to PinchBench and the response included submission status, rank, and leaderboard link.
Security Audit
Criticalv5 • 7/9/2026 Open versioned report
PinchBench is a legitimate benchmark runner, but it is not low risk. Confirmed issues include pipe-to-shell dependency installation, unsandboxed task grading code, external agent execution, token handling, and host metadata upload. Many documentation and task-rubric alerts are false positives, while binary assets remain manual-review blind spots.
38
Files scanned
7,808
Lines analyzed
29
Review items
0
False positives ignored
Confirmed security concerns (6)
Critical
Pipe to shell pattern
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
The Docker build pipes a downloaded installer directly into a shell. A compromised network path or upstream installer would execute during image build.
High
System Metadata Uploaded With Results
When uploads are enabled, the payload includes metadata.system and records OS, CPU, Python, memory, and a hostname hash. This goes beyond benchmark scores and may identify the host environment.
The upload payload explicitly includes collect_system_metadata output. The metadata function records platform, CPU, memory, Python, and hostname-derived data.
High
Registration Token Stored In Skill Directory
The registration helper writes the returned token to scripts/.pinchbench/config.json under the skill directory. Tokens in an installed skill tree can be accidentally shared or read by benchmarked agents.
The config path is inside the scripts directory and save_token_config writes the token there. _resolve_token later reads that stored token for uploads.
High
Installed Skills Copied Into Benchmark Workspace
prepare_task_workspace copies every directory under ~/.openclaw/workspace/skills into the benchmark workspace. This may expose private skill content or embedded credentials to the benchmarked model during tasks.
The code iterates over the user skill directory and copies each skill directory into the agent workspace. No allowlist or secret filtering is visible in that block.
Medium
System reconnaissance
"hostname_hash": hash(platform.node()) % 10000,
The code derives a hostname hash as part of system metadata. Even hashed, it contributes to host fingerprinting in uploaded results.
Medium
Agent Session Transcripts Deleted Without Confirmation
cleanup_agent_sessions removes stored session transcript files and sessions.json for the target agent. If the target agent ID collides with a real agent, benchmark runs can delete local conversation history.
The deletion code is explicit and is called before benchmark and judge runs. The risk is reduced because agent IDs are normally benchmark-prefixed, but collision is still possible.
Capability review items (29)
These are real local capabilities that may be expected for this skill, so they require review but are not counted as confirmed malicious behavior.
High
Python subprocess.run
subprocess.run(
The code deletes an OpenClaw agent when its workspace appears stale. It uses argv lists, but it can remove local agent state.
High
Python subprocess.run
result = subprocess.run(
This launches the OpenClaw CLI with benchmark or judge prompts and workspace access. It avoids shell injection, but it triggers external agent execution with local side effects.
High
Python subprocess.run
result = subprocess.run(
This launches the OpenClaw CLI with benchmark or judge prompts and workspace access. It avoids shell injection, but it triggers external agent execution with local side effects.
High
Python subprocess.run
result = subprocess.run(
This launches the OpenClaw CLI with benchmark or judge prompts and workspace access. It avoids shell injection, but it triggers external agent execution with local side effects.
High
Python shutil operations
shutil.rmtree(workspace)
The runner recursively removes the agent workspace before each task. If workspace resolution is wrong, this can delete local files.
High
Python environment access
api_key = os.environ.get("OPENROUTER_API_KEY")
The code reads OPENROUTER_API_KEY from the environment. It is expected for validation, but it handles credential material.
High
Generic API/secret keys
api_key = os.environ.get("OPENROUTER_API_KEY")
The code reads OPENROUTER_API_KEY from the environment. It is expected for validation, but it handles credential material.
High
Generic API/secret keys
"Authorization": f"Bearer {api_key}",
The API key is placed in an Authorization bearer header for an external service. This is legitimate use, but it is sensitive credential transmission.
High
Python exec() function
exec(grading_code, namespace)
The grader executes Python code extracted from task markdown with exec(). Any malicious or modified task file could run arbitrary Python in the benchmark process.
High
Process exec
exec(grading_code, namespace)
The grader executes Python code extracted from task markdown with exec(). Any malicious or modified task file could run arbitrary Python in the benchmark process.
High
Python subprocess.run
result = subprocess.run(
The macOS metadata helper executes sysctl to collect hardware attributes. The arguments are fixed, but the command expands host fingerprinting data.
High
Linux /proc filesystem access
cpuinfo_path = Path("/proc/cpuinfo")
The upload path reads Linux /proc CPU and memory files for system metadata. This collects host fingerprinting information for submission payloads.
High
Linux /proc filesystem access
meminfo_path = Path("/proc/meminfo")
The upload path reads Linux /proc CPU and memory files for system metadata. This collects host fingerprinting information for submission payloads.
High
Python environment access
resolved_official_key = official_key or os.environ.get("PINCHBENCH_OFFICIAL_KEY")
The upload code reads authentication or official submission tokens from environment or local config. These values are sensitive credentials.
High
Python environment access
env_token = os.environ.get("PINCHBENCH_TOKEN")
The upload code reads authentication or official submission tokens from environment or local config. These values are sensitive credentials.
High
Configuration library
return config.get("token")
The upload code reads authentication or official submission tokens from environment or local config. These values are sensitive credentials.
High
Generic API/secret keys
token = data.get("token") or data.get("api_key")
The registration response is parsed for a token or api_key. This is credential material that is later stored and used for uploads.
Medium
Hardcoded URL
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
The Docker build downloads an installer from an external URL. This is expected setup, but it is a real supply-chain dependency.
Medium
Temp directory access
run_root = Path("/tmp/pinchbench")
The runner uses a fixed directory under /tmp for benchmark workspaces. This is functional, but not as safe as a private temporary directory.
Medium
Temp directory access
agent_workspace = Path(f"/tmp/pinchbench/{run_id}/agent_workspace")
The runner uses a fixed directory under /tmp for benchmark workspaces. This is functional, but not as safe as a private temporary directory.
Medium
Hardcoded URL
specific_endpoint = f"https://openrouter.ai/api/v1/models/{encoded_model_id}"
The model validator contacts OpenRouter endpoints and can use an API key in the same request flow. This is legitimate but still external network activity.
Medium
Hardcoded URL
catalog_endpoint = "https://openrouter.ai/api/v1/models"
The model validator contacts OpenRouter endpoints and can use an API key in the same request flow. This is legitimate but still external network activity.
Medium
Temp directory access
workspace = Path(f"/tmp/pinchbench/{run_id}/{task.task_id}")
The fallback workspace uses a predictable path under /tmp. This can expose benchmark files to local path collision or permission risks.
Medium
Python shutil operations
shutil.rmtree(dest_skill_dir)
The runner deletes and copies skill directories inside the benchmark workspace. This can expose or overwrite copied skill content during task setup.
Medium
Python shutil operations
shutil.copytree(skill_dir_src, dest_skill_dir)
The runner deletes and copies skill directories inside the benchmark workspace. This can expose or overwrite copied skill content during task setup.
Medium
Temp directory access
judge_workspace = Path(f"/tmp/pinchbench/judge/{task.task_id}")
The judge uses predictable directories under /tmp for workspaces. This is expected, but not as isolated as a secure temporary directory.
Medium
Temp directory access
workspace = Path("/tmp/pinchbench/judge/workspace")
The judge uses predictable directories under /tmp for workspaces. This is expected, but not as isolated as a secure temporary directory.
Medium
Hardcoded URL
DEFAULT_SERVER_URL = "https://api.pinchbench.com"
The upload module posts benchmark results to the PinchBench API by default. This is intended, but it sends local run data off the machine.
Medium
Python environment access
resolved_server = server_url or os.environ.get("PINCHBENCH_SERVER_URL") or DEFAULT_SERVER_URL
The upload server can be overridden by environment variable. If the environment is compromised, results and auth headers can be sent away from the default API.
Risk Factors
⚙️ External commands (50)
Dockerfile.benchmark:33 scripts/benchmark.py:277 scripts/lib_agent.py:136 scripts/lib_agent.py:179 scripts/lib_agent.py:221 scripts/lib_agent.py:230 scripts/lib_agent.py:578 scripts/lib_agent.py:611 scripts/lib_agent.py:753 scripts/lib_grading.py:107 scripts/lib_grading.py:216 scripts/lib_grading.py:318 scripts/lib_grading.py:319 scripts/lib_upload.py:317 scripts/lib_upload.py:396 scripts/run.sh:6 SKILL.md:23-37 SKILL.md:37-43 SKILL.md:43-44 SKILL.md:44-45 SKILL.md:45-46 SKILL.md:46-47 SKILL.md:47-48 SKILL.md:48-49 SKILL.md:49-50 SKILL.md:50-51 SKILL.md:51-52 SKILL.md:52-53 SKILL.md:53-54 SKILL.md:54-55 SKILL.md:55-56 SKILL.md:56-57 SKILL.md:57-58 SKILL.md:58-59 SKILL.md:59-60 SKILL.md:60-61 SKILL.md:61-62 SKILL.md:62-63 SKILL.md:63-64 SKILL.md:64-65 SKILL.md:65-71 SKILL.md:71 SKILL.md:72 SKILL.md:73 SKILL.md:74-75 SKILL.md:75-76 SKILL.md:76-77 SKILL.md:77-78 SKILL.md:78-84 SKILL.md:84-90
🌐 Network access (13)
📁 Filesystem access (12)
Dockerfile.benchmark:21 Dockerfile.benchmark:30 scripts/benchmark.py:557 scripts/benchmark.py:562 scripts/lib_agent.py:293 scripts/lib_agent.py:298 scripts/lib_agent.py:340 scripts/lib_agent.py:341 scripts/lib_grading.py:157 scripts/lib_grading.py:298 scripts/lib_upload.py:354 scripts/lib_upload.py:355
⚡ Contains scripts (2)
🔑 Env variables (13)
scripts/benchmark.py:270 scripts/lib_agent.py:27 scripts/lib_agent.py:59 scripts/lib_agent.py:59 scripts/lib_agent.py:60 scripts/lib_agent.py:61 scripts/lib_agent.py:67 scripts/lib_upload.py:64 scripts/lib_upload.py:66 scripts/lib_upload.py:135 scripts/lib_upload.py:289 scripts/lib_upload.py:293 scripts/lib_upload.py:159
Detected Patterns
Pipe to shell patternSystem reconnaissance
Audited by: codex View Audit History →
Share & cite this report
Share the versioned assessment report, neutral badge, embed card, and citations. Skillstore reports evidence without deciding whether this Skill is safe.
Copy report link
https://skillstore.io/skills/pinchbench-pinchbench/audits/5?utm_source=security_passport&utm_medium=share&utm_campaign=versioned_reportMarkdown badge
[](https://skillstore.io/skills/pinchbench-pinchbench?utm_source=security_passport_badge)HTML badge
<a href="https://skillstore.io/skills/pinchbench-pinchbench?utm_source=security_passport_badge"><img src="https://skillstore.io/badges/skills/pinchbench-pinchbench/security.svg" alt="Skillstore security assessment" loading="lazy"></a>Embed card
<iframe src="https://skillstore.io/embed/skills/pinchbench-pinchbench.html" title="Skillstore Security Assessment" sandbox="allow-popups allow-popups-to-escape-sandbox" loading="lazy" referrerpolicy="no-referrer" width="420" height="180"></iframe>Academic citations (APA · BibTeX · CFF)
APA citation
pinchbench. (2026). pinchbench security audit report (audit version 5) [Author version 1.0.0]. Skillstore. https://skillstore.io/skills/pinchbench-pinchbench/audits/5BibTeX citation
@techreport{pinchbench-pinchbench-pinchbench-2026,
author = {pinchbench},
title = {pinchbench security audit report (audit version 5)},
institution = {Skillstore},
year = {2026},
number = {5},
url = {https://skillstore.io/skills/pinchbench-pinchbench/audits/5},
note = {Author version 1.0.0}
}CITATION.cff
cff-version: 1.2.0
message: "If you use this Skill, cite its author and this versioned security audit report."
title: "pinchbench security audit report (audit version 5)"
version: "1.0.0"
type: report
authors:
- name: "pinchbench"
date-released: "2026-07-09"
url: "https://skillstore.io/skills/pinchbench-pinchbench/audits/5"
identifiers:
- type: other
value: "skillstore:pinchbench-pinchbench:audit:5"
description: "Skillstore immutable audit report identifier"
Skillstore Score
Why this score Evidence Confidence: Medium 73
Architecture
100
Maintainability
87
Content
71
Community
83
Spec Compliance
What You Can Build
Compare Agent Models
Run the same task suite across models and compare score, cost, time, and category performance.
Validate OpenClaw Setup
Check that an OpenClaw installation can handle tools, memory, files, research, and multi-step workflows.
Publish Benchmark Results
Generate reproducible local results and submit authorized runs to the PinchBench leaderboard.
Try These Prompts
Run A Quick Benchmark
Use PinchBench to run the automated-only suite for [model id] with uploads disabled. Summarize the final score and failed tasks.
Compare Two Models
Run PinchBench for [model A] and [model B] using the same suite and run count. Compare category scores, cost, and time.
Upload A Previous Run
Validate my saved PinchBench result at [file path], upload it to the leaderboard, and report the submission link.
Audit A Custom Task
Review my new PinchBench task file for clear grading, safe workspace files, and reproducible behavior before I run it.
Best Practices
- Start with automated-only and uploads disabled when testing a new model.
- Use the same suite, run count, and timeout multiplier when comparing models.
- Review task files and upload settings before running community benchmarks.
Avoid
- Running all tasks with uploads enabled before validating local behavior.
- Comparing models with different task suites or timeout settings.
- Sharing result files without checking transcripts and system metadata.
Frequently Asked Questions
What does PinchBench evaluate?
It evaluates how OpenClaw agents complete real tasks across files, research, writing, coding, email, memory, and workflows.
Does it require OpenClaw?
Yes. The runner creates or reuses OpenClaw agents and sends benchmark prompts through the OpenClaw command line tool.
Can I run it without uploading results?
Yes. Use the no-upload option when you want local result files only.
How are tasks graded?
Tasks use automated Python checks, LLM judge rubrics, or a hybrid of both methods.
What data may leave my machine?
Uploads can include scores, timing, usage, task metadata, system metadata, and authentication headers for PinchBench.
Should I run it in a production workspace?
No. Use a dedicated sandbox because it creates workspaces, deletes benchmark state, and executes grading code.
Developer Details
Author
pinchbenchLicense
MIT
Author version
v1.0.0
Skillstore revision
r1
Ref
3e4b6c31a74a3bd1a291c98cf585d720cb9fbc88
Maintenance freshness
7/18/2026
Usage
16 downloads · 290 views
File structure