# Analyze Prefill Logprob Sensitivity

Model safety teams need reproducible evidence about when prefills make exploit behavior more likely. This skill guides Claude, Codex, and Claude Code through trajectory and logprob analysis.

## Install

```bash
npx skillstore add eleutherai/logprob-prefill-analysis
```

## Metadata

- Status: approved
- Slug: eleutherai-logprob-prefill-analysis
- Skillstore revision: r1
- Version status: missing
- Tree hash: be81022deb55729577c8798ac0bd47884ddfcc056e79e91c73858c367a6d0d11
- Author: EleutherAI
- GitHub username: EleutherAI
- License: MIT
- Repository: https://github.com/EleutherAI/rh-indicators/tree/main/.claude/skills/logprob-prefill-analysis
- Ref: 30c73eac2afe762f6aa9c4553158769369d47351
- Supported tools: Claude, Codex, Claude Code
- Audit status: complete
- Agent install advisory: confirmation\_required
- Manual install advisory: allowed
- Artifact signature: available
- Audit attestation: unavailable
- Human verification: not\_verified
- Risk factors: external\_commands, network
- Quality score: 38
- Quality tier: warning
- Public page: https://skillstore.pages.dev/skills/eleutherai-logprob-prefill-analysis
- Manifest: https://skillstore.pages.dev/api/skills/eleutherai-logprob-prefill-analysis/manifest

## Capabilities

- Runs a documented full prefill sensitivity analysis pipeline from a chosen run directory.
- Discovers checkpoints and prefill levels from a sensitivity experiment configuration.
- Computes token-based trajectories for minimum prefill levels across checkpoints.
- Calculates prefill logprob scores for checkpoint and prefill combinations.
- Merges token and logprob metrics into comparison tables and plots.
- Records run context with configuration, metadata, status, and output locations.

## Use Cases

- Reproduce a Safety Run: Run the documented pipeline again and compare outputs against a known sensitivity experiment.
- Compare Checkpoint Susceptibility: Track when model checkpoints become easier to steer with exploit-oriented prefills.
- Prepare Evaluation Evidence: Create tables, plots, and run context files for internal safety review.

## Prompt Templates

### Check Readiness

```
Use this skill to explain which files, checkpoints, and tools I need before running prefill sensitivity analysis.
```

### Plan a Dry Run

```
Use this skill to plan a dry run for my sensitivity run path and identify missing prerequisites.
```

### Compare Metrics

```
Use this skill to compare token-based trajectories and prefill logprob results for my run directory.
```

### Reproduce Full Pipeline

```
Use this skill to reproduce the full pipeline, validate run context files, and flag unsafe or incomplete outputs.
```

## Limitations

- Requires local project scripts, model checkpoints, vLLM, and djinn to already be available.
- Assumes the expected results directory layout and file names from the source project.
- Does not provide model checkpoints, datasets, or benchmark problem definitions.
- May require significant GPU memory when computing logprobs for large checkpoints.

## Best Practices

- Run a dry plan first and confirm checkpoint paths before starting GPU workloads.
- Use small sample limits when validating setup, then rerun full analyses for final evidence.
- Keep generated exploit reasoning data in authorized research environments only.

## Anti Patterns

- Do not run the pipeline against unknown checkpoints or datasets without permission.
- Do not treat a single threshold as a complete safety conclusion.
- Do not publish raw exploit reasoning traces or environment metadata without review.

## Security Audit

- Audited at: 2026-07-05T15:44:28.738\+00:00
- Summary: The static command findings are false positives caused by Markdown code fences and inline code formatting, plus localhost-only examples. The main residual risk is semantic: the skill teaches a dual-use workflow for eliciting and analyzing exploit-oriented model behavior.

## Stats

- Views: 261
- Downloads: 9
- Favorites: 0
- Popularity score: 0