# tinker_cookbook.stores.EvalStore

## _class_ [**tinker_cookbook.stores.EvalStore**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L71)()

Manages evaluation runs across checkpoints.

All file I/O goes through the [`Storage`](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/stores/storage/) protocol, making this backend-agnostic (local disk, S3, GCS).

Pickle-serializable when freshly constructed.

### [**url**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L91)( _path_)

Return a human-readable URI for a path within this eval store.

**Parameters:**

- [**path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L91) ( _str_)

**Returns:** _str_

### [**create_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L134)( _model_name_, _benchmarks_, _checkpoint_path_, _checkpoint_name_, _config_, _run_id_)

Create a new evaluation run and return its run_id.

**Parameters:**

- [**model_name**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L136) ( _str_)
- [**benchmarks**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L137) ( _list[str]_
- [**checkpoint_path**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L138) ( _str | None_)
- [**checkpoint_name**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L139) ( _str | None_)
- [**config**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L140) ( _dict | None_)
- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L141) ( _str | None_)

**Returns:** _str_

### [**run_dir**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L174)( _run_id_)

Return filesystem path for backward compat with BenchmarkConfig.save_dir.

Only works with LocalStorage (returns a local path string).
For cloud backends, use `url()` on the storage directly.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L174) ( _str_)

**Returns:** _str_

### [**finalize_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L188)( _run_id_)

Collect scores from benchmark results and update metadata.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L188) ( _str_)

**Returns:** _RunMetadata_

### [**list_runs**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L206)()

List all evaluation runs, most recent first.

**Returns:** _list[RunMetadata]_

### [**read_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L218)( _run_id_)

Load metadata for a specific run. Raises FileNotFoundError if missing.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L218) ( _str_)

**Returns:** _RunMetadata_

### [**list_benchmarks**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L225)( _run_id_)

List benchmark names that have results for a run.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L225) ( _str_)

**Returns:** _list[str]_

### [**read_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234)( _run_id_, _benchmark_)

Get aggregated result for a benchmark.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L234) ( _str_)

**Returns:** _BenchmarkResult | None_

### [**read_trajectories**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L248)( _run_id_, _benchmark_, _correct_only_, _incorrect_only_, _errors_only_)

Get trajectories with optional filtering.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L250) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L251) ( _str_)
- [**correct_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L253) ( _bool_)
- [**incorrect_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L254) ( _bool_)
- [**errors_only**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L255) ( _bool_)

**Returns:** _list[StoredTrajectory]_

### [**read_single_trajectory**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L276)( _run_id_, _benchmark_, _idx_)

Get a single trajectory by index (O(n) scan — loads all trajectories).

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) ( _str_)
- [**idx**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L277) ( _int_)

**Returns:** _StoredTrajectory | None_

### [**read_summary**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L285)( _run_id_)

Read the combined summary for a run, or `None` if missing.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L285) ( _str_)

**Returns:** _dict[str, Any] | None_

### [**write_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291)( _run_id_, _result_)

Save a benchmark result.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291) ( _str_)
- [**result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L291) ( _BenchmarkResult_)

**Returns:** _None_

### [**write_trajectory**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298)( _run_id_, _benchmark_, _traj_)

Append one trajectory to the JSONL file.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) ( _str_)
- [**traj**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L298) ( _StoredTrajectory_)

**Returns:** _None_

### [**write_summary**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302)( _run_id_, _results_)

Save a combined summary.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302) ( _str_)
- [**results**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L302) ( _dict[str, BenchmarkResult]_)

**Returns:** _None_

### [**delete_run**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L314)( _run_id_)

Delete all data for a run. Idempotent (no error if already gone).

Removes metadata, summary, and all benchmark result/trajectory files.
The `runs.jsonl` index is append-only and not modified; `list_runs()` checks for `metadata.json` existence so deleted runs are excluded.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L314) ( _str_)

**Returns:** _None_

### [**alist_runs**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L331)()

Async version of `list_runs`.

**Returns:** _list[RunMetadata]_

### [**aread_trajectories**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L335)( _run_id_, _benchmark_, **_kw_)

Async version of `read_trajectories`.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) ( _str_)
- [**__kw**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L336) ( _Any_)

**Returns:** _list[StoredTrajectory]_

### [**aread_result**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341)( _run_id_, _benchmark_)

Async version of `read_result`.

**Parameters:**

- [**run_id**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341) ( _str_)
- [**benchmark**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/stores/eval_store.py#L341) ( _str_)

**Returns:** _BenchmarkResult | None_