Session Metrics

Session metrics provide critical insight into your session's activity, allowing you to monitor usage and debug performance bottlenecks. The Session Metrics view shows exactly what your training and sampling traffic did over time, including throughput, training utilization, and in-flight requests. You can read the summary panels in the console to track general resource utilization, or export the full timeline to Perfetto for a detailed, per-request view.

The Session Metrics view in the console.

The console panels are a subset of the trace: In-flight sample requests (the console's "Active sample requests"), Training utilization, and the Training, Generation, and Prefill tokens/sec rates. The exported Perfetto trace is a strict superset, adding per-request sizes, latency percentiles, totals, rate-limit events, and the individual request spans.

Exporting and viewing the trace

For the full picture, every request and counter at fine resolution, export a Perfetto trace:

1. Click Download Perfetto on the session page (large sessions take a moment to build).
2. Open ui.perfetto.dev.
3. Drag the .pftrace in, or use Open trace file.

Counter tracks run along the top (grouped: Concurrency, Throughput, Per Request, Latency, Totals); below them, each training and sampling request is a clickable slice on its lane.

An example exported trace

New to Perfetto?

A general-purpose trace viewer; see the official Perfetto UI guide for the full tour. Essentials:

• Zoom/pan: W/S zoom, A/D pan (or Shift+drag, Ctrl+wheel).
• Click a slice for its details; , and . step between slices.
• Drag a time range for aggregate stats over it.
• Find a track: Ctrl+P (Cmd+Shift+P on Mac).

Aggregate stats for a selected time range

Metric Details

Concurrency

• In-flight sample requests (requests): sampling requests running at each moment (the console's "Active sample requests"). Split by weights version: each colored band is a different generation (e.g. a newer checkpoint), stacking to the total, so you see the checkpoint mix in flight.
• Training utilization (%): the percentage of wall-clock time the trainer was actively running forward/backward steps in each interval. Intervals are 1 minute long in Perfetto and can be longer in the console. It will not always be 100%; treat any value above 0% as a sign that training work was happening in that interval.
• Rate-limit events/sec (events/s): how often the session hit a rate or capacity limit, per second.

The Concurrency counter tracks

Throughput

• Training tokens/sec and Generation tokens/sec (tok/s): total training and generated tokens per second. Each request's tokens are spread evenly across the time that request was active (from when it started running to when it finished), instead of being credited all at completion, so a long request reads as a steady rate over its lifetime rather than a spike at the end. Note that because of this, these numbers are an estimate. Intervals are 2.5 minutes long in Perfetto and can be longer in the console.
• Prefill (prompt) tokens/sec (tok/s): total prompt tokens per second, credited at the moment each request starts, because prefill is a burst of work at the very beginning of a request. Note that because of this, these numbers are an estimate.

The Throughput counter tracks

Per Request

• Avg prompt tokens/request and Avg generation tokens/request (tokens): cumulative average size across all sample requests completed so far, so it settles toward the session-wide average.
• Max prompt tokens and Max generation tokens (tokens): largest single prompt and generation in each 1-second interval (outliers).

The Per Request counter tracks

Latency

• Sample request latency p50/p90/p99 (s): sampling request time at each percentile, per 1-second interval.
• Train request latency p50/p90/p99 (s): same for training requests.

The Latency counter tracks

Totals

• Total completed samples (samples): running total of finished sampling requests.
• Total generation / prompt / train tokens (tokens): cumulative tokens generated, prompted, and trained on.

The Totals counter tracks

Per-request spans

Below the counters, each training and sampling request is its own slice on a lane, plus p90/p95/p99 rows that isolate the longest-running samples by span length so the slowest requests stand out. Click one for its timing and to see how requests overlapped; sampling slices are colored by weights version (matching Active sample requests).

Per-request sample spans

Example insights

• Async RL is working. Sample request spans overlap heavily with little gap between them, so new requests start before earlier ones finish. That overlap is the goal: the sampler stays busy instead of idling between steps.
• Higher concurrency lifts throughput. In the second half of this trace, In-flight sample requests and Generation tokens/sec track each other: concurrency rising from about 80 to 110 requests lifts throughput from roughly 1,500 to 1,900 tokens/sec.
• High concurrency with low throughput points to a backend issue. When In-flight sample requests spikes early and stays high while Generation tokens/sec and Prefill (prompt) tokens/sec stay low, requests are piling up faster than they get served. That points to a Tinker backend bottleneck rather than your code, so email us with your session ID.

Tips

• Re-export to refresh. Downloading again picks up new activity; if nothing changed, you get the existing trace back instantly.
• Panels for a glance, trace for a deep dive. Panels for a quick read; the trace to inspect individual requests or correlate a dip with what was running.