# CISPO

CISPO ( [Chen et al., 2024](https://arxiv.org/abs/2506.13585); [Khatri et al., 2024](https://arxiv.org/abs/2510.13786)) is a policy gradient method that uses a clipped importance ratio as a coefficient for the policy gradient. Unlike PPO which clips the objective directly, CISPO clips the ratio and uses it to weight the log probability.

The CISPO objective is:

\[ \mathcal{L}_{\text{CISPO}}(\theta) = \mathbb{E}_{x \sim q}\left[\textbf{sg}\left( \text{clip}\left(\frac{p_\theta(x)}{q(x)}, 1-\epsilon_{\text{low}}, 1+\epsilon_{\text{high}}\right) \right) \cdot \log p_\theta(x) \cdot A(x)\right] \

This is implemented as:

```python
# Compute probability ratio
prob_ratio = torch.exp(target_logprobs - sampling_logprobs)
# Apply clipping
clipped_ratio = torch.clamp(prob_ratio, clip_low_threshold, clip_high_threshold)
# Compute CISPO objective (detach the clipped ratio)
cispo_objective = clipped_ratio.detach() * target_logprobs * advantages
# CISPO loss is negative of objective
loss = -cispo_objective.sum()
```

**Input tensors:**

- `target_tokens: array[(N,), int]` — Target token IDs (from the sampler qqq)
- `logprobs: array[(N,), float]` — `sampling_logprobs` for the tokens
- `advantages: array[(N,), float]` — Advantage values for RL

**Output tensors:**

- `logprobs: array[(N,), float]` — `target_logprobs` for the tokens

**Output diagnostics:**

- `loss:sum` (scalar) — Sum of CISPO losses

## Choosing clipping thresholds

Because the clipped ratio is a **detached coefficient** on log⁡pθ\log p_\thetalogpθ​ (rather than clipping the objective like PPO), CISPO never zeros a token's gradient — it only bounds the coefficient's magnitude. This is what makes the choice of thresholds, especially the lower one, matter.

**The default is a one-sided clip.** CISPO defaults to disabling the lower bound and only capping the upper side (`clip_low_threshold=0.0`, `clip_high_threshold=4.0`), so you get this without passing `loss_fn_config`. To set it explicitly:

```python
fwd_bwd_future = await training_client.forward_backward_async(
    data=data,
    loss_fn="cispo",
    loss_fn_config={"clip_low_threshold": 0.0, "clip_high_threshold": 4.0}
)
fwd_bwd_result = await fwd_bwd_future.result_async()
```

This is a safe default in any setting. With no lower bound, the worst case is that the coefficient just falls back toward the plain importance-sampling weight, which is well-behaved. A positive `clip_low_threshold` (e.g. `0.8`), by contrast, floors the coefficient even for tokens whose ratio has dropped well below 1 — tokens the policy has already moved away from — which removes the attenuation importance sampling provides for stale tokens. On-policy that rarely matters, but off-policy (async, where the sampler qqq lags the trainer pθp_\thetapθ​) it can set up a positive feedback loop: the sampler/trainer KL grows, more tokens fall outside the band, the bias grows, KL grows further. So there is little downside to dropping the lower bound and a real upside off-policy.

This matches both papers that introduced and scaled CISPO. MiniMax-M1 ( [Chen et al., 2024](https://arxiv.org/abs/2506.13585)) report: _"we did not impose_ _a lower bound on the IS weight by setting ϵlowIS\epsilon_{\text{low}}^{\text{IS}}ϵlowIS​ to a large value; instead, we_ _only tuned ϵhighIS\epsilon_{\text{high}}^{\text{IS}}ϵhighIS​._" ScaleRL ( [Khatri et al., 2024](https://arxiv.org/abs/2510.13786)) use clip(r,0,ϵmax⁡)\mathrm{clip}(r, 0, \epsilon_{\max})clip(r,0,ϵmax​) and find CISPO is **largely insensitive** to the upper bound — ϵmax⁡\epsilon_{\max}ϵmax​ of 4, 5, and 8 perform the same (Fig. 19b) — so a loose upper bound in that range is a safe default.