[update] readme content

README_en.md

@@ -913,17 +913,7 @@ Before introducing the implementation of specific algorithms, let me first descr
 
 The essence of all RL algorithms is just optimizing an expectation:
 
-$$
-\mathcal{J}_{PO} =
-\mathbb{E}_{q \sim P(Q),\, o \sim \pi(O \mid q)}
-\left[
-  \underbrace{f(r_t)}_{\text{policy term}}
-  \cdot
-  \underbrace{g(A_t)}_{\text{advantage term}}
-  -
-  \underbrace{h(\text{KL}_t)}_{\text{regularization term}}
-\right]
-$$
+$$\mathcal{J}_{PO} = \mathbb{E}_{q \sim P(Q), o \sim \pi(O|q)} \left[ \underbrace{f(r_t)}_{\text{policy term}} \cdot \underbrace{g(A_t)}_{\text{advantage term}} - \underbrace{h(\text{KL}_t)}_{\text{regularization term}} \right]$$
 
 During training, one only needs to **minimize the negative objective function**, i.e.: $\mathcal{L}_{PO} = -\mathcal{J}_{PO}$
 
@@ -939,7 +929,7 @@ This framework contains only three core components:
 |--------|---------|-------------|-------|
 | $q$ | Question/Prompt | Sampled from dataset $P(Q)$ | - |
 | $o$ | Model output sequence | Generated by policy $\pi$ | - |
-| $r_t$ | Probability ratio | $r_t(\theta) = \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{\mathrm{old}}(o_t \mid q, o_{<t})}$ | $(0, +\infty)$ |
+| $r_t$ | Probability ratio | $r_t = \frac{\pi_\theta(o_t \mid q, o_{<t})}{\pi_{ref}(o_t \mid q, o_{<t})}$ | $(0, +\infty)$ |
 | $A_t$ | Advantage function | Measures how much better a certain action is compared to the baseline | $(-\infty, +\infty)$ |
 | $\text{KL}_t$ | KL divergence | Prevents the policy from drifting too far from the reference model | $[0, +\infty)$ |
 
@@ -1094,9 +1084,9 @@ PPO is a very classic reinforcement learning algorithm proposed by OpenAI in 201
 $$\mathcal{L}_{PPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t)\right] + \beta \cdot \mathbb{E}[\text{KL}]$$
 
 Where:
-- **Policy term**: $\min\left(r_t A_t,\ \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon)A_t\right)$ (clips the probability ratio and advantage together to prevent overly aggressive updates)
-- **Advantage term**: $A_t$ (usually estimated by the Critic, or computed with GAE)
-- **Regularization term**: $\beta \cdot \mathbb{E}[\text{KL}]$ (global KL divergence constraint)
+- **Policy term**: $f(r_t) = \min(r_t, \text{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (clips probability ratio to prevent overly aggressive updates)
+- **Advantage term**: $g(A_t) = R - V(s)$ (estimates value function through Critic network)
+- **Regularization term**: $h(\text{KL}_t) = \beta \cdot \mathbb{E}[\text{KL}]$ (global KL divergence constraint)
 
 Compared to DPO,
 - DPO (Off-Policy): Training data consists of static preference pairs (chosen vs rejected), which can be reused across multiple training epochs, like traditional supervised learning. High data efficiency, low cost, and no Reward Model needed.
@@ -1136,9 +1126,9 @@ In early 2025, as DeepSeek-R1 exploded in popularity, GRPO from the DeepSeekMath
 $$\mathcal{L}_{GRPO} = -\mathbb{E}\left[\min(r_t \cdot A_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon) \cdot A_t) - \beta \cdot \text{KL}_t\right]$$
 
 Where:
-- **Policy term**: $\min\left(r_t A_t,\ \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon)A_t\right)$ (uses symmetric clip on the probability ratio)
-- **Advantage term**: $A_{i,j} = \frac{R_{i,j} - \mu_i}{\sigma_i + \epsilon}$ (intra-group normalization, without training an additional Critic)
-- **Regularization term**: $\beta \cdot \text{KL}_t$ (token-level KL divergence constraint)
+- **Policy term**: $f(r_t) = \min(r_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon))$ (uses symmetric clip on the probability ratio)
+- **Advantage term**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (intra-group normalization, eliminating the Critic network)
+- **Regularization term**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token-level KL divergence constraint)
 
 For the same question, the model generates N responses and computes their respective rewards, then uses the intra-group average reward as a baseline. Responses above the baseline are encouraged, and responses below the baseline are suppressed, thus no additional critic network needs to be trained.
 
@@ -1170,11 +1160,11 @@ CISPO does not focus on redesigning the group baseline. Instead, it uses a very
 
 **CISPO Loss**:
 
-$$\mathcal{L}_{CISPO} = -\mathbb{E}\left[\min(r_t, \varepsilon_{\mathrm{high}}) \cdot A_t \cdot \log \pi_\theta(a_t|s) - \beta \cdot \text{KL}_t\right]$$
+$$\mathcal{L}_{CISPO} = -\mathbb{E}\left[\min(r_t, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta(a_t|s) - \beta \cdot \text{KL}_t\right]$$
 
 Where:
-- **Policy term**: $f(r_t) = \min(r_t, \varepsilon_{\mathrm{high}}) \cdot \log \pi_\theta(a_t|s)$ (the ratio serves only as a clipped weight)
-- **Advantage term**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group} + \epsilon}$ (can directly reuse GRPO's intra-group relative advantage)
+- **Policy term**: $f(r_t) = \min(r_t, \varepsilon_{max}) \cdot \log \pi_\theta(a_t|s)$ (ratio serves only as a clipped weight)
+- **Advantage term**: $g(A_t) = \frac{R - \mu_{group}}{\sigma_{group}}$ (can directly reuse GRPO's intra-group relative advantage)
 - **Regularization term**: $h(\text{KL}_t) = \beta \cdot \text{KL}_t$ (token-level KL divergence constraint)
 
 CISPO, building on GRPO, rewrites the policy term that was easily clipped into a constant into the form "clipped weight × log probability". This way, even if the ratio is truncated, the gradient path is not truncated along with it. Therefore, CISPO can be directly viewed as a loss variant of GRPO to implement, rather than maintaining a separate standalone script. No separate experiment is listed here. One only needs to set `loss_type` to `cispo` in `train_grpo.py`; the rest of the training process still follows GRPO's group sampling, reward computation, and advantage construction logic.
@@ -1279,9 +1269,9 @@ Returning to the "**unified framework**", the table below summarizes how differe
 | Algorithm | Policy term $f(r_t)$ | Advantage term $g(A_t)$ | Regularization term $h(\text{KL}_t)$ | Number of training models |
 |-----------|---------------------|------------------------|-------------------------------------|--------------------------|
 | **DPO** | $\log r_w - \log r_l$ | No explicit advantage term | Implicit in $\beta$ | 1 (2 participate in forward) | 
-| **PPO** | $\min(r_t A_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon)A_t)$ | $A_t$ (typically estimated by Critic / GAE) | $\beta \cdot \mathbb{E}[\text{KL}]$ | 2 |
-| **GRPO** | $\min(r_t A_t, \mathrm{clip}(r_t, 1-\varepsilon, 1+\varepsilon)A_t)$ | $\frac{R - \mu}{\sigma + \epsilon}$ | $\beta \cdot \text{KL}_t$ | 1 |
-| **CISPO** | $\min(r_t, \varepsilon_{\mathrm{high}}) \cdot A_t \cdot \log \pi_\theta$ | $\frac{R - \mu}{\sigma + \epsilon}$ | $\beta \cdot \text{KL}_t$ | 1 |
+| **PPO** | $\min(r, \text{clip}(r))$ | $R - V(s)$ | $\beta \cdot \mathbb{E}[\text{KL}]$ | 2 | 
+| **GRPO** | $\min(r, \text{clip}(r))$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 |
+| **CISPO** | $\mathrm{clip}(r, 0, \varepsilon_{max}) \cdot A_t \cdot \log \pi_\theta$ | $\frac{R - \mu}{\sigma}$ | $\beta \cdot \text{KL}_t$ | 1 | 
 
 **To put it plainly, these RL algorithms are not isolated from one another. From a unified optimization perspective, they are natural variants formed by making different design trade-offs on the same objective function, presenting a beautifully self-consistent unity.**