nanochat 核心结构——RL 对齐

chat_rl.py (RL 对齐)# RL 训练脚本python -m scripts.chat_rl \--policy_model ./runs/speedrun/checkpoint.pt \--reference_model ./runs/speedrun/checkpoint.pt \--reward_model anthropic_hhrl \--algorithm ppo**支持算法**- PPO (Proximal Policy Optimization)- GRPO (Group Relative Policy Optimization)- DPO (Direct Preference Optimization)