Pi-GRPO — Explained

One line: physics-informed reinforcement-learning post-training for trajectory generation that treats hard physical constraints as a reward floor, so a policy optimized with PPO, DPO, or GRPO improves task quality without learning to produce physically impossible trajectories.

Honest status up front. Pi-GRPO is a workshop paper ("Physics as a Hard Reward Floor", MOSS @ COLM 2026) plus a runnable RL stack. The training code (PPO, DPO, and GRPO under a hybrid physics-aware reward) is real and runs. The headline result, hard-constraint violation rate driven from about 18% under vanilla DPO to about 0% with physics-augmented DPO, with KL bounded across the PPO run, is the paper's reported experimental result at workshop scale, not a large-scale production benchmark. In short: the method and the RL stack are mine and runnable; the numbers are the paper's reported results, which I would scale and re-report on a larger benchmark before treating them as final.

1. What it is and the gap it fills

Reinforcement-learning post-training (the RLHF family: PPO, DPO, GRPO) optimizes a policy against a learned or preference reward. The well-known failure mode is reward hacking: the policy finds outputs that score well under the reward but are wrong in a way the reward does not capture. For trajectory generation that means physically impossible motion, speeds, accelerations, or turns no real vehicle or vessel could execute, that nonetheless please the reward.

Pi-GRPO's idea: make the physics a hard floor, not a soft afterthought. The reward is a hybrid of the task reward and a physics term that heavily penalizes violations of kinematic constraints, so an output that breaks physics can never be high-reward no matter how well it games the task objective. The result is RL that improves the policy while keeping every sample physically admissible.

2. How it works

 policy (trajectory generator)
        │  rollouts / candidate trajectories
        ▼
 reward = task_reward + physics_floor
        │   physics_floor heavily penalizes kinematic-constraint violations
        │   (velocity / acceleration / curvature / turning-rate bounds)
        ▼
 policy update  (PPO clipped objective | DPO preference | GRPO group-relative advantage)
        │
        └────────────── improvement loop ──────────────┘

The same physics-floor reward plugs into three optimizers, so the contribution is the reward shaping, not a single algorithm.

3. The optimizers, briefly

• PPO (Proximal Policy Optimization): a clipped policy-gradient objective with a KL leash to the reference policy, the workhorse of RLHF. Pi-GRPO keeps KL bounded across the run.
• DPO (Direct Preference Optimization): trains directly on preference pairs without a separate reward model. Vanilla DPO is where the physics-violation problem is most visible; the physics-augmented variant is what drives violations toward zero.
• GRPO (Group Relative Policy Optimization): estimates advantage from the relative scores of a group of samples for the same prompt, dropping the value network. It pairs naturally with the physics floor because the floor reorders the group toward admissible samples.

4. The core idea, from scratch

A reward r = r_task + lambda * r_physics, where r_physics is a large negative penalty whenever a trajectory violates a kinematic bound (and zero when it is admissible), behaves like a floor: any policy improvement that increases r_task by hacking physics is cancelled by r_physics, so the optimizer is pushed into the physically feasible region first and then improves task quality within it. This is the RL analogue of the deterministic feasibility gates used elsewhere in the portfolio (the space-time-prism validator in GeoTrace-Agent, the kinematic prior in Pi-DPM): encode the physics as a hard constraint rather than hoping the model learns it.

5. What's real vs scaffold

6. What I'd do next

1. Scale beyond the workshop setting and re-report the violation rate and task quality against standard PPO/DPO/GRPO baselines as a reproducible benchmark.
2. Sweep the physics-floor weight lambda to show the task-quality vs feasibility trade-off explicitly.
3. Connect the floor to the same kinematic models used in the diffusion line (Pi-DPM) so the physics constraint is shared across the generative and RL stacks.