One line: physics-informed reinforcement-learning post-training for trajectory generation that treats hard physical constraints as a reward floor, so a policy optimized with PPO, DPO, or GRPO improves task quality without learning to produce physically impossible trajectories.
Honest status up front. Pi-GRPO is a runnable RL stack whose reward design was submitted as a workshop paper ("Physics as a Hard Reward Floor", MOSS @ COLM 2026) and rejected. The reviewer's core objection, that the paper's reward-hacking probe was a synthetic construction rather than a test of a trained model, was correct, and my own follow-up audit confirmed it point by point. That audit also found something more interesting than the number the paper led with: a real failure mode in how GRPO's own training dynamics interact with a hard constraint penalty, verified with a reproducible script. The GRPO training loop and the hybrid physics-aware reward are real and run. PPO and DPO are implemented but were never trained, so no result exists for either.
Reinforcement-learning post-training (the RLHF family: PPO, DPO, GRPO) optimizes a policy against a learned or preference reward. The well-known failure mode is reward hacking: the policy finds outputs that score well under the reward but are wrong in a way the reward does not capture. For trajectory generation that means physically impossible motion, speeds, accelerations, or turns no real vehicle or vessel could execute, that nonetheless please the reward.
Pi-GRPO's idea: make the physics a hard floor, not a soft afterthought. The reward is a hybrid of the task reward and a physics term that heavily penalizes violations of kinematic constraints, so an output that breaks physics can never be high-reward no matter how well it games the task objective. The same reward is shared unchanged across PPO, DPO, and GRPO by construction, so the contribution is the reward shaping, not a single algorithm.
policy (trajectory generator)
│ rollouts / candidate trajectories
▼
reward = task_reward + physics_floor
│ physics_floor heavily penalizes kinematic-constraint violations
│ (velocity / acceleration / curvature / turning-rate bounds)
▼
policy update (PPO clipped objective | DPO preference | GRPO group-relative advantage)
│
└────────────── improvement loop ──────────────┘A reward r = r_task + lambda * r_physics, where r_physics is a large negative penalty whenever a trajectory violates a kinematic bound, is meant to behave like a floor: any policy improvement that increases r_task by hacking physics should be cancelled by r_physics, so the optimizer is pushed into the feasible region first. That is the design intent, and it is true for the reward function in isolation.
It is not true once GRPO's own training mechanics are in the loop. GRPO does not train on the raw reward. It z-scores each completion against the mean and standard deviation of its own sampling group before computing a gradient. That normalization step is where the floor leaks.
The finding, verified with a reproducible script (run_groupnorm.py). Take a group of four sampled completions with speeds 20, 25, 60, and 200 against a 30-knot envelope, under the reward described above. The completion at 60 knots violates the envelope. Under raw reward it scores far worse than the two feasible completions, as intended. Under GRPO's group-normalized advantage, it receives positive advantage, +0.247, because it merely beats the group's worst member (200 knots) rather than being compared against the envelope itself. The optimizer reinforces it. Sampling 200,000 random four-member groups that contain at least one violator, 85.6% of them reinforce a violator this way, and the effect does not go away by raising the penalty: multiplying it by a million changes that advantage by under 0.001, because the group's own standard deviation absorbs the increase. A hard, unbounded penalty is exactly as leaky as a small one once it passes through group normalization.
This is the honest version of the paper's headline claim. The physics floor is real at the level of the reward function. Whether it holds during actual GRPO training depends on what else is in the sampling group, and for many groups it does not hold at all.
| Piece | Status |
|---|---|
| RL training stack (PPO / DPO / GRPO) with the hybrid physics reward | Real, runnable |
| Physics-floor reward (kinematic-constraint penalty) | Real |
| GRPO group-normalization failure mode | Real, measured, reproducible (run_groupnorm.py) |
| A clean, matched-budget violation-rate comparison | Not done. The workshop paper's own comparison was confounded (its two arms ran different step counts on different devices) and was withdrawn rather than reported |
| DPO / PPO training results | None exist. Both are implemented, neither was trained |
| Large-scale benchmark vs standard RLHF baselines | Next step, not done |
The real open question this work turned up: when does GRPO's group normalization act as a helpful adaptive gradient, and when does it actively invert a safety constraint? Both are true depending on the reward's shape and the group's composition, and no existing analysis draws the line between them. That is a sharper, more useful research question than the one the workshop paper originally asked, and it is where the next version of this work is headed: