CARLA-GS decouples representation, reasoning, and physics for autonomous driving corner-case synthesis

Researchers propose CARLA-GS, a modular pipeline that synthesizes photorealistic corner cases for autonomous driving by decoupling visual representation, semantic reasoning, and physics-based execution. The system reconstructs editable Gaussian scenes from real driving data, uses a multi-agent LLM to identify risky interactions and generate intent-level waypoints, and delegates low-level motion control to CARLA with a PID controller.

• Reconstructs editable Gaussian scenes with geometry-consistent constraints from real driving data.
• Employs a multi-agent LLM for scene-level reasoning to detect risks and generate waypoint trajectories.
• Uses CARLA and a PID controller to ensure kinematic and dynamic feasibility of vehicle motion.
• Re-projects simulated vehicle states into the Gaussian scene for ego-centric rendering.

Experiments on the Waymo Open Dataset demonstrate that the framework enables controllable generation of photorealistic, spatiotemporally consistent videos aligned with semantic intent and physical constraints.