TRL v1.14 introduces LoRA support to the AsyncGRPOTrainer, allowing it to train a Low-Rank Adaptation adapter and synchronize only that small file to vLLM inference workers. This architecture decouples training and generation jobs on separate machines by using a shared Storage Bucket as a filesystem bridge, eliminating the need for NCCL or direct network communication between nodes.

  • A rank-1 adapter is only a few megabytes, enabling it to travel through a mounted Storage Bucket instead of over NCCL.
  • The trainer and vLLM replicas run as separate Hugging Face Jobs on different machines.
  • A proxy server adds authentication headers and routes rollouts to the replica holding the relevant KV prefix while broadcasting adapter updates.
  • Five runs reduced training time for 500 steps from 3 hours 27 minutes to 53 minutes.

This setup allows the AsyncGRPOTrainer to operate effectively in distributed environments where nodes cannot share a local disk or form an NCCL group, significantly accelerating reinforcement learning workflows.