NVIDIA has proposed a method using nonuniform tensor parallelism to address infrastructure challenges in massive-scale LLM training. This approach aims to mitigate slowdowns caused by unscheduled interruptions and resource fluctuations that occur when jobs span thousands of GPUs over extended periods.
The technique is designed to improve goodput by handling device unavailability more effectively within tightly interconnected clusters, ensuring that infrequent disruptions do not disproportionately impact overall training efficiency.