media Hugging Face Forums · 3d ago · open_models

Proposal for progressive model growth and an open hardware profile for Transformers

from English

A proposal on Hugging Face discusses two interconnected problems in AI: the reliance on training separate large models and the ecosystem's dependence on general-purpose GPUs.

The author suggests "progressive model growth," where a model structurally expands (e.g., 4B to 8B) while preserving learned representations, rather than starting from scratch for each size.
This approach complements fixed-size knowledge transfer, allowing models to absorb more capability before structural expansion is necessary.
To address hardware limitations, the author proposes an "AI Hardware Profile" for Transformers, defining a stable set of common operations (like GEMM and attention) for specialized ASIC vendors.
The proposal argues that model releases should benchmark efficiency metrics such as tokens per joule and memory requirements alongside parameter counts.

The author believes these changes could help local AI users overcome VRAM limits and provide hardware manufacturers with a clear optimization target.

Importance 1/3 Trust 1/3 Hugging Face Forums Hardware & chips Training methods

Related articles

arxiv arXiv cs.AI · 62d ago · 29 views

SLAI T-Rex enables full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

SLAI introduces T-Rex, an end-to-end optimization framework for the full-parameter post-training of trillion-parameter-scale MoE models on Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, the system addresses memory pressure and communication overhead through hierarchical parallelism and kernel execution optimizations.

arxiv arXiv cs.CL · 62d ago · 23 views

SLAI T-Rex enables full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

SLAI introduces T-Rex, an end-to-end optimization framework for the full-parameter post-training of trillion-parameter-scale MoE models on Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, the system addresses memory pressure and communication overhead through hierarchical parallelism and kernel execution optimizations.

media r/LocalLLaMA · 69d ago · 9 views

Pluralis Research runs RL post-training on 14 consumer Macs

Pluralis Research has successfully executed the first RL post-training run where the entire rollout fleet operated on consumer Macs connected via the open internet. The setup utilized 14 Macs across four countries for int8 inference with MLX, while a single B200 handled bf16 gradient updates.

lab NVIDIA Technical Blog · 76d ago · 22 views

NVIDIA guide on GPU-initiated communication for molecular dynamics

The article introduces a practical guide focused on GPU-initiated communication techniques for large-scale molecular dynamics simulations.

lab NVIDIA Technical Blog · 78d ago · 21 views

NVIDIA proposes nonuniform tensor parallelism to enhance goodput in large-scale LLM training

NVIDIA has proposed a method using nonuniform tensor parallelism to address infrastructure challenges in massive-scale LLM training. This approach aims to mitigate slowdowns caused by unscheduled interruptions and resource fluctuations that occur when jobs span thousands of GPUs over extended periods.