Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel
Get smart on it
NVIDIA NeMo AutoModel is a library that accelerates the training of Mixture-of-Experts (MoE) models by building on top of HuggingFace Transformers v5 and adding specialized optimizations like Expert Parallelism and fused dispatch operations. MoE models have become the dominant architecture for frontier models but introduce training challenges such as routing tokens across experts and managing GPU memory efficiently. NeMo AutoModel achieves 3.4-3.7 times higher training throughput and 29-32 percent less GPU memory use compared to native Transformers v5 while maintaining API compatibility, meaning users can switch to it with only a single import line change and no other code modifications.
Related news
Vibe coding platform Base44 launches own model as AI startups seek defensibility
Wix-owned vibe coding platform Base44 has started rolling out its own AI model — with hopes that it will eventually outperform frontier models.
DiScoFormer: One transformer for density and score, across distributions
DiScoFormer is a transformer model that estimates both the density and score of a data distribution in a single forward pass without requiring retraining for new distributions. The density describes where data points cluster, while the score, the gradient of log-density, points toward more probable regions and is used in diffusion-based generative models, Bayesian sampling, and scientific simulations. Existing methods force a trade-off: classical kernel density estimation works on any distribution but loses accuracy in high dimensions, while neural score-matching models stay accurate in high dimensions but must be retrained for each new distribution. DiScoFormer significantly outperforms kernel density estimation, cutting score error by 6.5 times and density error by more than 37 times in 100 dimensions, while generalizing to distributions with shapes and complexities not seen during training.
