NVIDIA has unveiled a new compressed variant of its Nemotron-3-Super large language model, the Nemotron-Labs-3-Puzzle-75B-A9B. This model represents a significant advancement in efficient, high-throughput language modeling, achieving a 2.03x boost in server throughput compared to its predecessor.
Hybrid Compression Techniques
The model leverages a novel iterative approach combining hardware-aware structural compression with short knowledge distillation recovery phases. This method allows NVIDIA to reduce the model's size significantly—dropping from 120.7 billion total parameters to 75.3 billion—while maintaining active parameters at 9.3 billion. This compression strategy is designed to preserve performance without sacrificing the model’s ability to handle complex tasks.
Performance Gains and Scalability
On a single 8xB200 node, the new model achieves a throughput of 100 tokens per second per user, marking a substantial improvement over the Nemotron-3-Super. Additionally, on a single H100 GPU, the model supports 8 concurrent requests with 1 million tokens, a significant increase from just 1 request in the previous model. These gains are particularly important in production environments where server efficiency and concurrent processing capacity are crucial.
The release of Nemotron-Labs-3-Puzzle-75B-A9B signals NVIDIA's continued focus on optimizing large language models for real-world deployment. By combining structural compression with distillation techniques, the company is paving the way for more efficient, scalable AI systems that can handle the growing demands of enterprise and research applications.
