Tencent has made a significant contribution to the field of large language model (LLM) inference by open-sourcing AngelSpec, a unified training framework designed for speculative decoding on HY3 models. This development marks a notable advancement in improving the efficiency of LLM inference, particularly in scenarios where speed and resource utilization are critical.
Introducing DFly: A Novel Drafter for Speculative Decoding
At the heart of AngelSpec is DFly, a block-diffusion drafter that supports hybrid target conditioning and incorporates a hidden-correction autoregressive head. These features allow DFly to generate draft tokens more efficiently and accurately, reducing the computational overhead typically associated with traditional autoregressive decoding. The framework also integrates D-cut, a runtime-adaptive verification budgeting mechanism, which dynamically adjusts the verification process based on the context, further enhancing performance.
Performance Gains and Scalability
According to Tencent's benchmarks, DFly-8, when deployed on the HY3-295B-A21B model with a tensor parallelism (TP) of 8, achieves a speedup of 1.98 to 2.40 times compared to autoregressive decoding. This performance boost is observed across a range of concurrency levels, from 4 to 64, demonstrating the framework's scalability and adaptability in real-world deployment scenarios. These results underscore the potential of speculative decoding to significantly reduce latency in LLM inference without compromising output quality.
Implications for the AI Community
The open-sourcing of AngelSpec is expected to accelerate innovation in speculative decoding techniques, offering researchers and developers a robust foundation for experimenting with novel architectures and training methodologies. By providing support for multiple architectures and integrating advanced features like hybrid conditioning and adaptive verification, Tencent is empowering the broader AI community to push the boundaries of efficient LLM inference.
This initiative aligns with the growing trend of open-source collaboration in AI, where companies contribute tools and frameworks to foster collective progress in the field.
