Introduction
Perplexity has open-sourced Lily, a high-performance inference engine designed to run large language models (LLMs) locally on Apple Silicon chips. This development is significant in the context of on-device AI, where efficiency and performance are paramount. Lily leverages Rust for system-level programming and custom Metal kernels to optimize model execution, demonstrating a sophisticated approach to hardware-software co-design for AI workloads.
What is Lily?
Lily is a local inference engine that enables the execution of large language models directly on a user's device, without relying on cloud-based processing. It is specifically optimized for Qwen3.6-35B-A3B, a large language model with 35 billion parameters, and targets Apple's M5 Max chip. The engine is built using Rust, a systems programming language known for memory safety and performance, and utilizes Metal, Apple's low-level graphics and compute API, to harness the GPU's computational capabilities.
By open-sourcing Lily, Perplexity shares a critical piece of infrastructure that bridges the gap between cutting-edge AI models and practical, efficient on-device deployment. This is particularly relevant in the context of Hybrid Compute, a computing paradigm that combines local and cloud processing to balance performance, privacy, and latency.
How Does Lily Work?
Lily operates by implementing a custom inference engine that efficiently executes the forward pass of a neural network. It is designed to handle two critical phases of inference: prefill and decode. The prefill phase involves processing the initial input tokens to generate a context, while the decode phase iteratively generates new tokens based on this context.
Key technical innovations in Lily include:
- Rust-based Architecture: Rust's zero-cost abstractions and memory safety features allow for efficient, low-level control of compute resources while preventing common programming errors like buffer overflows.
- Custom Metal Kernels: These are specialized GPU programs written in Metal Shading Language (MSL) that are tailored to the specific computational patterns of the Qwen model, enabling optimized memory access and parallel execution.
- Performance Optimization: Lily achieves throughput improvements of 1.23x over MLX-LM's prefill and 1.35x over its decode phases on the M5 Max chip, demonstrating the impact of targeted optimizations.
The engine's architecture is optimized for Apple Silicon's unified memory architecture, where CPU and GPU share a common memory space. This eliminates the overhead of data transfers between separate memory pools, a key performance enabler for on-device AI.
Why Does This Matter?
Lily represents a critical step toward on-device AI scalability and performance. As AI models grow in size and complexity, running them locally on devices like laptops and tablets becomes increasingly important for privacy, latency, and accessibility. However, this requires specialized inference engines that can efficiently utilize the hardware's compute resources.
By open-sourcing Lily, Perplexity contributes to the broader ecosystem of local AI inference, offering developers and researchers a reference implementation that can be adapted for other models and hardware platforms. This is especially valuable for edge AI, where computational resources are constrained.
Moreover, Lily's performance gains over existing solutions like MLX-LM highlight the importance of domain-specific optimizations. While general-purpose libraries provide broad compatibility, custom implementations tailored to specific models and hardware can unlock substantial efficiency improvements.
Key Takeaways
- Lily is a custom inference engine for Apple Silicon, built in Rust with Metal kernels, for running Qwen3.6-35B-A3B locally.
- It achieves performance improvements of 1.23x (prefill) and 1.35x (decode) over MLX-LM on the M5 Max chip.
- The engine demonstrates the value of hardware-software co-design for efficient on-device AI.
- Open-sourcing Lily contributes to the advancement of local AI inference and edge computing.
- It exemplifies how custom optimizations can significantly outperform general-purpose solutions in specialized domains.
