Perplexity open-sources Lily for Qwen3.6 inference on Apple silicon

• Lily is now open source as a local Apple-silicon engine for Qwen3.6-35B-A3B, and the Lily announcement ties it directly to Perplexity Computer's Hybrid Compute workflow.
• The implementation is a compact Rust runtime plus custom Metal kernels, with an OpenAI-compatible chat endpoint and no PyTorch or MLX execution path, according to a Perplexity technical overview.
• On Perplexity's M5 Max test, Lily averaged 1.23x higher prefill throughput and 1.35x higher decode throughput than MLX-LM, as Perplexity's benchmark post reports.
• Hybrid Compute assigns privacy-sensitive subtasks to a Mac while cloud models handle research, planning, and reasoning, as AravSrinivas's post describes.

The technical write-up records 795 GPU kernel launches in one batch-one decode step, though only 555 stages are sequential. It also puts the Q4 checkpoint at 19.4 GB, down from roughly 70 GB in bfloat16, with weights expanded only in small GPU tiles.

Lily targets one pairing, Qwen3.6-35B-A3B on Apple silicon, rather than serving as a general local-model runtime. Its standalone demo exposes an OpenAI-compatible chat-completions API; the Rust process loads checkpoints, holds session state, and drives generation.

Perplexity's technical post fixes the runtime around Qwen's unusual workload:

• 35 billion total parameters, with about 3 billion activated for each token.
• A router selects eight of 256 experts, plus one shared expert.
• 10 full-attention layers and 30 Gated DeltaNet layers.

A LocalLLaMA post captured the trade-off succinctly: Lily is optimized for one model to get the best Apple-silicon performance.

Prefill processes many prompt rows and can reuse weights. Batch-one decode produces one token at a time, so memory traffic dominates once that reuse disappears.

• Prefill: Perplexity's implementation notes use matrix-oriented GEMM paths, dequantize Q4 weights inside the grouped multiplication, keep expert routing on the GPU, retain Gated DeltaNet state in registers, and process long prompts in bounded chunks.
• Decode: The same notes use vector-oriented GEMV, keep the selected next token in a GPU-resident slot, and schedule independent kernels concurrently when dependencies allow.

Perplexity's benchmark methodology tested Qwen3.6-35B-A3B Q4 at batch 1 on an M5 Max MacBook Pro with a 40-core GPU and 128 GB unified memory. It takes arithmetic means across ten equally weighted prompt lengths for prefill and ten context lengths for decode, each ranging from 256 to 128K tokens.

• Prefill: 3,388 tokens/s with MLX-LM to 4,156 with Lily, a 1.23x result.
• Decode: 126.4 tokens/s with MLX-LM to 170.0 with Lily, a 1.35x result.
• At 4K prompt and 4K context: 4,737.5 to 5,749.9 prefill tokens/s, and 140.9 to 186.6 decode tokens/s.

The comparison measures a vendor-built, model-specific engine against a general-purpose framework on one high-memory M5 Max configuration. It establishes Lily's narrow target, not a ranking across Mac inference stacks or models.

Perplexity says Hybrid Compute begins in the cloud and delegates private-file steps to the Mac, then combines the work without sending sensitive data to its servers. When its local gate detects PII, the Privacy Gate demo shows three choices: process locally, upload anyway, or skip the step.

The separately open-sourced PII masking model is a roughly 600M-parameter bidirectional Qwen3 encoder. Its token-classification head emits 37 BIOES tags covering nine PII categories, including account numbers, private URLs, addresses, phone numbers, and secrets; a second head assigns a conversation-level sensitivity score.

Perplexity's Hybrid Compute documentation lists three downloadable local models, Gemma 4 E4B, Qwen3.6 35B-A3B, and a Perplexity model, for the Mac-side half of that handoff.