Quail open-sources MIT-licensed AI-SQL engine for LLM queries

• Quail is now an open-source, MIT-licensed execution engine for AI-SQL filters and joins, according to sh_reya's release note; sh_reya's launch post reports more than 1 billion input tokens per minute for one query on one H100.
• A natural-language AI filter adds an LLM evaluation per row, while an AI join can add one per candidate pair, as sh_reya's SQL example explains.
• Quail targets queries whose requests are known ahead of time and are prefill-only, a workload sh_reya's workload description contrasts with latency-oriented interactive serving.
• The authors report a 1.84× average speedup over hand-tuned vLLM across 29 QUAIL-B queries, with a 14× result on the largest medical-report query in sh_reya's benchmark result.

The technical post works through a query combining 5,000 medical reports with 4,144 reaction terms, using the term set twice. The GitHub repository carries the MIT license, and the 0.1.0 PyPI package requires Python 3.12 and pins vLLM 0.26.0.

AI-SQL is an LLM call expressed as a SQL function, not a model generating SQL. The cardinality warning in sh_reya's SQL example is direct: filters make one model call per row, while joins can make calls across every candidate pair.

That turns filter order, join order, and the document placed first in a prompt into inference-scheduling decisions. The query plan determines how much model work survives to the next operator.

The system takes advantage of a constraint conventional serving engines do not have: the full query shape is visible before execution. As Full Stack Data Lab's technical post describes it, Quail moves from logical SQL to a physical AI-operator plan through four stages:

1. Estimate row counts and document lengths.
2. Set forward-pass token and KV limits from the selected model and GPU.
3. Push down projections and filters, reorder predicates, and select join order and anchors.
4. Lower AI operations into physical operators with an execution order and KV policy.

An optional selectivity estimate lets Quail reorder filters. An optional join anchor controls which input appears first in a prompt, preserving the prefix most likely to be reused.

In sh_reya's follow-up, she said the known shape of a SQL join can justify implementing attention differently, and suggested semantic groupby may expose similar headroom.

Quail streams batches between operators rather than materializing a full intermediate dataset. It can keep a report's KV state in HBM through downstream joins, as the technical post describes for its BIO-4 workload.

Its execution path has three pieces:

• A CPU physical-plan executor prepares document batches.
• A KV manager pins, rewinds, retains, and releases cache pages according to the plan.
• A GPU inference program runs the model forward pass for each batch.

After a filter answers true or false, the KV manager rewinds to the end of the document prefix and drops the predicate-specific suffix. It retains that document state only when a later operator needs it, and evicts shorter documents first when capacity runs out because longer inputs cost more to recompute.

At million- or billion-request scale, sh_reya's reply says even materializing every prompt string would be wasteful. The join path also uses tree-attention techniques related to SpecInfer and Hydragen to reuse forward-pass work when one document is compared against many others.

The reported average covers 29 queries in QUAIL-B, a benchmark the team says it is still building. The largest BIO-4 workload uses Qwen3 4B FP8 on one H100; its technical post measures a 14.91-minute optimistic roofline lower bound, 6.84 hours for vLLM 0.26.0, and 29 minutes for Quail.

The vLLM baseline processed 174.6 million tokens rather than 124.3 million because it discarded cache that was later needed, a 50.3 million-token excess that sh_reya's baseline report attributes to KV regret alongside host scheduling overhead.

The speedup is therefore a result for a tightly bounded, true-or-false prefill workload with related prompts, not a generic model-serving comparison.

The current release supports AI filters, AI joins, relational projections, and LIMIT. The team documents both Snowflake-style AI_FILTER and BigQuery-style AI.IF, plus a Python query builder, in its technical post.

Inputs are in-memory Arrow tables or datasets. The listed model and hardware scope is Qwen3 4B FP8, Qwen3 32B FP8, and DiffusionGemma on H100 GPUs; the project says its extension interface is intended to add operators, execution backends, models, and hardware.

vLLM wins on at least one agent-trace analysis workload, where rows share matching prefixes. Its automatic prefix cache made it 2.32× faster than Quail there, according to sh_reya's prefix-cache caveat.

Quail does not yet recognize those cross-row matches automatically. The team lists automatic prefix caching on its roadmap.