Perplexity releases WANDR benchmark with 500 deep-research tasks

• WANDR is now public: Perplexity released an internal wide-and-deep research benchmark with 500 tasks and 170,495 source-backed records, according to perplexity_ai's scale post.
• The task set comes from de-identified production use cases, with examples spanning competitive research, due diligence, literature review, market analysis, product comparison, and talent sourcing in perplexity_ai's task-source post.
• The grader is reference-free: perplexity_ai's grading post says WANDR re-fetches every cited page and checks each claim against the underlying evidence.
• Soft scores award partial credit, while hard scores require every component for a member to be complete and correct, per perplexity_ai's scoring post.
• Perplexity is treating WANDR as a training environment as well as an eval; denisyarats said the environments were synthesized from production traces and used internally to train models.

Perplexity's WANDR article includes a real ceo_cfo_appointments task: 70 U.S.-based companies, two evidence branches, 140 records. The GitHub repo splits the release into source data, a WANDR-to-Harbor adapter, Harbor task packages, and Relay. The strangest number is the ceiling: the strongest system in Perplexity's eval reached only 0.363 soft F1 and 0.133 hard F1.

WANDR targets jobs where an agent has to find a large set of qualifying entities, then verify structured facts about each one. Perplexity's official article defines the split as wide discovery plus deep per-entity evidence.

The public task set has:

• 500 tasks.
• 170,495 source-backed records.
• 167 lower-, 166 middle-, and 167 higher-difficulty examples.
• A median task size of 50 members, four records per member, and 245 records overall.

The hierarchy is the useful abstraction. A task can look like company(n) → employee(m) → url(k), where every complete path through the tree is independently checkable.

The scale chart in perplexity_ai's scale post shows why simple QA evals miss this shape: 30.8% of tasks require at least 100 members, and 44.4% require 100 to 299 records per task.

Perplexity says WANDR starts from de-identified production usage rather than synthetic prompts. The WANDR article describes a semi-automated pipeline that keeps human effort focused on quality control.

The pipeline has four stages:

1. Seeding: mine de-identified product requests for reusable wide-research patterns.
2. Authoring: run an author-critic loop to design the task, write fixtures, stress-test the judge, and patch conventions with a linter.
3. Admission: test feasibility by merging 10 to 12 authoring rollouts, audit the judge, and optionally add human sign-off.
4. Curation: label admitted tasks and select a balanced release across topics, scale, topology, and difficulty.

denisyarats framed the release as an eval and RL environment synthesized from production traces with weak human supervision. That makes the dataset more interesting than a static leaderboard.

WANDR does not grade against a fixed gold answer. Perplexity's article says each submitted record contains an item, URL, excerpts, and answer; the grader re-fetches the page and checks whether the evidence supports the claim.

The scoring stack is built for partial progress and diagnosis:

• Precision: quality of submitted records.
• Recall: quality-adjusted completion against the requested volume.
• F1: balance between precision and recall.
• Soft score: partial credit for incomplete members.
• Hard score: credit only when the full required subtree for a member is correct.
• Retrieval-only verdict: checks whether the fetched page satisfies the task before stricter validity and excerpt checks.

The benchmark is strongest as an error microscope. A soft-to-hard drop points to partially completed branches; high precision with low recall points to good records without enough coverage.

Perplexity's WANDR results compared six production systems across all 500 tasks under pinned configurations. Perplexity Search as Code led at 0.363 soft F1 and 0.133 hard F1.

The main reported scores:

• Perplexity Search as Code: 0.363 soft F1, 0.133 hard F1.
• Anthropic: 0.249 soft F1, 0.072 hard F1.
• Every other system: at most 0.121 soft F1 and 0.035 hard F1.

Perplexity's run cost $5.20 per task, with 14.9 minutes median solve time and 3.82 million reported tokens per task. OpenAI and Exa were faster and cheaper, while Anthropic came closest on quality but used more time, money, and tokens, according to Perplexity's article.

On a matched 45-task effort sweep, Perplexity reached 0.447 soft F1 and 0.224 hard F1 at xhigh. The same sweep put the cost range at $0.03 per task for Exa low and $324.83 for Gemini max.

The headline failure is completion, not web access. Perplexity's analysis says soft recall was below soft precision for every system, which means agents submitted plausible work but did not cover the full requested volume.

The hard-score collapse was severe:

• Perplexity fell from 0.363 soft F1 to 0.133 hard F1.
• Anthropic fell from 0.249 soft F1 to 0.072 hard F1.
• The best hard precision was 0.150.
• The best hard recall was 0.134.

Scale and hierarchy depth made the problem worse. From the smallest to largest target-volume bin, Perplexity's hard recall dropped from 0.219 to 0.079. From flat tasks to tasks with three or more intermediate keys, Perplexity's hard recall dropped from 0.378 to 0.017.

The record-level probes isolate the pain point. For five systems, only 3.2% to 8.9% of submitted pages were unusable, but 57.5% to 86.6% of submitted excerpts failed to support everything the record claimed.

The WANDR GitHub repo ships an Apache-2.0 project with Docker, Jinja, Python, and shell code. perplexity_ai's scoring post points to the public tasks and evaluation harness.

The repo is organized into four layers:

1. reference/wandr_tasks/: editable task configs, schemas, prompts, and artifacts.
2. adapters/wandr/: deterministic conversion from WANDR source data into Harbor task packages.
3. datasets/wandr/: generated, self-contained Harbor tasks with instruction, environment, evaluator, and scoring material.
4. agents/relay/: a generic Harbor agent that calls remote endpoints and materializes their output files in the workspace.

A run flows through Relay, task-local verification, page fetching, entity normalization, deduplication, judging, and Harbor reporting. The repo's quick start warns that full runs fan out across six providers and can be expensive; Harbor does not impose a spending cap.

The useful artifacts land under jobs/: provider-neutral Relay events, ATIF trajectories, reward files, WANDR metric rollups, verifier diagnostics, and an HTML report.

The release doubles as a training-data factory. perplexity_ai's WANDR thread says WANDR reveals whether an agent fails and where, while denisyarats said Perplexity has already used these RL environments internally to train capable models.

Perplexity's article says the same structure can produce denser rewards than a single terminal score: record-level and branch-level judgments, partial credit for discovery and enrichment, and held-out sibling tasks across domains and hierarchy shapes.

WANDR is already part of Perplexity's model-selection loop. AravSrinivas said Perplexity integrated Grok 4.5 inside Perplexity Computer within a few hours because it scored best on the company's evals, was the most cost-effective option, and had ZDR available from the start.

AravSrinivas also described Grok 4.5 inside the Computer harness as strong at work people actually do. Paired with WANDR's production-derived task construction, that is the practical claim behind the release: the eval is built to steer orchestrator choices, not just publish a benchmark table.