DataSpace finds harnesses shift data-task accuracy by 15 points

• The fixed MiMo-V2.5 backbone scored 30.98% to 46.34% across five harnesses in DataSpace's controlled test, a 15.36-point range that the DataSpace benchmark report attributes to the surrounding agent system.
• The best of six multimodal models reached only 66.34% across the 410 tasks, according to the DataSpace benchmark report.
• Complete tables, rather than plausible prose, determine success: missing or extra rows and columns fail the task, as the DataSpace benchmark report notes, while the open evaluator normalizes equivalent headers and value formats.
• Harbor is formalizing the outer loop as versioned RL environments and test-set hillclimbing, a model that Harbor's operating model sets out explicitly.

The paper's fund-risk example combines a video-defined alert rule, PDF benchmark data, and daily NAV values in SQLite before it can emit an answer. Harbor's task model reduces an agent test to an instruction, container environment, and test script, while ThinkingBox grades the backend state left after the run.

DataSpace gives an agent a natural-language question and a self-contained workspace, then asks for one complete tabular result. The benchmark paper spans 410 financial, macroeconomic, and healthcare tasks, 7,439 artifacts, 15.01 GB of material, six file modalities, and Chinese-English variation across questions and artifacts.

The paper's fund-risk example breaks the job into six operations:

1. Discover candidate files.
2. Load structured data and query the database.
3. Extract records from a PDF and a rule from video.
4. Align fund identities, categories, and units.
5. Compute and filter derived metrics.
6. Rank the surviving records and export the requested table.

The harness experiment fixed MiMo-V2.5 and compared DataSpace-Agent, Smolagents, Codex, Claude Code, and Grok Build. Grok Build reached 46.34%, while Smolagents reached 30.98%, according to the DataSpace benchmark report.

The repository calls this a unified harness comparison and supplies a shared offline Data Workbench Runtime for local execution across its adapters in the baseline release. The result is a system-level comparison: prompts, tools, context handling, control flow, and output materialization travel with each harness.

Joins and mixed data types reduced accuracy for every tested backbone, the DataSpace benchmark report found. A correct final table requires more than locating the right evidence.

The study flags three distinct breakdowns:

• Cross-source reconciliation can fail while joining database records, files, documents, and media.
• Agents can derive the right information, then misread the requested result.
• Agents can submit the wrong columns, or an incomplete table, at the final materialization step.

DataSpace evaluates the table with type- and precision-aware comparison, accepts header and column-order variants, and rejects missing or surplus rows and columns under its frozen task configuration. That makes output assembly part of the measured capability rather than presentation cleanup.

ThinkingBox applies the same standard to stateful business work. Its research paper places an agent in isolated MCP-compatible tool sessions and grades the final backend state across 507 policy-conditioned workflows.

The strongest model passed 65.36% on a single attempt but only 25.25% across 20 attempts, the ThinkingBox benchmark report reports. Four out of five failed runs ended politely and made a state-changing tool call, despite leaving the backend in the wrong state.

Harbor packages the environment around an agent as a task with an instruction, a container, and a verifier or test script. Its core documentation treats datasets as collections of those tasks, while isolated runs make code edits, dependency installation, and builds testable without sharing a prior attempt's container.

Harbor's stated premise is that RL environments should be maintained and versioned like software packages, while agents improve through systematic hillclimbing against a test set, as Harbor's operating model puts it. Its tooling list names environment management, judge tuning, large-scale simulation, and result auditing, reproduction, and optimization.

A Harbor hiring post describes the intended end state as billions of RL environments. Separately, Harbor's rollout estimate projected more than 1 billion rollouts and more than $10 billion in token spend this year.

Agent Lightning v1.0 moves RL training into an existing deployment harness by recording its LLM request-response pairs at the serving boundary, rather than rebuilding the harness loop inside the trainer. The Agent Lightning paper reports that 6,000 training examples raised Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%.

ClawGym II extends that approach to black-box harnesses: it runs OpenClaw or Claude Code unchanged in sandboxes, intercepts model calls, and reconstructs prefix-tree trajectories for PPO or GRPO. Its paper reports ClawGym-Bench Pass@1 gains of 9.98 points through OpenClaw and 14.81 points through Claude Code with Qwen3-30A3B.

The MIT-licensed DataSpace repository releases every task input and workspace in a versioned archive, with a checksum for the 15 GB download. It ships gold tables and frozen evaluation configurations for 60 representative tasks; references for the other 350 remain withheld for official full-benchmark evaluation.

Each local prediction is a task-scoped prediction.csv. The published evaluator uses only the Python standard library and emits a task-level summary, so the public subset can reproduce the complete-table contract without access to the withheld references.