CMU Gym-Anything creates verified desktop-app agent environments

• CMU's Gym-Anything turns ordinary software into agent environments with a two-agent loop: one creator sets up the app, another auditor checks screenshots, logs, files, and checklists rohanpaul_ai's setup note.
• The project produced CUA-World, a benchmark with 10,000+ tasks across 200 applications and all 22 major occupation groups, according to rohanpaul_ai's paper summary.
• The app-selection pipeline starts from job and GDP data, maps that work to software tools, filters for sandboxable apps, then balances coverage across work areas rohanpaul_ai's selection note.
• The same evidence window included a Microsoft and CMU PowerPoint benchmark for creating and modifying slides, which gneubig's benchmark post framed as a rigorous computer-use evaluation.

The Gym-Anything paper is a benchmark-factory paper: use agents to build the environments that other agents will later be tested inside. rohanpaul_ai's setup note says the audit side checks screenshots and logs, not just whether an install script exited cleanly. gneubig's benchmark post points at the same pain from a narrower direction: PowerPoint editing is still a serious UI task for agents.

Gym-Anything moves environment construction into the agent loop.

The system splits the job into two roles:

• Creation agent: writes scripts, installs software, loads real data, opens the app, and collects proof that the environment works.
• Audit agent: checks screenshots, logs, files, and checklists, then sends feedback when the setup is not good enough.

That split is the useful bit. It treats environment setup as a verifiable artifact, not a manual precondition hidden outside the benchmark.

Using Gym-Anything, the authors built CUA-World with:

• 10,000+ tasks
• 200 applications
• Coverage across all 22 major occupation groups

rohanpaul_ai's paper summary says strong models solved only a small share of the hardest long tasks. Real app work is still where computer-use agents get humbled.

The benchmark does not start from a grab bag of convenient demo apps. rohanpaul_ai's selection note describes a selection process tied to economic work:

1. Start from real job and GDP data.
2. Map that work to thousands of software tools.
3. Filter for apps that can run inside a test sandbox.
4. Keep a balanced set across important work areas.

The result is a task pool aimed at software people actually use, rather than web-toy tasks that are cheap to host.

The pipeline has three layers:

1. Pick important real-world software.
2. Turn each app into an environment where an AI agent can act.
3. Create realistic tasks inside those apps.

rohanpaul_ai's pipeline figure note frames the payoff as less manual benchmark creation: one agent builds, another checks, then target agents are evaluated on longer software tasks.

The adjacent PowerPoint work narrows the same question to slide decks. gneubig's benchmark post says the Microsoft and CMU benchmark tests whether agents can make and modify PowerPoint slides with rigorous evaluation.

Shahules786's reply also put automated environment and task synthesis into the surrounding discussion.