Enterprise Worlds opens ITSMBench with 93 tools for enterprise agent evals

• Enterprise Worlds is now open source, with ITSMBench as its first executable benchmark for enterprise-agent workflows, according to Shahules786's launch thread.
• ITSMBench puts agents inside persistent enterprise state with simulated users, policy-governed workflows, 93 typed tools, and deterministic final-state evaluation, as Shahules786's tool list describes.
• Success means leaving the ITSM environment in the right state across users, tickets, services, assets, approvals, incidents, changes, and offboarding, not producing a plausible transcript, Shahules786's workflow note says.
• Frontier models still struggle on policy-following, ambiguity resolution, and maintaining state across the workflow, according to Shahules786's results note.
• The failure cases are painfully enterprise-shaped: SLA rows left stale, invented audit timestamps, skipped questions, and customer emails sent before the exact message was approved, in Shahules786's failure examples.

The GitHub README ships runnable Python code, CLI commands, and an Apache-2.0 license. The ITSMBench leaderboard splits coding-style pass@k from enterprise-style pass^k, which makes flaky success look much less impressive. The repo credits ServiceNow's EnterpriseOps-Gym for seed databases and the typed tool surface, then adds a multi-turn user simulator and state-plus-NL verifier in the style of τ-bench.

ITSMBench's core shape is a live IT service-management world, not a static prompt set.

The benchmark surface includes:

• persistent enterprise state
• simulated users
• policy-governed workflows
• 93 typed tools
• deterministic final-state evaluation

The Enterprise Worlds README expands the tool layer: lookup, mutation, and aggregation calls enforce enum gates, required fields, referential integrity, and deterministic IDs and timestamps.

The released world has 53 tasks. They cover:

• users
• tickets
• services
• assets
• approvals
• access records
• incidents
• changes
• offboarding

The repo describes two tenant setups: a single-tenant company and a provider world serving 20 client organizations, with roughly twenty interconnected tables across incidents, SLA clocks, problems, change requests, CMDB records, services, knowledge articles, notifications, users, groups, roles, and permissions.

ITSMBench runs each model on 53 tasks with 4 independent trials per task, according to the public leaderboard. A trial passes only when the final database state matches the expected final state.

The two headline metrics separate lucky success from repeatable execution:

• pass@k: at least one of k attempts solves the task.
• pass^k: all k attempts solve the task.

The first leaderboard makes the gap obvious:

• Claude Opus 4.8 medium adaptive: pass^1 0.698, pass^4 0.396, pass@4 0.887.
• GLM-5p2: pass^1 0.547, pass^4 0.208, pass@4 0.887.
• GPT-5.6 Luna: pass^1 0.505, pass^4 0.226, pass@4 0.792.
• Qwen3.7-plus: pass^1 0.476, pass^4 0.189, pass@4 0.736.

The leaderboard notes that every model loses more than a third of its pass^1 score by k=4. It also says Muse Spark 1.1 and Nemotron-3 Ultra NVFP4 have published figures but no released trajectories.

The best part of the release is the failure texture. The errors look less like broken tool-calling and more like unfinished office work.

Shahules786 listed four recurring patterns:

• reporting an SLA pause without updating the SLA row
• inventing audit timestamps
• skipping required questions
• sending customer emails before getting the exact message

The trajectory viewer exposes full agent-operator conversations, tool calls, tool results, and the exact database fields that diverged from the expected final state for captured runs.

Enterprise Worlds is Apache-2.0, and the first release builds on ServiceNow Research's EnterpriseOps-Gym seed data and action space.

The EnterpriseOps-Gym repository describes its upstream benchmark as 1,150 expert-curated tasks across 8 domains, 512 tools, and 164 database tables. Enterprise Worlds reimplements the ITSM substrate as in-memory Python tool calls, so the README says it does not require Docker or a SQL server at runtime.

The runnable CLI is small:

The same README names the original additions: a multi-turn conversational user-simulator loop, an org-scoped multi-tenant task model, and a verifier that combines database-state comparison with natural-language assertions.