Skyfall AI launches Morpheus benchmark for persistent-world agents

• Morpheus is a persistent continual-RL benchmark, with the launch post describing simulated worlds where state does not reset, objectives drift, and decisions compound.
• Skyfall’s initial claim is that frontier LLMs can look stable while failing to learn online, with the model-summary post citing GPT-5.5 and Gemini 3.1 Pro as examples.
• The benchmark targets enterprise workflows, especially resource allocation and scheduling under drift, which the graph note framed as stable Task 1 performance and volatile Task 2 behavior.
• Access is spread across the website, blog, Hugging Face surface, and environment hub, with the resource post collecting the project links.

Skyfall’s results post has the bookmarkable bit: Task 1 can look steady while Task 2 leaves 93% to 98% of available reward unrealized. The Morpheus site says the benchmark removes resets, task IDs, and fixed curricula, then adds controllable drift sweeps. The environment hub exists, but Exa’s capture only showed a sign-in/search shell and “Loading environments...” rather than a browsable catalog.

Morpheus is Skyfall AI’s benchmark suite for continual reinforcement learning in “big, evolving enterprise worlds,” according to the project site. Its core design is a useful correction to leaderboard comfort food: agents are evaluated inside systems that keep moving.

The mechanics are concrete:

• World state persists across time.
• Actions compound and shape future dynamics.
• Objectives shift through structured configuration changes.
• Consequences can arrive several simulated days after the action.
• No explicit task boundaries, task IDs, resets, or predefined curriculum are provided.
• Interventions are parameterized so drift, assumption shifts, and regime transitions can be reproduced.

Skyfall’s blog post says the initial evaluation used two tasks across two warehouse environments, producing four experimental conditions.

• Task 1 Outbound: allocate fulfillment capacity across order categories under a capacity constraint; reward is tied to On-Time In-Full outcomes observed days later.
• Task 1 Inbound: allocate receiving resources across shipment categories under dock and labor constraints; decisions propagate through quality control and putaway before reward appears.
• Task 2 Outbound: schedule outbound jobs across dispatch lanes; configuration shifts can occur between dispatch decisions and delivery outcomes.
• Task 2 Inbound: schedule incoming shipments and putaway jobs across receiving lanes with regime-dependent arrivals, quality control, and inventory reconciliation.

The configuration intervals were a normal baseline, an EDI failure spike that degraded supplier and carrier data, and a WMS capacity drop that sharply reduced warehouse-management capacity.

Skyfall prompted both models with the full world state and operation logs at each decision step, used them off the shelf with no fine-tuning, and repeated experiments across five random seeds, according to the results post.

Task 1 looked strong on raw allocation reward:

• Gemini 3.1 Pro: 0.899, 0.853, 0.852 across the three intervals, 0.864 overall.
• GPT-5.5: 0.920, 0.920, 0.914 across the three intervals, 0.918 overall.
• Adaptation speed: 1.0 for both models.

Skyfall interpreted that flatness as pretraining coverage rather than online adaptation. Task 2 exposed the sharper failure: Gemini outbound reward fell 21% across two configuration shifts, Gemini inbound reward collapsed 95%, and GPT-5.5 inbound repeatedly hit zero with no stable recovery pattern.

The RL contrast is brutal. PPO closed 99.3% of the gap to the theoretical upper bound on Task 1 and 99.96% on Task 2 from random initialization, while Skyfall says LLMs left 93% to 98% of available reward unrealized on Task 2.

Skyfall’s allocation-shape analysis is the cleanest evidence that the models were applying different priors instead of converging through reward optimization.

• GPT-5.5 OUT spread allocation across 2.11 queues on average, with a maximum capacity share of 0.74.
• Gemini OUT concentrated on 1.34 queues on average, with a maximum capacity share of 0.91.
• Both models allocated to the highest-value queue at the same 0.36 hit rate.
• In the WMS capacity-drop interval, Gemini OUT left a 0.148 gap to the theoretical upper bound, while GPT-5.5 OUT left a 0.086 gap.

Skyfall’s read was that GPT-5.5 had a diversification heuristic and Gemini had a prioritization heuristic. Both sounded operationally plausible, but neither closed the reward gap over the run.

A reply asked the hard measurement question: do models detect the shift before their actions change?

Skyfall’s answer in the blog post is that external LLM failures collapse multiple causes into the same trace. Context overflow, pretraining gaps, and reward misalignment can all produce similar reward drops because the weights never update.

The metric suite is more interpretable for RL agents:

• Adaptation speed measures how long recovery took.
• Performance gap measures recovered-policy quality.
• Stability measures oscillation.
• Recovery time measures when behavior stabilized.

For fixed-weight LLMs, Skyfall says forgetting and recovery are formally computable but mechanistically weak labels. GPT-5.5 inbound showed detection lag greater than 25 steps at the second configuration shift and never recovered inside the available window.

The Morpheus site says the first environments are outbound warehouse management and inbound warehouse management. The public roadmap lists four next modules:

• EDI Invoice Processing.
• ERP Order to Cash.
• ERP Procure to Pay.
• Manufacturing Production Planning.

Morpheus is described as framework-agnostic and intended to work with PyTorch-based and TensorFlow-based RL workflows. The FAQ says selected environments and tooling are planned for distribution through Hugging Face and GitHub, while waitlist access provides early updates and design-partner feedback channels.