Modal launches globally available RDMA clusters

• Multi-node RDMA clusters are globally available behind @modal.clustered, according to Modal's announcement.
• VM Sandboxes give agents a full Linux kernel for Docker, FUSE and kernel-dependent workloads, as Modal explains.
• Sandbox Sidecars separate trusted harnesses and credential proxies from agent-generated code on the same host, per Modal's Sidecar announcement.
• Sticky Sessions route reconnecting WebSockets back to their original container, according to Modal's post.

Modal added RDMA support to gVisor and upstreamed it, a neat bit of plumbing for multi-tenant GPU clusters. The Sidecar docs contain a compatibility wrinkle: Sidecars currently cannot run with VM Sandboxes. Browser WebSockets need an HTTP-to-cookie bootstrap before upgrading.

Clusters reached general availability in every Modal workspace on October 1, after 18 months of testing. Modal promises acquisition in seconds and per-second billing in its launch post.

The launch example requests four eight-GPU B300 nodes, 32 GPUs total:

Modal documents the allocation and execution rules in its cluster guide:

• Bandwidth: B300 nodes have 6.4 Tbps RDMA networking; other GPU types have 3.2 Tbps.
• Placement: Nodes within a cluster share one RDMA fabric in one availability zone. Global availability does not imply a cross-region cluster.
• Allocation: Each container must request all GPUs on its host. CPU-only clusters are unsupported; the guide lists up to 256 devices, with actual size bounded by workspace GPU limits.
• Image dependencies: RDMA requires libraries such as libcudart.so, libibverbs.so.1 and libmlx5.so.1. Modal configures NCCL environment variables automatically.
• Execution: Every container receives the same function arguments. Only rank zero's output returns to the caller; Server traffic also goes to rank zero, which distributes work to peers.
• Coordination: Modal does not synchronize input execution across containers. The leader must wait for peers before advancing to the next input.

Existing Volumes, Cloud Bucket Mounts and Queues remain compatible. Modal's gVisor integration proxies RDMA setup calls to the host kernel; the data path moves GPU memory through NICs without staging it in host RAM.

Modal built a fleet-wide observe-plan-act scheduler for multi-node workloads, replacing per-node placement decisions with coordinated allocation. Its scheduler walkthrough lists four steps per iteration:

1. Collect metadata for all multi-node workloads.
2. Assign workloads to nodes, grouped by availability zone and network ID.
3. Provision capacity for workloads that do not fit.
4. Notify all live nodes of their assignments.

Code starts only after all requested hardware has been acquired. Under the cluster failure rules, an input failure on any node terminates its peers and fails the call; preemption terminates and retries the entire cluster with the same input.

VM Sandboxes are generally available through modal.Sandbox.create(..., runtime="vm"). Modal built the runtime on Rust-based Cloud Hypervisor, according to its VM announcement.

• API compatibility: Existing modal.Image definitions, exec and filesystem APIs carry over, with the same usage-based pricing.
• Startup and resources: Modal claims sub-second cold starts and retains CPU and memory bursting.
• Default runtime: gVisor remains the default; VM Sandboxes add the full-kernel environment.

Modal reports that early customers have already launched more than 20 million VMs. The announcement details three deployments:

• Linear: Every Coding Session runs in a VM Sandbox. Linux cgroups and network namespaces partition the coding agent, development server and Linear's host process.
• Legora: Legal-agent evaluations build the full application inside each Sandbox, including Postgres and a DOCX editor. Native Docker support eliminated networking and FUSE workarounds.
• Snorkel: Millions of monthly agent simulations use Harbor to exercise multi-service tasks, including database migrations and debugging database-backed applications.

Sidecars launch in beta as separately isolated containers on the same host as the main Sandbox. Trusted credential injection, proxies and harness logic can run outside the agent's execution environment, according to Modal's Sidecar announcement.

Modal's internal benchmark compares local Sidecar communication with HTTPS calls between separate Sandboxes in the same region:

| Measurement | Sidecar | Separate Sandbox over HTTPS |
| --- | ---: | ---: |
| Per-call travel, p50 | 0.58 ms | 2.6 ms |
| Per-call travel, p99 | 1.4 ms | 44 ms |
| 100 calls, total time | 0.6 s | 1.2 s |

• Control: The SDK creates Sidecars dynamically. Code inside the main Sandbox cannot create or modify them.
• Networking: Containers communicate by name over an internal TCP/UDP bridge. Each Sidecar has its own outbound network policy.
• Resources: Sidecars share the parent Sandbox's CPU and memory reservation. The hard ceiling is 250 concurrent Sidecars, with smaller reservations imposing lower limits.
• Lifecycle: Sidecars can be terminated and replaced independently. Terminating the parent Sandbox stops all its Sidecars.

The launch post describes isolation through gVisor or VMs, but the Sidecar limitations explicitly list VM Sandboxes as incompatible. Sidecars also require pre-built images, lack Cloud Bucket Mount support and support filesystem snapshots without capturing memory state.

The experimental proxy_traffic_via_sidecar option routes outbound TCP traffic on port 443 through a named Sidecar. Modal's proxy documentation specifies several boundaries:

• Startup: HTTPS traffic is refused until the designated Sidecar is running.
• Destination discovery: The proxy receives a raw TLS stream and reads the hostname from ClientHello SNI. The original destination IP is not forwarded.
• HTTP inspection: Reading or rewriting requests requires TLS termination with a certificate authority trusted by the Sandbox.
• Egress controls: Relayed traffic bypasses the main Sandbox's other egress controls, including outbound_cidr_allowlist. Non-relayed traffic remains subject to those controls.
• Configuration: The option cannot be combined with block_network or outbound_domain_allowlist.

Sticky Sessions preserve container affinity across requests and WebSocket reconnects. They are enabled with @modal.sessioned() on an @app.server() class.

Modal's session guide defines the protocol:

1. Call Server.sessions.start() to allocate a session and obtain its ID and token.
2. Send Modal-Authorization: Bearer <token> with HTTP requests or WebSocket handshakes.
3. For browser WebSockets, exchange a modal_session_token query parameter for a session cookie through an HTTP request before upgrading.
4. End the session explicitly, or let its idle timeout expire. The default is 600 seconds without an in-flight request.

An active session keeps its container alive up to the container's maximum runtime. Ordinary Modal-Session-ID affinity remains a best-effort routing hint; it does not provide that lifetime guarantee.

Autoscaling counts active sessions rather than requests. A container defaults to, and is capped at, 1,000 sessions; session creation can queue for capacity for up to 60 seconds.

Adding @modal.sessioned() to an existing Server causes requests without a session token to be rejected, even when unauthenticated=True.

Modal also announced Endpoint Candidates in its Runtime roundup. The feature mirrors production traffic to alternative inference-serving recipes for comparisons of quality, latency, throughput and cost.

Multiple candidates can be staged before promotion to production. Endpoint Candidates are in private beta for selected customers; Modal says general availability will follow, without giving a date.