Zach Mueller explains retimer-based PCIe GPU expansion over MCIO

• Longer MCIO runs can need active signal conditioning: Mueller explained how retimers help preserve PCIe signal quality as cable length increases.
• A switched backplane can connect eight GPUs through one x16 host link, according to his topology breakdown, while host traffic still shares that uplink.
• Two early assumptions changed: Mueller warned that cascade routing could be slower and withdrew his ninth-GPU wiring idea after a distributor clarification.

C-Payne’s 100-lane switch defaults to five x16 downstream links behind one x16 host uplink; Miwin’s 12-slot Broadcom 89104 board lists 18 MCIO x8 connectors. The cabling is delightfully fussy: a manufacturer-derived diagram specifies crossing the 0-7 and 8-15 lane halves at the uplink/direct interfaces.

Zach Mueller separated MCIO connections into passive and active paths:

• Passive: carry the existing PCIe signal over the cable.
• Active: add retimers to restore signal quality along the connection.

In the x16 wiring he described, each MCIO cable carries eight lanes, so two cables carry the full link. Retimers sample, re-time and re-equalize PCIe 5.0 signals at 32 GT/s, according to HighPoint’s architecture guide.

Mueller also asked another builder about signal integrity and system-log errors on a longer connection without retimers. That was a question about the running system, not a reported failure.

Eight core GPU slots form two four-GPU groups, as Mueller explained in an early reply. He laid out two host-wiring options in his longer breakdown:

1. Direct connect: two host x16 links, carried over four MCIO x8 cables, feed the two switches separately.
2. Cascade: one host x16 link feeds SW1; SW1’s downlink feeds SW2 through another pair of MCIO x8 cables.

Each switch has 104 lanes, Mueller noted in a follow-up. Those downstream lane budgets coexist with narrower host connections, so eight GPU links do not create eight independent x16 paths back to the CPU.

Mueller revised two parts of the original plan:

1. Cascade performance: additional switch hops could make cascade slower, he said in a later update, with root-based routing likely faster.
2. Auxiliary slots: the distributor told him the direct-connected slots require host-connector feeds. Mueller corrected his proposal to attach a ninth GPU through SW2’s output while using only one host slot.

Keeping cascade plus a ninth GPU would require another direct connection and retimer, he added afterward.

ACS introduces a separate routing hazard: NVIDIA’s NCCL documentation describes PCIe peer traffic being redirected to the CPU root complex, causing performance reductions or hangs.

For a build without the backplane, Mueller outlined another arrangement:

• Switch: a PCIe 5.0 100-lane MCIO switch. C-Payne lists its Microchip Switchtec PM50100 board at €2,250.
• Retimers: active adapters along the cabled path. C-Payne lists its Gen5 x16 host retimer at €240.
• GPU-side MCIO hosts: adapters connecting the cabled links to GPU slots.

Mueller estimated roughly $4,000 for the backplane and $200 to $300 for a retimer in a follow-up. He sourced the backplane directly from Miwin through AliExpress.

Host-platform costs also came up: he asked another builder to price a configuration with about 128 GB of DDR5.

Mueller explicitly chose a mining-style frame. His later plan favored two full host slots in direct-connect mode and put the cards on two levels, as described in a revised layout:

• Five server cards: mounted directly on the backplane, with cooling for the closely spaced cards.
• Pro cards: connected through risers to an upper level with more spacing.

Mueller posted a shipping update in an earlier thread, then confirmed the board had not arrived yet. He planned to publish measurements on GitHub and listed four configurations for his nine-GPU collection:

• Each P2P switch group individually.
• Both switch groups combined.
• One switch group plus one to four direct connections.
• Both groups combined plus one direct connection.

He also left TP4/PP2 versus TP8 open in the original plan. Mueller mentioned Local Inference Lab in the discussion; the project publishes a PCIe topology reference.

Before the backplane arrived, he proposed AIPerf baselines comparing four Pro cards against four Max-Q cards at PCIe 5.0 x8, preferring stable NVFP4 quantizations. His proposed model list was:

• GLM 5.3 Flash.
• DeepSeek v4 Flash.
• Qwen Flash Next.

SGLang is the serving engine he uses, Mueller said in a reply.

Mueller described four storage cards running in RAID10, mirrored across cards in pairs, in a related exchange. He identified fast model storage as their role, clarifying that his Hyper M.2 reference concerned storage rather than inference offload.