Goodside benchmarks GPT-5.6 Sol Pro on 150- and 1,025-Pokémon crossword tests

• Claude Fable 5 Max generated the 150-name source puzzle as a 47x50, ~5000px grid with 75 across entries, 75 down entries, and 175 crossings, according to Goodside's Fable generation post.
• ChatGPT 5.6 Sol Pro solved the blank 150-Pokémon version in 16m 21s, and Goodside's solve thread says the clue-free grid admitted 32 valid completions.
• The 1,025-Pokémon version broke the loop: Goodside's five-attempt test recorded partial fills of 45, 145, and 40 answers, plus two timeout or error runs around 30 minutes.
• One claimed 1,025-name success collapsed into retrieval: the model found Goodside's public answer key and linked the thread in its response, per his answer-key contamination post.

Tom's Guide covered the smaller no-clue solve before Goodside scaled the test to all 1,025 Pokémon. The odd details are the useful ones: no Pokémon name length is unique, the smaller blank grid had 32 valid fills, and one apparent win came from finding the answer key online. Goodside also tucked a recursive pelican GIF under his 1,025-Pokémon post, then answered another reply with a Lem aside.

Goodside first used Claude Fable 5 Max as the puzzle maker, not the solver. The prompt asked for a massive solved crossword PNG using the first 150 Pokémon, with clues omitted.

The generated artifact came with its own verifier pass:

• 47x50 grid
• roughly 5000px render
• 150 answers, Bulbasaur through Mewtwo
• 75 across entries and 75 down entries
• 175 crossings
• every horizontal and vertical run checked as exactly one of the 150 names

That makes the setup unusually clean for a creator benchmark: one model builds the visual puzzle, another model gets the blank geometry and the word universe.

Sol Pro solved the empty version of the 150-name grid without numbered clues. Goodside's screenshot shows a 16m 21s run ending in a completed crossword download.

The answer was not identical to Fable's key. It was still valid, because the clue-free grid had 32 completions.

Goodside's thread listed the ambiguity pockets:

• GOLDEEN / MOLTRES at 117 Across and 20 Down
• MACHOKE / STARMIE at 143 Across and 104 Down
• eight equivalent arrangements of NIDORAN♀, NIDORAN♂, NIDORINA, and NIDOKING across four slots

Recognizing those 32 valid completions without being prompted took a real agentic solve, Goodside argued in one reply.

A blank crossword is not empty information. The grid still gives slot lengths, intersections, and letter-agreement constraints, which a Columbia writeup on crossword CSPs describes as a classic constraint satisfaction setup.

Goodside's version removed semantic clues and left the model with geometry plus a closed word list. One Goodside reply noted that even length alone is weak here: there is no length for which only one Pokémon exists, and length 3 still leaves Muk or Mew.

The perception problem is separate from the fill problem. Goodside said the image contained only empty boxes, so the model needed to recover the grid layout, likely through a programmatic route rather than by reading letters.

A recent CrossWordBench paper frames crossword puzzles as a benchmark for LLMs and vision-language models because they combine text constraints with visual grid constraints. Goodside's test stripped out the text clues and made the geometry do almost all the work.

Goodside could not tell from Sol's reasoning summary whether it used code to read the layout from the image. He later separated that from the actual fill step: Sol was obviously using code to solve, while the open question was how it extracted the grid.

The solve did not look like a one-shot answer. Goodside described it as an agentic loop trying partial and iterative solutions, with a visible reasoning summary in the shared chat.

He had not tried smaller models at first, though a later reply said GPT-5.4 high seemed to solve the 150-name version too.

Goodside then scaled the same format to all 1,025 Pokémon and posted both solved and unsolved versions. In a reply, he said Claude's UI did not show the exact execution time, but Fable took somewhere between 5 and 10 minutes.

The 1,025-name prompt normalized the names before solving:

• punctuation, spaces, and hyphens dropped, as in MRMIME, FARFETCHD, HOOH, TYPENULL, IRONVALIANT
• accents stripped, as in FLABEBE
• Nidoran♀ and Nidoran♂ written as NIDORANF and NIDORANM
• PORYGON2 kept its digit

Sol Pro then got five attempts with memory off. The results were partial at best:

1. 45 answers in 28 minutes
2. 145 answers in 33 minutes
3. 40 answers in 31 minutes
4. timeout or error after about 30 minutes
5. timeout or error after about 30 minutes

Goodside's read was blunt: Sol understood that it needed a solver, but his follow-up reply said it messed something up in practice.

A reply later claimed Sol could solve the 1,025-name grid. Goodside inspected the shared chat and found a simpler explanation: Sol had located the exact solved counterpart from Goodside's own thread.

The model even linked to the thread in its response, according to Goodside's earlier reply. That turns the big-grid test into a neat benchmark hygiene note: once the solved image is public, a web-enabled run can look competent by retrieval rather than by solving.