Kaigani builds video-QC benchmark after Gemini and Codex frame tests

• kaigani's benchmark post says he is building a generated-video QC benchmark by deliberately glitching clips.
• In kaigani's Codex comparison, Codex found a woman's anatomy and placement error that Claude and Gemini missed, while his follow-up says the winning method used contact sheets and selected frame analysis.
• kaigani's Gemini note says Gemini can find issues with multiple passes when attention is focused on specific defects, and his subtler test says Gemini caught another hard-to-see error.
• DavidmComfort's Seedance experiment shows the adjacent creator problem: modern AI video shots now combine dialogue, sound effects, blocking, camera movement, and object handoffs in one 15-second render.

The screenshot in kaigani's Codex comparison names four defect buckets: body geometry, vehicle continuity, malformed plate text, and road/reflection continuity. Google's Gemini video-understanding docs describe video input as a first-class prompt surface, but kaigani's Codex follow-up is a reminder that frame-based inspection can still beat broad video reasoning on a single physical error. DavidmComfort's blocking test found a different failure mode: Seedance executed one character cross and silently dropped the second mover.

Kaigani moved from one-off model shootouts to a benchmark: deliberately glitched generated clips for video QC.

The target is post-render defect detection. His nearby tests include a subtle synthetic character issue that Gemini caught and a clip labeled QC TEST 01 that he called a good QC test because the defect was not obvious even to people.

Kaigani first framed the surprising result as Codex beating Claude and Gemini on an anatomy and placement error visible to humans.

His follow-up narrowed the method:

• Model: GPT 5.5 inside Codex, according to kaigani.
• Input: a contact sheet of multiple frames.
• Drilldown: specific frames after the contact-sheet pass.
• Strength: still-frame anatomy, placement, and geometry.
• Blind spot: no audio understanding and no direct motion understanding beyond frame extrapolation.

OpenAI's image input guide describes the underlying kind of image analysis workflow. Kaigani's test used that frame-first route against a video artifact.

The error inventory inside kaigani's Codex screenshot is useful because it reads like a starter label set for generated-video QC:

1. Strong generative visual artifacts: the woman's body position was physically implausible, with arms intersecting the hood and window area.
2. Vehicle continuity and geometry errors: grille, headlights, hood ornament, antenna, roof rack, panels, and windshield relationships shifted.
3. Malformed readable text: the license plate appeared warped and nonsensical.
4. Road and reflection continuity issues: lane markings and road geometry appeared to slide across the hood.

Those are concrete failure classes, not generic bad-video vibes.

Kaigani said Gemini can catch issues if it runs multiple passes and each pass focuses attention on specific issue types.

That puts prompt design inside the evaluation harness. The model is aimed at one defect family at a time, then asked again for the next one.

DavidmComfort's two-day Seedance 2.0 experiment shows why video QC is becoming more film-grammar aware. The test combined three characters, one continuous shot, lip-synced dialogue, a moving camera, and a physical handoff.

His audio finding was mechanical: Seed Audio had to run twice, once for dialogue and once for sound effects, because reference voices could wipe out sound effects, music, or other audio layers.

His staging rule was also mechanical: the 180° rule governed cuts, while continuous camera movement worked when the prompt described the travel instead of teleporting to a new angle.

With three characters, he treated the active axis as the line between the current speakers. When dialogue handed off to a different pair, the axis rotated with the conversation.

DavidmComfort prompted a slow arc from two kids to a third character and then ran scene-cut detection on the output. He reported zero cuts, with the shot moving from wide to two-shot to close-up and arriving on the final line.

The caveat was over-delivery: asking for less than 30 degrees produced an arc plus a push.

His harder limit was blocking. A shot with one character crossing to a table and another countering a step executed the cross and dropped the counter.

The prompt evidence shows the discarded clause directly: Liora crosses two steps to the table, Tobin counters half a step, and the note says the Tobin counter was silently dropped.

The handoff test used one decisive motion, then a lock clause: the vial goes into the palm, fingers close over it, and both hands hold steady.

The rendered prompt also pinned the object as a single rigid vial and kept the camera locked off in a static close shot.

Coverage had its own failure mode. DavidmComfort said a fresh text-to-image close-up produced wrong faces, the wrong room, and a missing third character.

His derived close-up instead reframed the exact master wide via image-to-image, kept identical faces and lighting, and placed Tobin just off frame-left so the model would not invent another face.

DavidmComfort put hard costs on the two-day investigation: about $4.50 for a 15-second 720p video shot with two or three people, about $0.07 for a staged still, and about $0.10 for conversation audio.

The full set of experiments and retries cost about $65, according to his breakdown.