CodeClimb checkpoints coding experiments through API-credit pauses of up to three days
CodeClimb’s builder describes Inngest checkpoints that preserve coding experiments through pauses of up to three days. Separate coding and evaluation sandboxes use scores and reviewer feedback to accept or revert changes.

TL;DR
- CodeClimb can pause for up to three days after API credits run out, without keeping sandboxes running, according to its builder’s implementation report.
- The first reported GraphRAG night cost $10.44 across five experiments, according to the run report.
- The reviewer remains the weak link: the builder says it can reject better implementations while interpreting multiple metrics.
Inngest’s sandbox reference draws an easily missed boundary: memoized API results do not preserve files or processes after a sandbox ends. CodeClimb even sets aside money for the final test before funding another experiment.
Completed-step checkpoints
Paul Iusztin, CodeClimb’s builder, says the system saves code commits and scores outside its sandboxes. A credit-starved workflow can wait up to three days, then resume after a top-up without keeping sandboxes running during the pause.
Inngest’s step documentation specifies the recovery boundary: completed step outputs are saved and reused when later work fails. Its step.waitForEvent() API supports suspending execution until a matching event arrives or a timeout expires.
Iusztin also warns that feeding a credit-related evaluation failure to the reviewer as a bad score can cause it to reject good code.
Coding and evaluation sandboxes
Pi runs on Iusztin’s local worker, while Inngest supplies durable orchestration and cloud sandboxes. The architecture separates coding from evaluation:
- The coder never sees development or test answers.
- Test errors stay hidden, and the test split is scored once after optimization ends.
- Each experiment receives a scoped API token rather than the builder’s real provider key.
- A proxy holds credentials, restricts allowed models, and tracks spending.
- CLI and MCP tools expose traces with evaluation scores, costs, and timings.
CodeClimb’s experiment, development-evaluation, and final-test paths
Experiment budgets
Iusztin’s reported limits bound both spending and execution:
- $2.50 per experiment.
- $15 per night.
- Six rounds maximum.
- Six-hour deadline.
The metering proxy rejects excess requests with HTTP 402. Additional stopping mechanisms include:
- Plateau detection.
- Timeouts.
- An Inngest kill switch.
The first reported GraphRAG night consumed $10.44 across five experiments.
Keep-or-revert review
Each hypothesis passes through a five-step feedback loop:
- Pi implements a proposed solution.
- The implementation is evaluated against development data in a separate sandbox.
- A reviewer examines metrics, errors, code changes, and previous hypotheses.
- The reviewer decides whether to KEEP or REVERT the changes.
- It generates feedback for the next experiment.
Changes must pass acceptance guardrails and show meaningful development-set improvement. Extraction quality, graph precision and recall, and retrieval can move in opposite directions.
GraphRAG targets
The builder says CodeClimb has not yet reached its full GraphRAG target set:
- Extraction F1: 0.90.
- Relation F1: 0.40.
- Graph merge precision and recall: 0.98.
- Single-hop recall: 0.70.
- Multi-hop completeness: 0.30.
The follow-up’s eight-round extraction F1 chart, including kept and reverted changes and the final test score
Decision memory
CodeClimb already stores scores and a history list as decision memory. Nevertheless, experiment five repeated a failed choice from experiment two, Iusztin reported in his follow-up.