eval-engineering
by langchain-ai2 months ago1.2kInspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.
Install
Show step-by-stepHide step-by-step
Open your terminal
- Mac: Press ⌘ Space, type "Terminal", press Enter
- Windows: Press Win R, type "cmd", press Enter
Paste the command above and press Enter
Use the Copy command button, then paste in your terminal (Mac: ⌘V, Windows: Ctrl V).
Restart Claude Code
Close and reopen Claude Code, or start a new session, so it picks up the new skill.
Where it lives
Comments
"this is a practical guide (+ updated skill) on how we use real world data like traces + human feedback to make synthetic environments + evals so we can..."
"there’s a limited amount of work per day where I can dedicate my time to being fully “in the loop” -> choosing to spend it on evals & environments bc imo..."
"Synthetic Environment Generation with Human Feedback Agents are poor 1-shot eval/environment generators because they're misaligned with human goals, successful..."
Related skills
dynamic-workflow
Plan-in-code fan-outs, adversarial verification, waves.
comfyui
Generate images, video, and audio via diffusion workflows.
hyperframes
Render MP4/WebM videos from HTML compositions.
claude-api
Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).