eval-engineering
by langchain-ai3 months agoInspect an agent repository and optional traces, interview the user, write reviewed Task Specs, build and audit Harbor tasks, and bootstrap reusable project World Knowledge Skills. Use for agent evals, benchmark design, Task generation, controlled Environments, synthetic data, Verifiers, Harbor runs, calibration, or continuous benchmark maintenance.
Install
Show step-by-stepHide step-by-step
Open your terminal
- Mac: Press ⌘ Space, type "Terminal", press Enter
- Windows: Press Win R, type "cmd", press Enter
Paste the command above and press Enter
Use the Copy command button, then paste in your terminal (Mac: ⌘V, Windows: Ctrl V).
Restart Claude Code
Close and reopen Claude Code, or start a new session, so it picks up the new skill.
Where it lives
Comments
"this is a practical guide (+ updated skill) on how we use real world data like traces + human feedback to make synthetic environments + evals so we can..."
"there’s a limited amount of work per day where I can dedicate my time to being fully “in the loop” -> choosing to spend it on evals & environments bc imo..."
"Synthetic Environment Generation with Human Feedback Agents are poor 1-shot eval/environment generators because they're misaligned with human goals, successful..."
Related skills
hyperframes
Mandatory entry point: read this first for any request to make, create, edit, animate, or render a video, animation, or motion graphic, including a promo, explainer, captioned clip, title card, overlay, slideshow or interactive deck, Remotion port, or any HyperFrames HTML composition. Also use it to inspect, diagnose, validate, preview, publish, or batch-render an existing HyperFrames project. Inputs may be a website URL, GitHub PR, Figma design or URL, text or brief, existing footage, or music. It resumes project state, captures intent when applicable, selects and installs the owning workflow, and routes domain capabilities. HyperFrames is the default output framework unless the user explicitly chooses another framework for the deliverable or asks only to record a browser session.
agent-context-audit
Audit a repo's agent context — CLAUDE.md files, codebase docs, skills, and tool/MCP designs — against Anthropic's Claude 5 context-engineering guidance ("unhobbling": Anthropic cut ~80% of Claude Code's system prompt with no eval loss). Finds overconstraint, conflicting instructions, redundancy, stale facts, and missing "unknown knowns"; produces a scored findings report with concrete rewrites, then applies approved fixes. Use when someone says "audit my CLAUDE.md", "context audit", "unhobble this repo", "review our agent docs/skills/tools", or after upgrading to Claude 5-generation models.
connect-our-agents
Connect your agent with the agent of anyone who uses Fulcra (a friend, a colleague, a company's support bot) and exchange messages privately: only the people in the connection can read them, and nothing else on either account is shared. Use when the user says connect my agent with X, invite X's agent, send this to X's agent, did X's agent reply, check my agent messages, accept this agent invitation, or who is my agent connected to.
docetl
Build and run LLM-powered data processing pipelines with DocETL. Use when users say "docetl", want to analyze unstructured data, process documents, extract information, or run ETL tasks on text. Helps with data collection, pipeline creation, execution, and optimization.