LangSmith opens trace-based fine-tuning in public beta

• LangSmith Fine-Tuning is in public beta, pairing its trace store with the smithtune CLI to make specialized models, which LangChain's beta announcement says can target lower cost and latency.
• The pipeline moves from exported trajectories to SFT examples, provider-hosted training, and evaluation, according to LangChain's four-step workflow.
• smithtune trains with Baseten or Fireworks, and baseten's launch note says a finished checkpoint can be deployed to Baseten with one command.
• Human review remains part of the loop: Vtrivedy10's response calls autonomous eval generation "super noisy" and says people still review individual evals and the system that produces them.

The smithtune README includes an optional agent council that votes on which trajectories to keep, while LangSmith's trace-export docs document lower-level list_runs and /runs/query access to run data. Its trajectory-evaluation docs focus on testing whether a live-model agent followed the expected sequence of actions.

LangSmith's public-beta product connects its trace data to an open-source CLI, smithtune. The smithtune repository describes a complete path from recorded trajectories through base-versus-tuned comparisons in LangSmith, with endpoint deployment optional.

The CLI’s full command flow is dataset pull → optional dataset triage → dataset push → prepare → plan → train --evaluate → optional deploy. Triage can use the optional council to judge traces against a rubric before training.

A trajectory is LangSmith’s chronological, conversational representation of an agent session. It gathers human, AI, tool, and system messages from the main agent and subagents, then orders them by when they first appeared.

Vtrivedy10 had already framed traces as the key asset for improving agents over time. In that earlier post, the proposed operations loop includes:

• Monitoring every trace with custom models, Jev, small models, and built-in rubrics.
• Clustering traces and errors into human-digestible views.
• Turning errors into evals that can be hill-climbed.
• Open-sourcing tooling for Harbor environments built from a team’s data.

The four stages in LangChain's workflow are:

1. Identify and export trajectories from LangSmith.
2. Transform traces into SFT-ready training examples.
3. Fine-tune open models with Baseten or Fireworks.
4. Evaluate the result in LangSmith.

baseten describes pointing an agent at LangSmith traces to generate a dataset and start post-training on Baseten Loops. Its launch note adds the one-command deployment path; Vtrivedy10 said LangChain Labs uses Loops to move from an idea to a training experiment in minutes.

The launch premise is that agent trajectories contain useful operational behavior, but Vtrivedy10 said converting raw trajectories into environments and curated training data remains difficult because agents are poor data curators.

mattlam_ asked in a reply how the system would detect regressions and whether agents could cheat at eval creation. Vtrivedy10's answer named evals and dogfooding, with agents mining eval outputs, but retained human review at both the individual-eval and semi-autonomous-system levels.

Fine-tuning arrived alongside four other Interrupt announcements, as LangChain's recap enumerated:

• LangSmith Engine v2.
• Managed Deep Agents 0.8.
• LangSmith Trajectories.
• LangSmith Fine-Tuning.
• Custom Apps.

That bundle also includes cron-based autonomous runs. In LangChain's scheduled-agent demo, a file under schedules/ contains a cron expression and prompt, then the deployed agent executes on that schedule.