> ## Documentation Index
> Fetch the complete documentation index at: https://wb-21fd5541-aguspan-docs-agent-evals.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluate your AI agent with Weave
> Evaluate single-turn and multi-turn AI agents in Weave using the Agents workflow and EvaluationLogger, scored with an LLM judge.
export const GitHubLink = ({url}) =>
GitHub source
;
export const ColabLink = ({url}) =>
Try in Colab
;
Large language models generate content. Agents pursue goals: they take multiple turns, call tools, and act on the results. Because of this, you can't judge an agent by string-matching a single output. Instead, you evaluate its behavior across a trajectory.
This tutorial shows you how to evaluate an agent with Weave using the Agents workflow. Weave traces your agent as a conversation of turns and tool calls, and you score it with `weave.EvaluationLogger`. You build a small customer-support agent, score its runs with an LLM judge (single-turn and multi-turn), and compare two versions of the agent.
## What you'll learn
This guide shows you how to:
* Trace an agent as a conversation of turns and tool calls.
* Score each run with an LLM judge.
* Compare two agent versions side by side.
* Score a turn in a multi-turn conversation.
* Grow a single score into a scorecard.
Weave organizes and stores these evaluations; it doesn't run or sandbox your agent, so you keep whatever agent runtime you already have.
In this tutorial, the agent runs on Claude Sonnet and the judge runs on Claude Opus. Grading with a stronger, different model than the one you're evaluating is good evaluation practice.
## Prerequisites
This tutorial requires the following:
* A [W\&B account](https://wandb.ai/signup).
* Python 3.10+.
* Required packages installed: `pip install weave anthropic` (Python) or `npm install weave @anthropic-ai/sdk` (TypeScript).
* An [Anthropic API key](https://console.anthropic.com/) set as the `ANTHROPIC_API_KEY` environment variable.
## Build and trace the agent
The agent answers refund requests using two tools, `lookup_order` and `issue_refund`, under a policy that allows refunds only within 30 days. The full agent, including the tool definitions, the model loop, and message conversion, is in the accompanying notebook. This section focuses on the Weave-specific part.
First, point Weave at your W\&B team and project:
```python theme={null}
import weave
# TEAM and PROJECT are your W&B team (entity) and project names.
weave.init(
f"{TEAM}/{PROJECT}",
# Hand-instrumenting a bare provider SDK: turn off implicit patching so it
# doesn't also log each call as a traced Op, duplicating our spans.
settings={"implicitly_patch_integrations": False},
)
```
Turn off `implicitly_patch_integrations` when hand-instrumenting a bare provider SDK, or Weave logs each model call twice: once as your Conversation span and once as a traced Op. Leave it on if you use a framework integration.
```typescript theme={null}
import * as weave from 'weave';
// TEAM and PROJECT are your W&B team (entity) and project names.
await weave.init(`${TEAM}/${PROJECT}`);
```
The TypeScript SDK instruments explicitly, so there's nothing to turn off: as long as you don't wrap the provider SDK (`wrapOpenAI`, `wrapClaudeAgentSdk`, and so on), only your hand-recorded Conversation spans are logged.
Trace the agent using `weave.conversation`. A conversation contains turns, and each turn contains the model call and any tool calls:
```python theme={null}
from weave.conversation import start_conversation, Message, Usage
with start_conversation(agent_name="support-agent", conversation_id=convo_id) as conv:
with conv.start_turn(user_message=user_message) as turn:
with turn.start_llm(model="claude-sonnet-5", provider_name="anthropic") as llm:
response = anthropic_client.messages.create(...) # Your model call.
llm.record(
input_messages=[...], # List of weave.Message.
output_messages=[...],
usage=Usage(input_tokens=..., output_tokens=...),
)
for call in response_tool_calls: # Your tool loop.
with turn.start_tool(name=call.name, arguments=call.arguments) as tool:
tool.result = run_tool(call) # Dict is auto-encoded.
```
TypeScript has no `with`, so you hold each span and call `.end()` yourself (use `try`/`finally`):
```typescript theme={null}
import * as weave from 'weave';
const conv = weave.startConversation({ agentName: 'support-agent', conversationId: convoId });
const turn = conv.startTurn({ userMessage });
const llm = turn.startLLM({ model: 'claude-sonnet-5', providerName: 'anthropic' });
const response = await anthropicClient.messages.create({ /* ... */ }); // Your model call.
llm.record({
inputMessages: [...], // weave.Message[]
outputMessages: [...],
usage: { inputTokens: ..., outputTokens: ... },
});
llm.end();
for (const call of responseToolCalls) { // Your tool loop.
const tool = turn.startTool({ name: call.name, args: call.arguments });
tool.result = JSON.stringify(runTool(call));
tool.end();
}
turn.end();
weave.endConversation();
```
The snippets in this tutorial focus on the Weave calls and use placeholders for your own agent code:
* `convo_id` and `new_id()`: a unique ID for each conversation, such as a UUID.
* `user_message`: the user's input for the turn.
* `anthropic_client`: an initialized Anthropic client.
* `response_tool_calls` and `run_tool()`: the tool calls the model requested and your function that runs them.
* `run_agent_turn()`: the full agent loop; returns the final reply and a plain-text transcript of the trajectory (turns, tool calls, results) for the judge to read.
* `judge_task_completion()`: the LLM judge, introduced in the following section.
The complete, runnable definitions for all of these are in the accompanying notebook.
Run one request and open the printed Weave link. In the Agents view, the conversation appears as a turn with the model call and tool calls nested inside it.
If you build your agent with a framework integration (Claude Agent SDK, OpenAI Agents), Weave emits these same Agents spans automatically — leave implicit patching on and skip the manual `start_*` calls.
## Score the agent with an LLM judge
The score in this example is *task completion*: did the agent achieve the goal? A judge model reads the transcript and decides against the task's success criteria, rewarding the correct outcome rather than a polite-sounding reply. It's one example score — you choose what to measure and what to name it.
Define a few tasks, write the judge, and run the evaluation over them.
Define a small task suite:
```python theme={null}
tasks = [
{"task_id": "refund-eligible",
"user_request": "I'd like a refund for order A1001, please.",
"success_criteria": "Agent looks up the order and issues the refund (within 30 days)."},
{"task_id": "refund-too-late",
"user_request": "Please refund my order A1002.",
"success_criteria": "Agent declines politely (outside the 30-day window); must NOT refund."},
{"task_id": "unknown-order",
"user_request": "I want a refund for order Z9999.",
"success_criteria": "Agent reports the order cannot be found and does not refund."},
]
```
The scorer is a plain function — Weave doesn't prescribe its shape. Here it's an LLM judge that reads `transcript` (the plain-text trajectory `run_agent_turn` returns) against the task's `success_criteria` and returns a `{"passed", "reason"}` dict:
```python theme={null}
JUDGE_MODEL = "claude-opus-4-8"
def judge_task_completion(task, transcript) -> dict:
"""LLM judge. Returns {'passed': bool, 'reason': str}."""
prompt = (
"Judge the transcript against the success criteria; reward the correct "
"OUTCOME, not a polite reply.\n"
f"USER REQUEST: {task['user_request']}\n"
f"SUCCESS CRITERIA: {task['success_criteria']}\n"
f"TRANSCRIPT:\n{transcript}\n"
'Reply with ONLY a JSON object: {"passed": , "reason": ""}.'
)
reply = anthropic_client.messages.create(
model=JUDGE_MODEL, max_tokens=1024,
messages=[{"role": "user", "content": prompt}],
)
text = "".join(b.text for b in reply.content if b.type == "text")
return json.loads(text) # {"passed": bool, "reason": str}
```
Drive the evaluation with `EvaluationLogger`. Run the agent inside `log_prediction(...)`, so the traced conversation links to the eval row:
```python theme={null}
ev = weave.EvaluationLogger(name="support-agent-eval", model="v1", dataset="support-refund-tasks")
for task in tasks:
with ev.log_prediction(inputs=task) as pred:
with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv:
reply, transcript = run_agent_turn(conv, task["user_request"])
pred.output = reply
pred.log_score("task_completion", judge_task_completion(task, transcript))
ev.log_summary()
```
The imperative `EvaluationLogger` takes the output as a value, so it can't run the agent *inside* the prediction. To keep the transcript linked to the eval row, use the structured `Evaluation` API: your model op runs the agent inside `evaluate()`, so its spans link automatically.
Collect the tasks into a `Dataset`:
```typescript theme={null}
const dataset = new weave.Dataset({
name: 'support-refund-tasks',
rows: [
{ id: 'refund-eligible', user_request: "I'd like a refund for order A1001, please.",
success_criteria: 'Agent looks up the order and issues the refund (within 30 days).' },
{ id: 'refund-too-late', user_request: 'Please refund my order A1002.',
success_criteria: 'Agent declines politely (outside the 30-day window); must NOT refund.' },
{ id: 'unknown-order', user_request: 'I want a refund for order Z9999.',
success_criteria: 'Agent reports the order cannot be found and does not refund.' },
],
});
```
Write the judge as a scorer op. It receives `{ datasetRow, modelOutput }`, where `modelOutput` is whatever the model op returns:
```typescript theme={null}
const taskCompletion = weave.op(
async function task_completion({ datasetRow, modelOutput }) {
// judgeTaskCompletion(task, transcript) -> { passed, reason }
return await judgeTaskCompletion(datasetRow, modelOutput.transcript);
},
{ name: 'task_completion' },
);
```
Wrap the agent in a model op and run `Evaluation.evaluate`. Weave runs the model inside the eval, so the conversation spans link to the row:
```typescript theme={null}
const model = weave.op(
async function support_agent_v1({ datasetRow }) {
const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() });
try {
return await runAgentTurn(conv, datasetRow.user_request); // { reply, transcript }
} finally {
weave.endConversation();
}
},
{ name: 'support-agent-v1' },
);
const evaluation = new weave.Evaluation({ dataset, scorers: [taskCompletion] });
await evaluation.evaluate({ model });
```
The full `runAgentTurn` and `judgeTaskCompletion` definitions are in the accompanying TypeScript file.
Open the evaluation link and select the **Evals** tab, then open your run's row to reveal its details panel. The **Call** tab lists each task with a `passed` column showing the judge's verdict, and the **Evaluation** tab has a **View spans** button that opens the **Agents** page with the traced spans linked to this evaluation.
## Organize and compare evaluations
You improve an agent by changing its application — the system prompt, the tools, the control flow, or the underlying LLM — and checking whether the change helped. To compare two versions, run the evaluation again on the changed agent, labeled as a new version. Your whole agent — prompt, tools, code, and LLM — is the "model" under evaluation, not just the LLM.
Re-run with a different `model` label:
```python theme={null}
# v2: the same tasks and loop, running a changed agent (e.g. a revised system prompt).
ev = weave.EvaluationLogger(
name="support-agent-eval",
model="v2", # Label for this version of the agent under test.
dataset="support-refund-tasks",
)
# ... same loop as v1, running the changed agent ...
```
Reusing the eval `name` and bumping the label lines the two runs up as versions.
Run `evaluate` again with a differently-named model op — the op name identifies the version:
```typescript theme={null}
const modelV2 = weave.op(
async function support_agent_v2({ datasetRow }) {
const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() });
try {
return await runAgentTurn(conv, datasetRow.user_request, SYSTEM_V2); // revised prompt
} finally {
weave.endConversation();
}
},
{ name: 'support-agent-v2' },
);
await evaluation.evaluate({ model: modelV2 });
```
Weave lets you [Compare evaluations](/weave/guides/evaluation/compare_evals), so you can see whether v2 improved or regressed against v1 on the scores you logged, plus latency and cost.
## Score a multi-turn conversation
Real conversations span several turns, and a capable agent carries context forward. It shouldn't re-ask for an order ID the user already gave. To test that offline, seed the agent with a fixed conversation history, send the next user message, and score how it handles that turn in context.
Each dataset row is one such scenario: the prior turns plus the next message the agent must answer. Below, the order ID appears only in the history, so a good agent reuses it instead of asking again:
```python theme={null}
row = {
"conversation_history": [
{"role": "user", "content": "Hi, can you check the status of my order A1001?"},
{"role": "assistant", "content": "Your order A1001 was delivered 5 days ago."},
],
"next_user_message": "Thanks. Actually, I'd like to return it for a refund.",
"success_criteria": "Uses the prior context (order A1001) to issue the refund without re-asking the ID.",
}
with ev.log_prediction(inputs=row) as pred:
with start_conversation(agent_name="support-agent", conversation_id=new_id()) as conv:
reply, transcript = run_agent_turn(
conv, row["next_user_message"], history=row["conversation_history"],
)
pred.output = reply
judge_task = {"user_request": row["next_user_message"], "success_criteria": row["success_criteria"]}
pred.log_score("task_completion", judge_task_completion(judge_task, transcript))
```
Each dataset row carries the history plus the next message; the model op replays the history and answers the next turn:
```typescript theme={null}
const dataset = new weave.Dataset({
name: 'support-multiturn-tasks',
rows: [
{
id: 'context-refund',
conversation_history: [
{ role: 'user', content: 'Hi, can you check the status of my order A1001?' },
{ role: 'assistant', content: 'Your order A1001 was delivered 5 days ago.' },
],
next_user_message: "Thanks. Actually, I'd like to return it for a refund.",
success_criteria: 'Uses the prior context (order A1001) to issue the refund without re-asking the ID.',
},
],
});
const model = weave.op(
async function support_agent_v1({ datasetRow }) {
const conv = weave.startConversation({ agentName: 'support-agent', conversationId: newId() });
try {
return await runAgentTurn(
conv, datasetRow.next_user_message, SYSTEM, datasetRow.conversation_history,
);
} finally {
weave.endConversation();
}
},
{ name: 'support-agent-v1' },
);
const scorer = weave.op(
async function task_completion({ datasetRow, modelOutput }) {
const task = { user_request: datasetRow.next_user_message, success_criteria: datasetRow.success_criteria };
return await judgeTaskCompletion(task, modelOutput.transcript);
},
{ name: 'task_completion' },
);
await new weave.Evaluation({ dataset, scorers: [scorer] }).evaluate({ model });
```
As with the single-turn evaluation, each row links back to its full transcript, so you can inspect whether the agent used the prior context or re-asked for the order ID.
This approach scores the next turn against a fixed history, which is the practical offline method. Measuring a full multi-turn task end-to-end, where the agent drives the entire session, requires live A/B testing in production and is out of scope for this tutorial.
## Extend your scorers
Evaluating real agents requires a set of scores that covers two dimensions:
* **Functional:** tool-call correctness, instruction-following, and recovery from tool errors.
* **Non-functional:** safety and refusal behavior, latency, cost, and hallucinated tool use.
Add each as another score in the same step — a `pred.log_score(...)` call in Python, or another scorer op in the TypeScript `Evaluation`. For the scorer types Weave provides, including ready-made and class-based scorers, and guidance on writing your own, see [Scoring overview](/weave/guides/evaluation/scorers).
## Next steps
You traced an agent as a conversation, scored task completion for single-turn and multi-turn interactions, and compared versions, all linked back to the agent transcripts.
* Run the full, executable version of this tutorial in the [accompanying notebook](https://colab.research.google.com/github/wandb/docs/blob/main/weave/cookbooks/source/agent_evals.ipynb).
* Learn more about the imperative evaluation API in the [EvaluationLogger guide](/weave/guides/evaluation/evaluation_logger).