- Inline in SDK code: Define scorers directly in your evaluation scripts for local development or application-specific logic.
- Pushed via CLI: Define scorers in TypeScript or Python files and push them to Braintrust for team-wide sharing and automatic evaluation of production logs.
- Created in UI: Build scorers in the Braintrust web interface using the built-in code editor.
Score spans
Span-level scorers evaluate individual operations or outputs. Use them for measuring single LLM responses, checking specific tool calls, or validating individual outputs. Each matching span receives an independent score. Your scorer function receives these parameters:input: The input to your taskoutput: The output from your taskexpected: The expected output (optional)metadata: Custom metadata from the test case
null. Treat both forms as unavailable.
Return a number between 0 and 1, or an object with score and optional name and metadata. When name is omitted, Braintrust uses the scorer function’s name as the score key.
In Ruby, declare only the parameters you need as keyword arguments. The runner automatically filters out the rest: |output:, expected:|.
- SDK
- CLI
- UI
Use scorers inline in your evaluation code:
equality_scorer.eval.ts
Score traces
Trace-level scorers evaluate entire execution traces including all spans and conversation history. Use these for assessing multi-turn conversation quality, agent behavior such as tool usage and trajectory, or overall workflow completion. Trace-level scorers are the right choice whenever a scorer needs the full execution context rather than a single span. The scorer runs once per trace. Your handler function receives thetrace parameter, which provides methods for accessing execution data:
-
Get spans: Returns spans matching the filter. Each span includes
input,output,expected,metadata,tags,scores,metrics,error(populated when the span failed),span_id,span_parents, andspan_attributes. Omit the filter to get all spans, or pass multiple types like["llm", "tool"].- TypeScript:
trace.getSpans({ spanType: ["llm"] }) - Python:
trace.get_spans(span_type=["llm"]) - Java:
trace.getSpans("llm") - Ruby:
trace.spans(span_type: "llm") - C#:
trace.GetSpansAsync("llm")
getSpansaccepts additional filter fields beyondspanType:name(string array, matched againstspan_attributes.name),hasError(boolean),metadata(object, matched at any nesting depth), andduration({ min, max }in seconds). Combine them withspanTypeto narrow by both type and behavior. The TypeScript SDK requires v3.33.0 or later for these fields, and the Python SDK requires v0.39.0 or later.In the Python SDK, these fields are passed through a separatefiltersargument (has_error,metadata, andduration), as shown above. - TypeScript:
-
Get thread: Returns an array of conversation messages extracted from LLM spans.
- TypeScript:
trace.getThread() - Python:
trace.get_thread() - Java:
trace.getLLMConversationThread() - Ruby:
trace.thread - C#:
trace.GetThreadAsync()
- TypeScript:
input, output, expected, and metadata are automatically populated from the root span and passed to your scorer function.
Trace-level scoring requires TypeScript SDK v2.2.1+, Python SDK v0.5.6+, Java SDK v0.3.8+, Ruby SDK v0.2.1+, or C# SDK v0.2.3+.
- SDK
- CLI
- UI
Use scorers inline in your evaluation code:
trace_code_scorer.eval.ts
Trace scorer recipes
Use trace scorers for checks that depend on the agent’s trajectory, such as tool usage, tool failures, or step budgets. Add any of these scorers to thescores array in an Eval, or adapt the handler body for a CLI or UI scorer.
trace_scorer_recipes.eval.ts
Score groups
Group scope is available only for online scoring rules.
trace. Braintrust backs the trace argument with the traces included in the group for that scoring run, so getSpans() returns a flat list of spans from those traces and getThread() returns their conversation history.
Group scope isn’t needed when each conversation is a single trace. To log every turn to the same trace, pass span.export() as each turn’s parent. A trace-level scorer then reads the whole conversation from trace, in experiments as well as in online scoring.
For example, this custom code scorer reads the grouped conversation with trace.getThread(), checks whether the final user message includes thanks, and returns the number of user turns in score metadata:
TypeScript
trace. The row-level input, output, expected, and metadata parameters do not describe the group.
Set pass thresholds
Define minimum acceptable scores to automatically mark results as passing or failing. When configured, scores that meet or exceed the threshold are marked as passing (green highlighting with checkmark), while scores below are marked as failing (red highlighting).Pass thresholds apply only to scorers that output numeric scores. Classifiers, which output labels, don’t use them.
- SDK
- UI
Add To set a pass threshold for a Ruby scorer without the UI, register it with the API.
__pass_threshold to the scorer’s metadata (value between 0 and 1):Return multiple scores
A single scorer can return an array of score objects to emit multiple named metrics from one call. This is useful when several quality dimensions can be computed together or share computation. Each item appears as its own score column in the Braintrust UI. Each item requiresname and score. metadata is optional.
Apply classification labels
A classifier returns a categorical label instead of a numeric score. Define custom code classifiers inline in your eval code, as a function that evaluates a result and constructs one or more classifications. Each classification your function returns sets aname (the group it belongs to, such as intent), an id (the value you filter by, such as password_reset), an optional label for display (such as Password reset), and optional metadata. Unlike an LLM-as-a-judge classifier, custom code sets these fields independently and can return more than one classification at a time.
To create a classifier in the UI, build an LLM-as-a-judge classifier.
For the C# and Java examples, use the
BRAINTRUST_DEFAULT_PROJECT_NAME environment variable to set a project name. Otherwise, the default project is default-dotnet-project (C#) or default-java-project (Java).classifier_errors.
A classifier can also assign multiple labels at once:
Classifiers require TypeScript SDK v3.9.0+, Python SDK v0.16.0+, Go SDK v0.8.0+, Java SDK v0.3.12+, Ruby SDK v0.4.0+, or C# SDK v0.2.8+.
Build Python bundles for BYOC
Python bundles with native dependencies must match the Python major and minor version and CPU architecture of the runtime that executes them. On BYOC deployments on Google Cloud, custom code runs in the API container. Ask your account team for its Python version and architecture before building a bundle. See code execution security for isolation details. Create an environment with the matching interpreter, install your dependencies, and use that interpreter when you push the function. For example, if your account team confirms Python 3.13:--runner option selects the build interpreter. It does not change the deployment’s runtime. Rebuild and push the bundle if invocation fails because a native extension targets a different Python version or architecture.
Next steps
- Autoevals for pre-built scorers without writing code
- LLM-as-a-judge for natural language evaluation criteria
- Run evaluations using your scorers
- Score production logs with online scoring rules