Manage evaluators

​

View evaluators

​

Create an evaluator

​

Create an evaluator in the UI

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Click + Evaluator to open the new evaluator panel.
3. The panel lets you:
   • Create from scratch: Build a new LLM-as-a-Judge or Code evaluator.
   • Add a LangChain Tuned Evaluator: Attach a specialized judge managed by LangChain to a compatible tracing project without configuring a prompt, model, or API key.
   • Create from a template: Start from a ready-made evaluator (also known as a prebuilt evaluator) for common evaluation patterns. A Recommended section surfaces popular templates first, followed by templates organized by the following categories:

     Category	Description
     Security	Detect leaks, injections, and adversarial inputs.
     Safety	Evaluate content safety and moderation.
     Quality	Measure output quality and accuracy.
     Conversation	Evaluate conversational quality and user experience.
     Trajectory	Evaluate agent tool use and decision paths.
     Image Evaluations	Evaluate image content quality and safety.
     Voice Evaluation	Evaluate voice and audio interaction quality.

​

Create an evaluator with the SDK

import asyncio

from langsmith import Client


async def main():
    client = Client()

    created = await client.evaluators.create(
        name="Correctness evaluator",
        type="code",
        code_evaluator={
            "code": "def perform_eval(run, example):\n    return {'score': 1}",
            "language": "python",
        },
    )
    print("Created evaluator:", created.evaluator.id)


asyncio.run(main())

import { Client } from "langsmith";

const client = new Client();

const created = await client.evaluators.create({
  name: "Correctness evaluator",
  type: "code",
  code_evaluator: {
    code: "def perform_eval(run, example):\n    return {'score': 1}",
    language: "python",
  },
});
console.log("Created evaluator:", created.evaluator?.id);

​

View evaluator details

• Overview: The evaluator’s feedback configuration and prompt or code definition.
• Traces: Traces processed by this evaluator across all attached resources.
• Logs: Execution logs for this evaluator across all attached resources.
• Projects & Datasets: The tracing projects and datasets this evaluator is attached to, with each attachment’s weekly spend and limit.

​

Edit an evaluator

​

Manage evaluator trace retention

1. When you create or edit an online evaluator, set the source to a tracing project, rather than a dataset.
2. Expand the Advanced section in the evaluator configuration panel.
3. Clear Extend trace retention.

​

Include extended stats

1. When you create or edit a run-level evaluator, expand the Advanced section in the evaluator configuration panel.
2. Select Include extended stats (feedback, costs, tokens).

​

Access extended stats

​

Chain evaluators

​

Delete an evaluator

1. In the LangSmith UI, select Evaluators in the left sidebar.
2. Select the evaluator you want to delete.
3. Open the Projects & Datasets tab. For each attached tracing project and dataset, select Detach in the Actions menu at the right of the row.
4. Return to the Evaluators page and click Delete at the top of the page.