Evaluate Managed Deep Agents

Managed Deep Agents evals are Harbor tasks. Use a coding agent with the eval-engineering skill to inspect the project, draft a Task Spec for your review, and write complete tasks under evals/. Managed Deep Agents initializes the Harbor workspace. Harbor runs the managed agent against each task in an isolated environment and records the result.

Managed Deep Agents is in public beta and available on LangSmith Cloud in the US region only.

Prerequisites

Before you evaluate, make sure you have:

A Managed Deep Agents project created with mda init, or an existing project with an agent entry.
uv, which runs the pinned Harbor version and plugins.
Docker, which Harbor uses for task environments.
A coding agent. Agents that support Agent Skills can install eval-engineering directly. For other agents, provide the skill instructions in the session.

Add the `eval-engineering` skill

The eval-engineering skill walks a coding agent through discovering the agent, proposing a Task Spec, and building a reviewed Harbor task. To add it to the current project, run:

uvx --from npx-skills skills add langchain-ai/langchain-skills --skill eval-engineering --yes

You can use any coding agent.

To use Deep Agents Code (dcode), install it with:

curl -LsSf https://langch.in/dcode | bash

See the Deep Agents Code quickstart for provider setup and interactive use.

Develop evals with a coding agent

Initialize the eval workspace

From the project root, run:

mda evals init -i

The interactive handoff lists detected coding agents, including Deep Agents Code, Claude Code, Codex, and Cursor. Selecting an agent starts that agent in the project directory and runs the eval-engineering prompt. You can also copy the prompt for another agent, or exit and return later.Initialization creates:

my-agent/
├── evals/
│   └── harbor-job.json
└── .mda/
    └── evals/
        ├── runtime.json
        └── harbor-adapter/

evals/harbor-job.json is user-owned. Managed Deep Agents writes it only when it is missing, so later edits are preserved. Files under .mda/evals/ are generated.

Start the coding-agent session

The handoff asks the selected coding agent to install the eval-engineering skill and use it for the project. If you already added the skill, continue in that session.Ask the coding agent to follow the skill’s review flow and use the Managed Deep Agents task layout:

Use the eval-engineering skill to develop Harbor evals for this Managed
Deep Agent. Inspect the project and existing evals first. Draft the Task
Spec and wait for my review before implementing the approved task directly
under evals/<task>/.

Work with the coding agent to review the Task Spec, task instruction, environment, verifier, and reusable project knowledge. The coding agent writes the runnable task after you approve the design.

Review the Harbor task

Each direct child of evals/ that contains an instruction and tests is a Harbor task:

evals/
├── harbor-job.json
└── <task>/
    ├── Task.md
    ├── instruction.md
    ├── task.toml
    ├── environment/
    │   └── Dockerfile
    └── tests/
        ├── test.sh
        └── <verifier>

Task.md is the human-reviewed spec. instruction.md tells the agent what to do. Harbor builds the task environment, runs the managed agent, and then runs tests/test.sh. The verifier writes a numeric reward to /logs/verifier/reward.txt or numeric metrics to /logs/verifier/reward.json.For the full task format, see the Harbor task documentation.

Run the evals

The coding-agent handoff includes a command configured for the project and current shell. Run that command from the project root.On macOS or Linux, it has the following form:

HARBOR_LANGSMITH_DATASET=mda-my-agent-evals \
PYTHONPATH=.mda/evals/harbor-adapter \
uv run --env-file .env --python 3.12 --with 'harbor[langsmith]==0.21.0' harbor run \
  --config evals/harbor-job.json --yes \
  --plugin mda_harbor.job_plugin:MDAJobPlugin \
  --plugin mda_harbor.langsmith_plugin:LangSmithPlugin

Replace my-agent with the project directory name. The generated command fills in the name and uses PowerShell syntax on Windows.Re-running the command after editing the agent picks up the project changes.

Inspect the results

Open the Harbor results:

uv run --python 3.12 --with 'harbor[langsmith]==0.21.0' \
  harbor view .mda/evals/jobs

Review failed trials with your coding agent. Update the task or verifier when the eval does not measure the intended behavior. Update the managed agent when the eval exposes a product failure, then run the same Harbor command again.

Edit the Harbor job

Edit evals/harbor-job.json to change datasets, attempts, concurrency, environment settings, or agent environment variables. Managed Deep Agents preserves the file when you run mda evals init again.

Record runs in LangSmith

When LANGSMITH_API_KEY is available, the LangSmith plugin records the Harbor runs in the dataset named by HARBOR_LANGSMITH_DATASET.

Get started

Agent definition

Build and deploy

Prerequisites

Add the `eval-engineering` skill

Develop evals with a coding agent

Edit the Harbor job

Record runs in LangSmith

See also

​Prerequisites

​Add the eval-engineering skill

​Develop evals with a coding agent

​Edit the Harbor job

​Record runs in LangSmith

​See also

Prerequisites

Add the `eval-engineering` skill

Develop evals with a coding agent

Edit the Harbor job

Record runs in LangSmith

See also