> ## Documentation Index
> Fetch the complete documentation index at: https://langchain-5e9cc07a-preview-opensw-1787674483-6bd21af.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate Managed Deep Agents

> Develop Harbor evals for Managed Deep Agents with a coding agent and the eval-engineering skill.

Managed Deep Agents evals are [Harbor](https://www.harborframework.com/docs/tasks) tasks. Use a coding agent with the [`eval-engineering` skill](https://github.com/langchain-ai/langchain-skills/blob/main/config/skills/eval-engineering/SKILL.md) to inspect the project, draft a Task Spec for your review, and write complete tasks under `evals/`.

Managed Deep Agents initializes the Harbor workspace. Harbor runs the managed agent against each task in an isolated environment and records the result.

<Note>
  Managed Deep Agents is in **public [beta](/langsmith/release-stages)** and available on [LangSmith Cloud](/langsmith/cloud) in the US region only.
</Note>

## Prerequisites

Before you evaluate, make sure you have:

* A Managed Deep Agents project created with `mda init`, or an existing project with an agent entry.
* [`uv`](https://docs.astral.sh/uv/), which runs the pinned Harbor version and plugins.
* [Docker](https://docs.docker.com/get-docker/), which Harbor uses for task environments.
* A coding agent. Agents that support Agent Skills can install `eval-engineering` directly. For other agents, provide the skill instructions in the session.

## Add the `eval-engineering` skill

The [`eval-engineering` skill](https://github.com/langchain-ai/langchain-skills/blob/main/config/skills/eval-engineering/SKILL.md) walks a coding agent through discovering the agent, proposing a Task Spec, and building a reviewed Harbor task. To add it to the current project, run:

```bash theme={null}
npx skills add langchain-ai/langchain-skills --skill eval-engineering --yes
```

You can use any coding agent.

<Tip>
  To use [Deep Agents Code](/oss/deepagents/code/overview) (`dcode`), install it with:

  ```bash theme={null}
  curl -LsSf https://langch.in/dcode | bash
  ```

  See the [Deep Agents Code quickstart](/oss/deepagents/code/quickstart) for provider setup and interactive use.
</Tip>

## Develop evals with a coding agent

<Steps>
  <Step title="Initialize the eval workspace" id="initialize-the-eval-workspace">
    From the project root, run:

    ```bash theme={null}
    mda evals init -i
    ```

    The interactive handoff lists detected coding agents, including Deep Agents Code, Claude Code, Codex, and Cursor. Selecting an agent starts that agent in the project directory and runs the eval-engineering prompt. You can also copy the prompt for another agent, or exit and return later.

    Initialization creates:

    ```text theme={null}
    my-agent/
    ├── evals/
    │   └── harbor-job.json
    └── .mda/
        └── evals/
            ├── runtime.json
            └── harbor-adapter/
    ```

    `evals/harbor-job.json` is user-owned. Managed Deep Agents writes it only when it is missing, so later edits are preserved. Files under `.mda/evals/` are generated.
  </Step>

  <Step title="Start the coding-agent session" id="start-the-coding-agent-session">
    The handoff asks the selected coding agent to install the `eval-engineering` skill and use it for the project. If you already added the skill, continue in that session.

    Ask the coding agent to follow the skill's review flow and use the Managed Deep Agents task layout:

    ```text theme={null}
    Use the eval-engineering skill to develop Harbor evals for this Managed
    Deep Agent. Inspect the project and existing evals first. Draft the Task
    Spec and wait for my review before implementing the approved task directly
    under evals/<task>/.
    ```

    Work with the coding agent to review the Task Spec, task instruction, environment, verifier, and reusable project knowledge. The coding agent writes the runnable task after you approve the design.
  </Step>

  <Step title="Review the Harbor task" id="review-the-harbor-task">
    Each direct child of `evals/` that contains an instruction and tests is a Harbor task:

    ```text theme={null}
    evals/
    ├── harbor-job.json
    └── <task>/
        ├── Task.md
        ├── instruction.md
        ├── task.toml
        ├── environment/
        │   └── Dockerfile
        └── tests/
            ├── test.sh
            └── <verifier>
    ```

    `Task.md` is the human-reviewed spec. `instruction.md` tells the agent what to do. Harbor builds the task environment, runs the managed agent, and then runs `tests/test.sh`. The verifier writes a numeric reward to `/logs/verifier/reward.txt` or numeric metrics to `/logs/verifier/reward.json`.

    For the full task format, see the [Harbor task documentation](https://www.harborframework.com/docs/tasks).
  </Step>

  <Step title="Run the evals" id="run-the-evals">
    The coding-agent handoff includes a command configured for the project and current shell. Run that command from the project root.

    On macOS or Linux, it has the following form:

    ```bash theme={null}
    HARBOR_LANGSMITH_DATASET=mda-my-agent-evals \
    PYTHONPATH=.mda/evals/harbor-adapter \
    uv run --env-file .env --python 3.12 --with 'harbor[langsmith]==0.21.0' harbor run \
      --config evals/harbor-job.json --yes \
      --plugin mda_harbor.job_plugin:MDAJobPlugin \
      --plugin mda_harbor.langsmith_plugin:LangSmithPlugin
    ```

    Replace `my-agent` with the project directory name. The generated command fills in the name and uses PowerShell syntax on Windows.

    Re-running the command after editing the agent picks up the project changes.
  </Step>

  <Step title="Inspect the results" id="inspect-the-results">
    Open the Harbor results:

    ```bash theme={null}
    uv run --python 3.12 --with 'harbor[langsmith]==0.21.0' \
      harbor view .mda/evals/jobs
    ```

    Review failed trials with your coding agent. Update the task or verifier when the eval does not measure the intended behavior. Update the managed agent when the eval exposes a product failure, then run the same Harbor command again.
  </Step>
</Steps>

## Edit the Harbor job

Edit `evals/harbor-job.json` to change datasets, attempts, concurrency, environment settings, or agent environment variables. Managed Deep Agents preserves the file when you run `mda evals init` again.

## Record runs in LangSmith

When `LANGSMITH_API_KEY` is available, the LangSmith plugin records the Harbor runs in the dataset named by `HARBOR_LANGSMITH_DATASET`.

## See also

* [CLI reference](/langsmith/javascript/managed-deep-agents-cli): review `mda evals init` and related flags.
* [Deploy an agent](/langsmith/javascript/managed-deep-agents-deploy): deploy the agent after its evals pass.
* [Deep Agents Code quickstart](/oss/deepagents/code/quickstart): install and run `dcode`.
* [Harbor integrations](/langsmith/harbor-integrations): record Harbor jobs in LangSmith.
* [Harbor task documentation](https://www.harborframework.com/docs/tasks): configure tasks, environments, and verifiers.

***

<div className="source-links">
  <Callout icon="terminal-2">
    [Connect these docs](/use-these-docs) to Claude, VSCode, and more via MCP for real-time answers.
  </Callout>

  <Callout icon="edit">
    [Edit this page on GitHub](https://github.com/langchain-ai/docs/edit/main/src/langsmith/managed-deep-agents-evals.mdx) or [file an issue](https://github.com/langchain-ai/docs/issues/new/choose).
  </Callout>
</div>