Harbor

Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. You can use Harbor to:

• Evaluate arbitrary agents like Claude Code, OpenHands, Codex CLI, and more.
• Build and share your own benchmarks and environments.
• Conduct experiments in thousands of environments in parallel through providers like Daytona and Modal.
• Generate rollouts for RL optimization.

Check out the Harbor Cookbook for end-to-end examples and guides.

About This Fork

This repository extends the open-source Harbor framework from the verifiable upstream baseline harbor-framework/harbor@3396e6f. The fork adds the following capabilities:

• Decoupled custom-agent runtimes for Claude Code, OpenCode, and OpenHands SDK. Prebuilt, self-contained runtime images are mounted read-only into task containers, skipping per-task agent installation, reducing time to inference, and isolating agent dependencies from heterogeneous task images.
• Phase-scoped network isolation with separate environment, agent, and verifier policies for no-network and allowlist-based evaluations.
• Automatic patch artifacts that capture the agent's complete Git diff as predict_patch.diff on a best-effort basis, including tasks that do not initially contain Git metadata. The pre-agent commit SHA is retained outside the task container so agent-side commits remain visible in the artifact.
• High-fidelity agent trajectories and local inference across supported agents. Harbor preserves native agent events alongside normalized per-call model requests and responses, providing auditable source data for SFT dataset preparation and detailed trajectory analysis. LiteLLM/vLLM launchers also support locally hosted inference.
• Reproducible and resumable jobs through tracked benchmark configurations, versioned agent runtimes, and recorded evaluation settings, with invalid-trial cleanup and config-drift-aware recovery.
• Extended Docker execution with image mounts, artifact mounts, bounded build concurrency, and host-readable output ownership.
• Evaluation analysis tooling for task and trajectory scoring, failure attribution, reward-hack signals, patch reconstruction, and runtime profiling.
• Additional benchmark adapters for OpenSWE and SWE-rebench-V2, including multi-language verification support.

See the Developer Guide for architecture, configuration, usage, limitations, and reproducibility details for each extension.

Installation

To use the fork-specific features, clone this repository and run it from the development environment:

git clone "<this-repository-url>"
cd harbor
uv sync --all-extras --dev
uv run harbor --help

The package commands below install the published upstream Harbor release and do not include this fork's unreleased extensions:

uv tool install harbor

or

pip install harbor

Example: Running Terminal-Bench-2.0

Harbor is the official harness for Terminal-Bench-2.0:

export ANTHROPIC_API_KEY="YOUR_API_KEY"
harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-4-1 \
   --n-concurrent 4 

This will launch the benchmark locally using Docker. To run it on a cloud provider (like Daytona) pass the --env flag as below:

export ANTHROPIC_API_KEY="YOUR_API_KEY"
export DAYTONA_API_KEY="YOUR_API_KEY"
harbor run --dataset terminal-bench@2.0 \
   --agent claude-code \
   --model anthropic/claude-opus-4-1 \
   --n-concurrent 100 \
   --env daytona

To see all supported agents, and other options run:

harbor run --help

To explore all supported third party benchmarks (like SWE-Bench and Aider Polyglot) run:

harbor datasets list

To evaluate an agent and model one of these datasets, you can use the following command:

harbor run -d "<dataset@version>" -m "<model>" -a "<agent>"

Citation

If you use Harbor in academic work, please cite it using the “Cite this repository” button on GitHub or the following BibTeX entry:

@software{Harbor_Framework,
author = {{Harbor Framework Team}},
month = jan,
title = {{Harbor: A framework for evaluating and optimizing agents and models in container environments}},
url = {https://github.com/harbor-framework/harbor},
year = {2026}
}