langsmith-cli

An agent-first CLI for querying and managing LangSmith resources.

Built for AI coding agents (deepagents, Claude Code, Cursor, etc.) and developers who need fast, scriptable access to projects, traces, runs, datasets, evaluators, experiments, and threads.

Installation

Install script (recommended)

macOS / Linux:

curl -fsSL https://cli.langsmith.com/install.sh | sh

Windows (PowerShell):

irm https://cli.langsmith.com/install.ps1 | iex

Upgrade

langsmith self-update

GitHub releases

Download the latest binary for your platform from GitHub Releases.

Authentication

Set your API key as an environment variable:

export LANGSMITH_API_KEY="lsv2_pt_..."

Optionally set defaults:

export LANGSMITH_ENDPOINT="https://api.smith.langchain.com"  # For self-hosted
export LANGSMITH_WORKSPACE_ID="<workspace-id>"                # Default workspace
export LANGSMITH_PROJECT="my-default-project"                 # Default project for queries

Or pass them as flags:

langsmith --api-key lsv2_pt_... --workspace <workspace-id> trace list --project my-app

Quick Start

# List tracing projects
langsmith project list

# List recent traces in a project
langsmith trace list --project my-app --limit 5

# Get a specific trace with full detail
langsmith trace get <trace-id> --project my-app --full

# List LLM calls with token counts
langsmith run list --project my-app --run-type llm --include-metadata

# List datasets
langsmith dataset list

# List experiments for a dataset
langsmith experiment list --dataset my-eval-set

Output Formats

langsmith trace list --project my-app

langsmith --format=json trace list --project my-app

langsmith trace list --project my-app -o traces.json

Command Reference

project — List tracing projects

A tracing project (session) is a namespace that groups related traces together. This lists only tracing projects, not experiments — use experiment list for those.

Results are paginated — by default, only the first 20 projects are returned (use --limit to change). Projects are sorted by most recent activity (last_run_start_time, descending).

# List tracing projects (default: 20 results, most recently active first)
langsmith project list
langsmith project list --limit 50

# Filter by name
langsmith project list --name-contains chatbot

# Machine-readable JSON
langsmith --format=json project list

trace — Query and export traces

A trace is a tree of runs representing one end-to-end invocation of your application.

Results are paginated — by default, only the first 20 traces are returned (use --limit to change). Traces are sorted newest-first by start time. By default, only traces from the last 7 days are returned; use --since or --last-n-minutes to change the time window.

# List recent traces (default: 20 results, newest first)
langsmith trace list --project my-app
langsmith trace list --project my-app --limit 50 --last-n-minutes 60

# Filter traces
langsmith trace list --project my-app --error           # Only errors
langsmith trace list --project my-app --min-latency 5   # Slow traces (>5s)
langsmith trace list --project my-app --tags production  # By tag
langsmith trace list --project my-app --name "agent"     # By name

# Include additional fields
langsmith trace list --project my-app --include-metadata   # + status, duration, tokens, costs
langsmith trace list --project my-app --include-io         # + inputs, outputs, error
langsmith trace list --project my-app --include-feedback   # + feedback_stats
langsmith trace list --project my-app --full               # All fields (metadata + io + feedback)

# Show trace hierarchy (fetches full run tree for each trace)
langsmith trace list --project my-app --show-hierarchy --limit 3

# Get a specific trace
langsmith trace get <trace-id> --project my-app --full

# Export traces to JSONL files (one per trace)
langsmith trace export ./traces --project my-app --limit 20 --full

# Custom filename pattern (supports {trace_id} and {name} placeholders)
langsmith trace export ./traces --project my-app --filename-pattern "{name}_{trace_id}.jsonl"

run — Query individual runs

A run is a single step within a trace (LLM call, tool call, chain step, etc.).

Results are paginated — by default, only the first 50 runs are returned (use --limit to change). Runs are sorted newest-first by start time. By default, only runs from the last 7 days are returned; use --since or --last-n-minutes to change the time window.

# List LLM calls (default: 50 results, oldest first)
langsmith run list --project my-app --run-type llm
langsmith run list --project my-app --run-type tool --name search

# Find expensive calls
langsmith run list --project my-app --run-type llm --min-tokens 1000 --include-metadata

# Include feedback scores
langsmith run list --project my-app --include-feedback

# Get a specific run
langsmith run get <run-id> --full

# Export to JSONL (default limit: 100)
langsmith run export llm_calls.jsonl --project my-app --run-type llm --full

For agents querying runs: prefer --version v2 first (SmithDB-backed; faster on tenants that are rolled out). If the call fails with a 4xx (typically 403, 404, or 422), retry the same command without --version to fall back to v1. Example:

langsmith run list --project my-app --version v2 \
  || langsmith run list --project my-app

thread — Query conversation threads

A thread groups multiple root runs sharing a thread_id (multi-turn conversations).

Results are paginated — by default, only the first 20 threads are returned (use --limit to change). Threads are sorted by most recent activity (newest first).

# List threads (default: 20 results, newest first; requires --project)
langsmith thread list --project my-chatbot
langsmith thread list --project my-chatbot --last-n-minutes 120

# Get all turns in a thread
langsmith thread get <thread-id> --project my-chatbot --full

dataset — Manage evaluation datasets

List results are paginated — by default, only the first 100 datasets are returned (use --limit to change).

# List datasets (default: 100 results)
langsmith dataset list
langsmith dataset list --name-contains eval

# Get dataset details
langsmith dataset get my-dataset

# Create and delete
langsmith dataset create --name my-eval-set --description "QA pairs for v2"
langsmith dataset delete my-old-dataset --yes

# Export examples to JSON
langsmith dataset export my-dataset ./data.json --limit 500

# Upload from JSON file
langsmith dataset upload data.json --name new-dataset

example — Manage dataset examples

List results are paginated — by default, only the first 20 examples are returned (use --limit to change). Use --offset to paginate through results.

# List examples (default: 20 results)
langsmith example list --dataset my-dataset
langsmith example list --dataset my-dataset --split test --limit 50

# Paginate through examples
langsmith example list --dataset my-dataset --limit 20 --offset 20

# Create an example
langsmith example create --dataset my-dataset \
  --inputs '{"question": "What is LangSmith?"}' \
  --outputs '{"answer": "A platform for LLM observability"}'

# Create with metadata and split assignment
langsmith example create --dataset my-dataset \
  --inputs '{"question": "What is tracing?"}' \
  --outputs '{"answer": "Recording LLM application execution"}' \
  --metadata '{"source": "manual", "version": 2}' \
  --split test

# Delete an example
langsmith example delete <example-id> --yes

evaluator — Manage evaluator rules

# List evaluators
langsmith evaluator list

# Upload an offline evaluator (for experiments)
langsmith evaluator upload evals.py \
  --name accuracy --function check_accuracy --dataset my-eval-set

# Upload an online evaluator (for production monitoring)
langsmith evaluator upload evals.py \
  --name latency-check --function check_latency --project my-app

# Set sampling rate (evaluate a fraction of runs, 0.0-1.0)
langsmith evaluator upload evals.py \
  --name latency-check --function check_latency --project my-app --sampling-rate 0.5

# Replace an existing evaluator
langsmith evaluator upload evals.py \
  --name accuracy --function check_accuracy_v2 --dataset my-eval-set --replace --yes

# Delete an evaluator
langsmith evaluator delete accuracy --yes

experiment — Query experiment results

List results are paginated — by default, only the first 20 experiments are returned (use --limit to change).

# List experiments (default: 20 results)
langsmith experiment list
langsmith experiment list --dataset my-eval-set

# Get experiment results (feedback stats, run stats)
langsmith experiment get my-experiment-2024-01-15

hub — Manage agent and skill repos on the LangSmith Hub

The hub stores versioned directories of files grouped into repos of type agent or skill. Each push creates a new commit; pull downloads a commit's files into a local directory. This is the CLI surface for the langsmith Python/JS SDK's hub methods (pull_skill, push_skill, pull_agent, push_agent, etc.).

# Scaffold a starter skill (or agent)
langsmith hub init --type skill --dir ./my-skill --name my-skill

# Push a local directory as a new commit (creates the repo if missing)
langsmith hub push my-skill --type skill --dir ./my-skill

# Pull a commit (latest by default; pin a tag with :ref)
langsmith hub pull my-skill --dir ./out
langsmith hub pull acme/my-skill:production --dir ./out

# Discover, inspect, delete
langsmith hub list --type skill --query foo
langsmith hub list --type skill --source external
langsmith hub get acme/my-skill
langsmith hub delete acme/my-skill --yes

Identifiers use [OWNER/]REPO format. Omitting owner defaults to - (the API's "current tenant" wildcard).

Push excludes .git/, node_modules/, __pycache__/, .venv/, dist/, build/, target/, .next/, .cache/, plus .env* files, common secret extensions (.pem, .key, .pfx, .p12, .crt), and rejects binary or oversize (>1 MiB) files. Pull wipes the destination dir before writing; non-empty directories without a SKILL.md/AGENTS.md marker require --yes.

self-update — Update langsmith to the latest version

# Check if an update is available
langsmith self-update --dry-run

# Update to the latest version
langsmith self-update

If langsmith was installed through a package manager, self-update won't replace the binary in place — it points you at the right command instead:

Installed via	Update with
Homebrew	brew upgrade langchain-ai/tap/langsmith-cli
Scoop	scoop update langsmith-cli
go install	go install github.com/langchain-ai/langsmith-cli/cmd/langsmith@latest

Installs from the install.sh/install.ps1 scripts or a direct GitHub Releases download are updated in place as usual. Pass --force to update in place regardless of how langsmith was installed.

trace setup — Trace coding agents to LangSmith

Configure Claude Code or Codex to send full-content traces (prompts, responses, tool calls) to a LangSmith project, by writing the agent's local config files. Requires an API key — it is written to the agent config at 0600 (OAuth profiles are not supported here). Each command previews the exact changes and asks you to confirm (pass --yes to skip the prompt), then installs the plugin via the agent's own CLI.

# Bare: try both Claude Code and Codex (best-effort; an uninstalled agent just fails)
langsmith trace setup

# Configure Claude Code: API key, URL, and project as positional args (bare host gains https://)
langsmith trace setup claude demo-key dev.smith.com shared-claude

# Or take the key + URL from env/profile
langsmith trace setup claude

# Configure Codex (writes ~/.codex/config.toml + ~/.codex/langsmith.json)
langsmith trace setup codex

# Trace to a named project (default: "claude-code" / "codex", or $LANGSMITH_PROJECT)
langsmith trace setup claude --project my-agent

# Override the auto-detected name/email attached to every trace
langsmith trace setup claude --user "Jane Doe" --email jane@example.com

# Pass everything explicitly (self-hosted or a specific workspace key)
langsmith trace setup claude demo-key https://my-host/api/v1 my-team   # all positional

# Apply without the interactive confirmation prompt
langsmith trace setup claude --yes

# Write config only; skip running the plugin install
langsmith trace setup claude --no-install

# Write project-local config instead of user-global
langsmith trace setup claude --scope project    # ./.claude/settings.local.json
langsmith trace setup codex --scope project     # ./.codex/...

trace setup claude installs the plugin via claude plugin marketplace add + claude plugin install; trace setup codex fetches it via codex plugin marketplace add. Once enabled, the plugin runs on every session and sends your prompts, responses, and tool output to LangSmith. Your name and email (auto-detected from git config user.name/user.email, or set via --user/--email) are attached to every trace as user_name/user_email metadata. Verify Claude Code with tail -f ~/.claude/state/hook.log.

Filter Options

Most trace and run commands share these filter options:

Flag	Description	Example
--project	Project name	--project my-app
--limit, -n	Max results	-n 10
--last-n-minutes	Time window (overrides 7-day default)	--last-n-minutes 60
--since	After ISO timestamp (overrides 7-day default)	--since 2024-01-15T00:00:00Z
--error / --no-error	Error status	--error
--name	Name search (case-insensitive)	--name ChatOpenAI
--run-type	Run type (run commands only)	--run-type llm
--min-latency	Min latency (seconds)	--min-latency 2.5
--max-latency	Max latency (seconds)	--max-latency 10
--min-tokens	Min total tokens	--min-tokens 1000
--tags	Tags (comma-separated, OR logic)	--tags prod,v2
--filter	Raw LangSmith filter DSL	--filter 'eq(status, "error")'
--trace-ids	Specific trace IDs	--trace-ids abc123,def456

Local Development

For local dev, create a wrapper script at ~/.local/bin/langsmith that loads your .env and uses go run:

cat > ~/.local/bin/langsmith << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd /path/to/langsmith-cli
set -a && source .env && set +a
exec go run ./cmd/langsmith "$@"
EOF
chmod +x ~/.local/bin/langsmith

Ensure ~/.local/bin is in your PATH before ~/go/bin. This way commands like langsmith sandbox list and SSH ProxyCommand entries work without manually sourcing .env each time.

Requirements

• Go 1.23+
• golangci-lint (for linting)

License

MIT