Bonus: Build an AI SRE with a handover agent

A glowing orange horizon under a black sky

You build two agents that watch a small live service: a cheap watcher that looks every 2 minutes, and a root-cause agent it hands off to when something looks wrong. Then you break the service and watch them find out why.

Time: about 45 minutes.

This is the local guide: Tares runs on your laptop. The cloud guide, for a Tares workspace we host, is added here before the workshop.

Before you start

Do the setup: Everyone, then Bonus. You need the tares-demo folder from it.
Docker is running.
You have your Anthropic API key at hand.

The demo service

The tares-demo folder starts three containers on your laptop:

api-server, the service you watch, on port 8080. It has three endpoints, /api/orders, /api/stats and /api/users, and a fault switch that ./inject.sh flips.
traffic-generator, which calls the three endpoints every second, so there is always something to look at.
Prometheus, on port 9090. It reads api-server’s metrics every 5 seconds and checks three alert rules.

What api-server gives out:

Metrics on /metrics: requests by endpoint and status, request time, active database connections, and whether its database (postgres) is up.
Logs: one line per request, in its Docker container’s output.
Alerts, fired by Prometheus: HighErrorRate (more than 5% of requests fail), HighLatency (p99 over 1 second) and DependencyDown (postgres down).

The fault switch has three settings, one per alert:

./inject.sh error_spike: 40% of requests return 500.
./inject.sh latency: every request takes 1.5 seconds.
./inject.sh dependency_outage: postgres is down and every request returns 503.
./inject.sh clear puts it back to normal.

How Tares sees it

The AI SRE demo template (step 5) creates three sources. Each one pulls one kind of data into Tares:

demo_metrics asks Prometheus every 5 seconds for four numbers: the 5xx rate, the p99 latency, active database connections and postgres up.
demo_logs reads the api-server container’s log every 5 seconds. It drops the normal 200 lines, so only errors come in.
demo_alerts asks Prometheus every 10 seconds which alerts are firing.

All three label their events service=api-server. That label is what joins them: Tares puts every event with it on one timeline, for the entity service=api-server. Your agents read that timeline, and so does Claude Code.

What you build

A wake-up (watch) that fires every 2 minutes and hands the timeline to an agent.
A watcher on a small, cheap model. Each time, it says “nothing to do” or “investigate”.
A root-cause agent on a bigger model. Nothing wakes it on a schedule: only the watcher’s “investigate” does. It writes what it finds onto the project page.

Steps

Start the demo service

In the tares-demo folder:


docker compose up -d
curl -s localhost:8080/api/stats

You should see {"ok":true,"endpoint":"/api/stats"}.

Start Tares

In a new terminal:


tares up

The console opens at http://127.0.0.1:8787 . Leave this terminal running.

By default Tares listens on port 8787 and keeps its data in ~/.tares.
Already have a Tares on 8787, or want a clean one? Pick another port and folder: tares up --port 8797 --data-dir ~/tares-workshop. Then add --taresd http://127.0.0.1:8797 to the command in step 3. Read 8797 wherever this guide says 8787.

Start the MCP endpoint for Claude Code

In another terminal:


tares mcp --transport streamable-http

It listens at http://127.0.0.1:8788/mcp. Leave it running.

Add your key

In the console:

Open Settings, then Model providers.
On the Anthropic row, click Edit, paste your key into API key, and click Save.

You should see the Anthropic row with status CONFIGURED.

Start the demo project

Open Projects, click Create new, and at the bottom, Start from a template.
On the AI SRE demo card, click Set up.
Tares finds your running demo by itself. You should see “Start the demo stack DONE”, and the Prometheus URL, api-server URL and Log container filled in. Click Start.

You land on the project page, with Cause an incident and Clear the fault at the top. This created the three sources above, plus a trigger and an agent of its own.

Turn off the agent the template ships

So that only your two agents write findings:

On the project page, click Setup.
In the 3. Agents card, click Change, then Disable.

You should see incident-first-look marked OFF. You are now in Advanced setup; the rest of the steps happen here.

Connect Claude Code to Tares

Connect it, and look at the timeline the agents will read. In a terminal:


claude mcp add --transport http tares http://localhost:8788/mcp

Then open claude and ask:


Use the tares MCP server. Read the last 10 minutes of the timeline of the entity service=api-server and tell me which metrics it carries and their current values.

You should see three metrics: p99 api-server (about 5 ms), db connections active (4 or 5) and dependency postgres up (1). The 5xx rate is missing because nothing fails yet.

Say “the entity service=api-server”. If you just say “the api-server timeline”, Claude Code may read the empty project timeline instead.

Create the root-cause agent first

In Advanced setup, click the + next to AGENTS, then:

name: rca-from-watcher
start from: Root cause after a handoff (this fills the prompt)
trigger: No trigger: only a handoff starts it
model: claude-sonnet-5

Click Create agent. You should see “Nothing starts it yet”. That is right: the watcher will start it.

Create the wake-up that runs every 2 minutes

Click the + next to TRIGGERS, then:

name: watch
in plain words: every 2 minutes, the watcher looks at api-server
sources: add demo_alerts, demo_logs and demo_metrics
entity label: service
fires: on a schedule
run every: 2m

Click Create trigger. You should see watch with “every 2m” and “no agent”.

Create the watcher

Click the + next to AGENTS, then:

name: watcher
trigger: watch
start from: Triage (on a schedule) (this fills the prompt and the verdict investigate)
at the end of the prompt, add the paragraph below. It tells the watcher what normal looks like for this service.
model: claude-haiku-4-5-20251001
under When it concludes, Add a handoff: verdict investigate, agent rca-from-watcher, cooldown 5m


For api-server, also read the last few demo_metrics lines: p99 api-server (normal about 5 ms), dependency postgres up (normal 1), db connections active (normal 4 to 5). A firing alert, a p99 above 50 ms, or postgres up at 0 is a problem even when no count grew.

Click Create agent. New agents start OFF: click Enable. You should see “watcher ON” and “investigate goes to rca-from-watcher · once per entity per 5m”.

Wait for one quiet run

Up to 2 minutes. Click Back to the project.

Under What the agents found you should see No action, “No anomalies detected”, with the numbers the watcher checked.

Break the service

In the tares-demo folder:


./inject.sh error_spike

From now on 40% of api-server’s requests fail. What follows, in order:

About 30 seconds later, Prometheus fires HighErrorRate, and the alert and the error log lines land on the timeline.
At the next 2-minute mark, the watcher reads the timeline, concludes investigate and hands off.
About 20 seconds after that, a Needs action finding appears under What the agents found, marked “watcher, then rca-from-watcher”. It says what is failing, since when, the likely cause and a next step.

From the inject to the finding takes 2 to 4 minutes.

Check the agents’ work with Claude Code

Ask:


Use the tares MCP server. What is wrong with the entity service=api-server right now?

Compare its answer with the finding: do both name the same failing endpoints, the same start time and the same cause?

What just happened

Tares put the service’s metrics, error logs and alerts on one timeline for service=api-server.
Every 2 minutes the watch wake-up handed that timeline to the watcher, a small, cheap model. Most runs it said “nothing to do”.
When the errors started, the watcher concluded investigate. That handoff woke rca-from-watcher on a bigger model, which read the evidence and wrote the root cause back onto the project page.
Claude Code read the same timeline over MCP, so you can check the agents’ work.

Use it on your own services

Nothing in the demo is special: it is three sources, one trigger and two agents. To watch a service of yours, swap the sources and keep the rest.

Add your sources. In the console, Sources → Add source, or ask Claude Code to set them up over MCP. Give every source the same entity label, for example service, so each service gets one timeline.
- Metrics: the Prometheus connector with your own PromQL, reading service from the metric’s labels.
- Logs: Docker logs per container, or OpenTelemetry if your services already send it.
- Alerts: Prometheus alerts or Alertmanager . Your alerting stays where it is; Tares reads what fires.
- Something else: any JSON endpoint with HTTP API , or anything that can POST with a webhook .
Add your deploys. Most real incidents start with a change, and the demo has none. Make a webhook source and add one curl to your deploy pipeline, so “deploy at 14:02, errors from 14:04” shows on the timeline.
Point the wake-up at them. Add your sources to watch (or make a new trigger with the same settings), and rewrite the watcher’s added paragraph with what normal looks like for your service.
Decide where findings go. Keep them on the project page, or send them to a Slack channel.

The AI SRE guide walks through each of these.

If something is off

Runs fail with “anthropic 401 … invalid x-api-key”. The key is wrong or revoked. You see it in Advanced setup → TRIGGERS → watch → Recent firings. Paste a working key in Settings → Model providers → Edit.
The project page says “Waiting for the first event” for logs and alerts. That is normal before you break anything: only error log lines and fired alerts come in.
No alert after 60 seconds. Check curl -s localhost:9090/api/v1/alerts. If it is empty, check the inject worked: curl -s localhost:8080/api/stats.
The watcher never concludes investigate. Open watcher and look at its last run. If it called the errors normal, make the added paragraph stricter, or try ./inject.sh dependency_outage.
The watcher concluded investigate but no root cause appears. Check the handoff verdict is spelled investigate in both places, and that the cooldown has passed (5 minutes per service).
The watcher never runs. Check it says ON and that watch is not PAUSED.
Cannot connect to the Docker daemon: start Docker Desktop, then step 1 again.
Clicking a + does not open the form. Click it again, or reload the page.

Try next: ./inject.sh latency and ./inject.sh dependency_outage. Run ./inject.sh clear between them, and wait 5 minutes for the cooldown.

Clean up

Stop the fault: ./inject.sh clear.
Turn the wake-up off: Advanced setup → TRIGGERS → watch → Pause. You should see PAUSED: “it is not evaluated and does not fire”. This stops the model calls.
Stop the demo: docker compose down in the tares-demo folder.
Stop Tares and the MCP endpoint with Ctrl+C in their terminals. Your data stays in ~/.tares.