Diagnostics

The diagnostics system provides structured event logging and real-time streaming across seven channels for debugging and monitoring agent behavior.

Overview

Diagnostics captures internal agent events and delivers them through two subscriber types:

• DB subscriber — Persists events to the diagnostics table for historical queries
• WebSocket subscriber — Streams events in real time to connected clients

Channels

Events are organized into seven channels:

Channel	Description
agent.state	State transitions, hibernation, wake-up, connection changes
rpc	Incoming and outgoing RPC calls, including callable methods
message	Inbound and outbound messages across all gateways
schedule	Schedule execution, overlap detection, stale-run recovery
lifecycle	Agent creation, deletion, alarm triggers, error recovery
workflow	Workflow step execution, variable mutations, loop iterations
mcp	MCP server connections, tool calls, resource reads

Event Structure

Each diagnostic event includes:

{
  "id": "evt_abc123",
  "channel": "message",
  "timestamp": "2026-03-11T10:30:00Z",
  "level": "info",
  "summary": "Inbound message from Telegram",
  "data": {
    "gateway": "telegram",
    "from": "+1234567890",
    "length": 142
  }
}

Subscribing to Events

Via WebSocket

Connect to the agent WebSocket and subscribe to diagnostics:

{
  "type": "subscribe",
  "channels": ["diagnostics"]
}

Events arrive as WebSocket messages:

{
  "type": "diagnostic",
  "channel": "schedule",
  "level": "warn",
  "summary": "Schedule overlap detected",
  "data": { "schedule_id": "sched_456", "running_since": "2026-03-11T10:25:00Z" }
}

Filtering by Channel

Subscribe to specific channels only:

{
  "type": "subscribe",
  "channels": ["diagnostics"],
  "filter": {
    "diagnosticChannels": ["agent.state", "mcp"]
  }
}

Querying Historical Events

GET /api/agents/{id}/diagnostics?channel=workflow&limit=50

Response:

{
  "events": [
    {
      "id": "evt_789",
      "channel": "workflow",
      "timestamp": "2026-03-11T10:28:00Z",
      "level": "info",
      "summary": "Workflow step completed: send-email",
      "data": { "workflow_id": "wf_1", "step": 3, "duration_ms": 420 }
    }
  ]
}

Dashboard

Navigate to Diagnostics in the sidebar to access the diagnostics UI.

The page provides:

• Channel filter — Toggle channels on/off to focus on relevant events
• Level filter — Filter by debug, info, warn, error
• Live stream — Real-time event feed via WebSocket
• Search — Full-text search across event summaries and data
• Timeline — Chronological view with expandable event details

Use Cases

Debugging Agent Behavior

Monitor the agent.state and message channels to trace how the agent processes a conversation:

1. Filter to agent.state to see when the agent wakes and what triggers it
2. Switch to message to see inbound/outbound messages
3. Check rpc for any callable method invocations

Monitoring Schedules

Track schedule execution with the schedule channel:

• Verify schedules fire at expected times
• Detect overlapping runs
• Identify stale runs that exceeded the 5-minute timeout

Tracking MCP Calls

Use the mcp channel to debug MCP server integrations:

• Tool discovery requests and responses
• Tool call arguments and results
• Connection failures and retries

Workflow Troubleshooting

The workflow channel logs each step:

• Step start/complete with duration
• Variable mutations and loop state
• Error details when steps fail

Best Practices

1. Use Channel Filters

Avoid subscribing to all channels in production. Filter to the channels relevant to your investigation to reduce noise.

2. Set Appropriate Levels

Use warn and error levels for alerting. Reserve debug for active troubleshooting sessions.

3. Monitor Lifecycle Events

The lifecycle channel captures critical events like error recovery and alarm triggers that may indicate systemic issues.

4. Correlate Across Channels

Many operations span multiple channels. An inbound message (message) may trigger a workflow (workflow) that calls an MCP tool (mcp). Use timestamps to correlate events across channels.