What an AI agent needs to investigate an incident

Checkout is slow. The alert says so, and nothing else. The cause is somewhere between a query plan, a log file and a container's memory graph - three systems, usually three consoles, often three ways in.

An AI agent can do this legwork well. It is patient, it reads logs faster than you do, and it doesn't mind running the fourth EXPLAIN of the afternoon. What stops it is rarely the model. It is everything around it.

Six things the agent needs

1. A way in to each system. The database, the log host and the Docker host usually sit behind a firewall. If the agent can't reach them, nothing else matters.

2. A way to find the right system. "Checkout" is a name in your head, not a hostname. The agent needs to turn it into the database, the log directory and the container that belong to it - and to ask when the match isn't obvious.

3. Read access that stays read access. Investigation is reading. A tool that can also UPDATE will eventually be asked to, by a confident model fixing the wrong thing. Read-only should be the default, enforced somewhere the model can't talk its way past.

4. Bounded output. A log file can be gigabytes and a table millions of rows. The agent needs paging, search and limits, or it fills its context with noise and loses the thread.

5. Errors that say what to do next. "Permission denied" leaves the agent guessing. "This connection is read-only; use a writer role" lets it stop or ask.

6. A record of what it did. When the agent says "an index was dropped", someone will want to see the query that showed it. Every call should land in an audit log, marked as the agent's.

Five ways to give it that

Give it SSH

The fastest start, and the riskiest. An agent with a shell on the box can read the logs, run psql, check docker stats - and also run anything else the account can. Four problems come with it:

• Read-only is up to the model. Nothing in a shell separates "look" from "change". The agent stays read-only only as long as it chooses to.
• What it reads can steer it. A log line or a web page the agent reads can contain text that looks like an instruction. With a shell, a bad instruction becomes a command.
• The key travels. The private key, and often database passwords, end up wherever the agent runs.
• The trail says "you". If the agent uses your account, the logs show you ran it.

If you do it anyway, narrow it. A dedicated account with no sudo. A key in authorized_keys with restrict - which, per OpenSSH's sshd manual, disables port, agent and X11 forwarding - and command="…", which runs that one command "whenever this key is used for authentication", ignoring whatever the client asked for. At that point you have built a very small gateway by hand.

Paste things into the chat

Copy the error, the slow query and the docker stats output into the conversation. It works, it needs no setup, and a person decides what the model sees. It is also slow, you are the transport, and the agent can't follow a lead on its own.

Good for a one-off. Not for an incident at 3am.

One MCP server per system

There are open-source MCP servers for most databases and for Docker. Some take read-only seriously: Postgres MCP Pro has a restricted mode that, in its README's words, "Limits operations to read-only transactions and imposes constraints on resource utilization". Worth knowing - its README's config examples use --access-mode=unrestricted, so you switch restricted on.

The cost is in the plumbing. Each server needs its own network path to its system, its own copy of credentials, and its own idea of an audit log. Five systems is five of each.

Your observability platform's MCP server

If your telemetry already flows into an observability platform, its MCP server is the fastest start. Datadog describes its server as a bridge "between your observability data in Datadog and any AI agents". Grafana's mcp-grafana is open source under Apache-2.0.

The limit is in that phrase: your observability data. The agent reads what you already shipped there - metrics, logs, traces - not the live system. It can see that latency rose; it can't run EXPLAIN ANALYZE on your database.

A gateway to the systems themselves

The last option puts one service between the agent and every system, so the agent gets one connection and the systems keep their credentials. This is what we are building with Levirge DevOps - read that as a disclosure, not a neutral recommendation. A small client runs next to the systems and connects outward, so nothing needs an inbound port; writes need a role, checked at both ends; and the audit log marks the agent's actions. It is in private testing, so for now it is a direction, not something you can install.

Whatever you choose

Some rules hold regardless of the tool:

• Make the database enforce read-only too. Point the agent at a replica or a database user with read permissions only. A tool's read-only mode is a second lock, not the first.
• Set limits. Query timeouts, row limits, log read sizes.
• Give the agent its own identity. Not your token. You want "the agent did this" in the log, not "you did this at 3am".
• Start with read. Let it investigate for a few weeks before you consider letting it change anything.

The model is the easy part. Give it a way in, a way to find things, a safe default and a paper trail, and it does the legwork while you decide what to do about it.

More on the gateway approach: Give AI agents the ability to monitor your infrastructure and Let AI agents query your production database, read-only.