Monitoring is built on thresholds, and a threshold needs an event to fire. It can tell you a backup failed. It cannot tell you a backup was never attempted — the policy quietly stopped covering a volume, the schedule was disabled during a migration, the job was removed with the host it protected. Nothing errors, no page fires, and the gap is found at restore time. Absence is not a louder alert. It is a different question — and like a frozen virtual desktop that reports itself healthy, or a pool that has quietly dropped below the seats you owe, it is a question that gets asked out of hours or not at all.
Seven nights. Six of them produced a job with a result — success or failure, either way a record. On the fourth, nothing was produced at all. There is no alert for that cell, because there is no event to attach one to.
The dashed cell is the incident. It is invisible to every threshold you have.
It also has to be safe to be wrong in the right direction. A failed read of the backup system means we could not look, not that nothing ran — so a guard that cannot see stays quiet and says why, rather than announcing at 03:00 that an estate is unprotected because a token expired. One wave of that and the guard gets switched off for good, which is worse than never having it.
A standard service window is around 45 hours. A week is 168. That leaves roughly 123 hours — about three quarters of the calendar — covered by an on-call rota, a phone, and the hope that whatever fires is worth waking someone for. In French service contracts this is the HNO window, heures non ouvrées, and it is where the cost and the risk of a managed service actually sit.
Ten hours inside the window. Fourteen outside it — and that is a weekday.
The alert already fires correctly at 03:00. What is missing is anything that reads it until the morning. An agent that is woken by the monitoring system rather than by a rota closes that gap without adding a night shift: it establishes what is actually true, acts inside an envelope that was agreed in advance, and leaves the customer's own ITSM carrying the observation, the action, the task id and the verification — so the morning starts with a record rather than an archaeology exercise.
VDI is the surface where the end customer's own users feel the service first, and the one where a status page is least likely to be right. A desktop estate does not degrade the way a server does.
Stopped, and every layer agrees. Easy to see and easy to fix — but at 02:00 nobody is looking at the dashboard that says so, which is the entire problem.
The hypervisor reports the guest as running while its guest agent does not answer. This is what a user calls frozen and what every dashboard above it calls healthy. Host-level monitoring cannot see it, because the host is fine.
No single desktop being unusable is an incident. The twelfth one is, because the pool can no longer seat its users. Nothing watching individual guests can see a number that only exists across all of them.
Liveness has to be checked from inside the guest, and that costs one call per desktop — so it is bounded, and it rotates. A check that always samples the first twenty-five desktops of four hundred reports a healthy estate no matter what is happening to the other three hundred and seventy-five, and reads exactly like a healthy estate. A bounded check that cannot reach most of its subject is worse than no check, because it is believed.
Every automation programme starts with the same request — write down what you do — and it fails for the same three reasons. The documentation is stale. The people who know are the people with no time. And in a programme whose stated purpose is reducing headcount, asking someone to describe their work has an obvious and entirely rational answer: describe less of it. Discovery that depends on goodwill is discovery built on the one thing the engagement removes.
So we start from evidence instead. Ticket timelines say what was requested and when it closed. Platform audit trails say what was actually executed against which system. Joining them reconstructs the runbook your team practises — including the steps nobody wrote down, and the ones the written procedure gets wrong.
Not the documented one. Each recurring request type, the actions actually taken, in the order they were actually taken, with the timings they actually took.
Every reconstruction is labelled: matched by actor and time window, matched by window only, unmatched — or unobservable, meaning the system in question emits no trail we can read. Unobservable is not the same as "no actions taken", and conflating the two is how automation programmes quietly encode a fiction.
Your engineers review a draft rather than author one. That is a materially different conversation to have with a team that has heard the word "automation" before.
Where procedure is missing or wrong, the reconstruction becomes the new document — and it is regenerated from evidence, so it stops being stale the moment it is written.
Nobody sensible gives an agent write access to a production storage estate on day one, and nobody should ask you to. Each loop moves through the same three stages independently, so the low-risk ones can be autonomous while the destructive ones are still shadowing.
Read-only. The agent builds the inventory, learns the expectations, reconstructs the runbook and reports what it would have done — against real events, with no ability to act.
The agent proposes on every real ticket, and a human decides. The disagreement rate is the number that matters, and it is measured, not asserted.
The agent acts within an explicit envelope, with destructive operations still gated, a stop that is checked on every call, and a signed record of what it did.
Promotion is earned per loop against a measured threshold — not scheduled, and not a judgement call made in a steering committee.
Infrastructure operations is not one job. It is a small number of loops that recur, each with its own rhythm, its own evidence and its own risk profile.
Agents act through the same APIs your engineers use, with credentials scoped per system and per customer. No new agent on your hosts, no new collector in the data path.
Enterprise NAS and scale-out platforms — volumes, quotas, snapshots, replication relationships, cluster health.
Backup policies, schedules, job history, retention, and restores that are validated before they are trusted.
Clusters, nodes, guests and desktop pools — power state, guest-agent liveness, task verification, and graceful power verbs only.
Problem feeds, host groups and metric history — as a source of truth about state, and as the thing that wakes an agent at 03:00 instead of a person.
Entity topology and problem detection above the infrastructure layer, for when the estate is healthy and the service is not.
Incidents, requests and changes: where the work arrives, where it is recorded, and where the agent stays accountable.
Least privilege is derived, not guessed: the permission specification for each system is generated from the operations the agent is actually allowed to perform, and drifts are caught when the two stop matching.
If you operate infrastructure for several end customers from one delivery team, the hard part is not the automation — it is the separation. Credentials, data, evidence and reporting stay partitioned per end customer, and the work an agent does is attributed to the customer it was done for. That attribution is what lets you show each of them what was delivered, and bill for it.
Point it at one platform, one desktop pool and one queue. In a few weeks you get the reconstructed runbook, the honest confidence level on each part of it, the protection gaps the absence guard found on your own estate, what the out-of-hours window actually contained, and a measured disagreement rate from shadowing. If the queue turns out to be less automatable than anyone hoped, you will see that too — and that is worth knowing before a programme is committed to.