Every network team that has put an agent in front of Cisco gear hits the same wall. The agent reads beautifully — it parses show ip bgp summary, correlates syslog, catches the flapping adjacency — and then proposes a two-line configuration change that nobody will let it apply. Diagnosis was never the bottleneck. The write is.
The read side of Cisco is solved; the write side is where remediation dies
The read path has been engineered for a decade: streaming telemetry, NETCONF, gNMI, SNMP polling that finally works. The write path is a different risk class. A bad read produces a bad dashboard. A bad write produces a dark site. In practice, teams approve the read-only agent in a week and stall the write-capable version for roughly two quarters, and the blocker is almost never the model's reasoning. It is that nobody on the call can answer the only question that matters: how do we undo this in under sixty seconds at 3 a.m.?
Blast radius is topology depth, not command count
Consider one line: switchport trunk allowed vlan remove 42. On a single-homed access port, it is a non-event. On a port-channel trunking 1,100 VLANs between a pair of Catalyst 9500s, it can take a campus offline — and the diff looks identical to a parser either way. Command count is a terrible risk proxy; one line can carry more blast radius than forty.
Score risk on the variables that actually determine it: device role (core, distribution, access), current redundancy state (is the peer genuinely up right now?), interface criticality, active session count, and whether the change is additive or subtractive. Subtraction and shutdown are categorically more dangerous than addition. A remediation engine that scores by diff size is scoring the wrong variable.
Three pre-flight gates every Cisco write must pass
- Archive plus revert timer. On IOS and IOS-XE,
configure terminal revert timer 5combined witharchiveandconfigure replacegives you a device-side undo that fires even if the session dies mid-change. It is the highest-leverage primitive on the platform, and most tooling ignores it entirely. - Dry-run diff. Render the intended config, compare it against a freshly pulled running-config, and fail closed on any line the engine did not predict. Unexpected lines mean your model of the device is stale — stop.
- AAA attribution. Every pushed command lands in TACACS+ command accounting under a named, per-job service account. If you cannot attribute a change to a specific remediation run, you cannot audit it, and if you cannot audit it you should not have write access.
| Gate | Mechanism | Fail condition |
|---|---|---|
| Revert | revert timer + archive | timer unsupported or archive missing |
| Diff | running-config vs rendered | unmodeled line present |
| Attribution | TACACS+ command accounting | shared or anonymous account |
A remediation you cannot reverse in one minute is not automation — it is a change request with a faster keyboard.
Remediate the platform, not the text file
Cisco is four CLI dialects behind one badge: IOS, IOS-XE, NX-OS and IOS-XR. The same intent — set the OSPF cost on an interface, drain a link, tighten an ACL — is expressed differently on each, and YANG coverage is uneven across families and versions. Teams that build remediation as string templates against show running-config end up maintaining four dialects times every intent, and that maintenance cost grows faster than the fleet does. Teams that model intent once and compile it per family — pyATS/Genie parsers where they exist, YANG models where they are complete, structured CLI otherwise — maintain one copy per intent.
Idempotency is the other trap. no on something already absent returns an error on some platforms and success on others. Your engine has to treat "already in desired state" as a successful no-op rather than a failed command, or your success metrics become noise and your on-call stops trusting the run log.
Measure remediation in exposure minutes, not tickets closed
Run your own arithmetic: monthly drift and fault incidents, times mean time-to-fix, times the fraction of affected devices that are single-homed or sitting outside a maintenance window. That product is your exposure. Ticket counts flatter you; exposure minutes do not. The gains concentrate in the tail — the 3 a.m. BGP flap where the on-call spends forty minutes confirming what the agent already knew in four.
Where this is heading
The next step is not a smarter agent, it is a narrower write surface. Change-calendar-aware remediation that refuses to touch a device inside a freeze. Blast-radius simulation driven by live telemetry rather than a static topology diagram someone drew last year. Pre-approved, role-scoped playbooks, so an access switch gets a different autonomy level than a core pair. On-box EEM applets as a last-resort guardrail that reverts configuration if the control plane goes quiet. And rollback promoted from exception handling to the default posture of every automated change.
The teams that get this right will not describe their remediation platform as intelligent. They will describe it as boring: small, reversible, attributed, and timed.