How to safely auto-restart a downed service
Auto-healing is great until it masks a real problem or melts your CPU in a restart loop. Here’s how to do it without shooting yourself in the foot.
By The Opservo team
Automatically restarting a service that’s gone down is one of the highest-value automations you can run — and one of the easiest to get wrong. Done carelessly, it hides a crash that needs fixing, or hammers a broken process in a tight restart loop. Here’s how to do it safely.
Rule 1: Always leave a trail
An automatic restart should never be silent. Every time the policy fires, record what happened — which service, when, and the result. If a service is quietly restarting ten times a day, that’s not “healthy,” that’s a bug you need to see.
Rule 2: Cooldowns, always
A service that won’t stay up will, without a cooldown, get restarted in an infinite loop — burning CPU and filling logs. A cooldown window between attempts turns a runaway loop into a bounded, observable retry.
Automation without a cooldown and an audit log isn’t self-healing — it’s a hidden incident.
Rule 3: Opt-in, per box
Auto-remediation should be a deliberate choice for each server, not a global default that sweeps your whole fleet. The servers where an automatic restart is safe are usually not the same ones where it’s risky.
How Opservo does it
Opservo’s autonomous policies follow exactly these rules. You pre-approve a fix — “if this service is down, restart it” — per server or fleet-wide. Each policy has a built-in cooldown so it can’t loop, and every run is written to a permanent log you can review. It stays dashboard-only and read-only by default, so automation is always something you turn on deliberately, never something that happens to you.