Agent-based uptime probes

When to use a probe

Use a probe for anything only reachable from inside your network: an RDS endpoint, a Redis instance bound to a private IP, an internal API. For public URLs, the cloud check in Uptime monitors is simpler and does not depend on one of your servers being up.

Probes are TCP connect checks. The agent opens a connection to host:port, confirms it was accepted, and closes it without sending a byte. That is enough to know a port is up; it is not a protocol-level health check.

Step 1: opt the server in

Probes are off on every server until you enable them locally. Pick a stable server inside the network (a bastion is a good choice), set the value in its config and restart the agent:

OPSERVO_ENABLE_PROBES=true

sudo systemctl restart infrapilot-agent

The agent logs a warning on start that probes are enabled, and advertises this in its next heartbeat. The dashboard refuses to assign a check to a server that has not advertised it. Re-running the installer keeps the setting.

Step 2: create the monitor

1. 1.Open Uptime and add a monitor of type TCP port.
2. 2.In Check from, choose the server instead of *Opservo cloud*. The list shows every server; a server without probes enabled is rejected when you save, with a reminder of the config line above.
3. 3.Enter the private target as host:port and save.

A server can run at most 20 checks. Each runs once per heartbeat, about every 60 seconds; the monitor’s interval setting cannot make it faster. The per-check timeout is clamped between 0.5 and 10 seconds.

How results flow

• •Assignments ride on the heartbeat response. Each assignment is Ed25519-signed by the API over the monitor ID, the agent ID, the kind, the target and the timeout, so nobody with database access alone can point your server at a new target. The agent verifies the signature before dialing and reports a rejection instead of running an unsigned check.
• •Results come back in the server’s next heartbeat, so a change of state shows up within about a minute.
• •From there a probe result is an ordinary uptime result: the same up/down transitions, alerts, escalation, SLA maths and status page entries as a cloud check.
• •Creating or retargeting a probe monitor writes an audit entry.

When the probing server goes quiet

A probe is only as available as the server running it. If no result arrives for about 5 minutes (three missed heartbeats), the monitor flips to unknown with the message *Probe offline — the checking agent has stopped reporting*, and a warning notification goes out once. It never sits on a stale “up”. When the server reports again, the monitor picks up where it left off.

Turning probes off

Set OPSERVO_ENABLE_PROBES=false and restart the agent. Monitors assigned to that server will go to *Probe offline* after 5 minutes; edit them to check from another server, or delete them.