Auto Scaling groups and golden AMIs

The problem with baking an installed agent into an image

The agent’s identity lives in /etc/infrapilot/config. Bake an AMI from an installed server and every instance launched from it boots with the original server’s ID and key. They all report as one server: a hostname that flaps, interleaved metrics, meaningless alerts. And the normal install token from Add server is single-use and expires in an hour, so an instance launched next week has nothing to register with.

Fleet keys

A fleet key (fleet_…) is a reusable, long-lived install token for launch templates and user data. Only owners and admins can create or revoke one, and both are audited.

1. 1.Open Add server and click Auto Scaling group or a golden AMI? Set up automatic enrollment.
2. 2.Click Create fleet key. The dialog shows the key once, with a ready-made user-data script.
3. 3.Paste the script into the launch template’s user data. Every instance installs the agent on first boot and registers as its own server:

#!/bin/bash
# Opservo — installs the monitoring agent when the instance first boots.
curl -sSL https://api.getopservo.com/install.sh | sudo env OPSERVO_API_URL=https://api.getopservo.com OPSERVO_TOKEN=fleet_… bash

The same dialog lists existing keys with Revoke. Revoking stops new registrations immediately; servers already registered are unaffected. Your plan’s server limit is checked when each instance registers, not when the key is created.

Clone detection

At install time the installer records the machine’s identity in the config as OPSERVO_MACHINE_ID. It prefers the cloud instance ID (EC2 instance metadata, then the DMI board_asset_tag) and falls back to /etc/machine-id — because baking an image preserves machine-id, so every clone would share it.

On every start the agent recomputes the identity and compares it with the recorded one. If they differ, the box is a clone:

• •If the config holds a fleet key (OPSERVO_ENROLL_KEY, written by the installer only when you installed with a fleet_ token), the agent drops the inherited credentials, registers as a new server, and rewrites the config with the new identity. The fleet key stays, so the next generation of clones does the same.
• •If there is no fleet key, the agent logs a warning and keeps the inherited identity. Fix it by creating a fleet key and re-running the installer on the image, or by baking images before installing the agent.

If either identity is unknown (an older installer, metadata that cannot be read), nothing happens; a false alarm would create duplicate servers.

The safest order is: bake the image without the agent and install from user data with a fleet key. Baking from a server that was installed with a fleet key also works, because clones detect themselves on first boot.

Scale-in: automatic archiving

When an Auto Scaling group terminates an instance, that server stops reporting. A fleet-enrolled server that has been offline for 30 minutes is archived automatically: its open alerts are resolved and an audit entry server_auto_archived is written. Archived servers:

• •move into a collapsed Archived servers section at the bottom of the fleet table, with a Restore button;
• •do not count toward your plan’s server limit, per-server billing or AI credit grants;
• •come back on their own if the server sends another heartbeat, so a long network partition is not mistaken for a termination.

Servers installed by hand are never archived automatically. For them, silence is an outage, and the offline alert fires as usual. You can archive any server manually from the fleet table, and restore it later.