How to debug a service that keeps crashing and restarting (systemd, Docker, PM2)

You notice your app is slow, or a health check fails, and you dig in — only to find systemd has restarted your service 47 times in the last hour. The service is technically "running" right now, but it's in a crash loop. This is one of the most frustrating situations in production: everything looks fine at a glance, but something is quietly broken. Here's how to systematically find the cause and stop the cycle.

Step 1: Check the Service Status and Restart Count

Start with the basics. Run systemctl status yourservice to get an immediate snapshot. Look at two things: the "Active" line (it may say "active (running)" even mid-crash-loop) and the restart count. For a deeper picture, use:

• systemctl status yourservice — shows current state, PID, recent log lines, and restart info
• systemctl show yourservice | grep -E 'NRestarts|ExecMainStatus|ActiveState' — gives you the numeric restart count and last exit status
• cat /proc/$(systemctl show -p MainPID --value yourservice)/status — inspect the live process if it's currently up

The exit status is especially useful. An ExecMainStatus of 1 usually means the app itself exited with an error. A status of 143 means it received SIGTERM — possibly killed by the OOM killer or a watchdog. A status of 255 often means the binary couldn't start at all.

Step 2: Read the Logs — All of Them

This is where most debugging happens. The key is looking at logs across multiple crash cycles, not just the current run. journalctl is your best friend here:

• journalctl -u yourservice -n 200 — last 200 lines across all restarts
• journalctl -u yourservice --since '30 minutes ago' — time-bounded view
• journalctl -u yourservice -b -1 — logs from the previous boot if the crash took the whole system down
• journalctl -u yourservice | grep -i 'error\|fatal\|panic\|killed\|oom' — filter for the words that matter

Look for the last lines before each "Started" entry — that's the tail of a crash. Patterns like "out of memory" or "killed" point to resource exhaustion. Stack traces or "segmentation fault" mean a code-level bug. "Permission denied" or "no such file" means an environment problem — missing config, wrong paths, or a secret that didn't mount.

Step 3: Rule Out the OOM Killer

The Linux OOM killer silently terminates processes without leaving a clear message in your service's own logs. Check the kernel ring buffer directly:

• dmesg | grep -i 'oom\|killed process' — look for your service name or PID
• grep -r 'oom_kill' /var/log/ — catches it in syslog on older systems
• journalctl -k | grep -i oom — kernel messages via journald

If the OOM killer is responsible, you'll see a line like "Out of memory: Kill process 12345 (yourservice) score 800". The fix might be tuning memory limits in your systemd unit file (MemoryMax=), optimizing the application's memory usage, or simply adding more RAM. If you're on a small VPS, also check whether other services are competing for the same memory pool.

Step 4: Check for Environment and Dependency Problems

A surprising number of crash loops aren't bugs — they're configuration problems. The service starts, can't find something it needs, and exits immediately. Check these in order:

• Missing environment variables: add ExecStartPre=/usr/bin/env to your service unit temporarily to dump the environment at start
• Database or upstream not ready: use Requires= and After= in your unit file to enforce start order, or add a retry loop in your app
• Missing files or wrong permissions: run sudo -u yourserviceuser ls /path/to/config to test access as the actual service user
• Library linking errors: run ldd /usr/bin/yourservice and look for 'not found' entries

It's also worth temporarily disabling the restart policy while debugging. Set Restart=no in your unit file and run systemctl daemon-reload, then start the service manually with systemctl start yourservice. This stops the loop so you can read the full output without it racing to restart.

Step 5: Reproduce It Outside systemd

Once you've narrowed down the cause, try running the service binary directly in a terminal as the same user systemd uses. Switch to that user with sudo -su yourserviceuser, set the same environment variables from the unit file's Environment= lines, and run the binary with the same arguments. This makes errors immediately visible in your terminal rather than buried in logs. If it works perfectly on the command line but fails under systemd, the issue is almost certainly in the unit file itself — missing environment, wrong working directory (set with WorkingDirectory=), or a sandboxing option like ProtectSystem= or NoNewPrivileges= blocking a file access.

Making This Easier to Catch Early

Crash loops are painful partly because they often start quietly — a service restarts once or twice and nobody notices until it's restarted hundreds of times and caused real user impact. Setting up alerting on NRestarts or on failed service state changes in your monitoring means you catch this in the first few cycles, not after the fact.

This is exactly the kind of pattern Opservo is built to surface — it monitors systemd service state across your servers, explains what the logs mean in plain English, and flags crash loops as they begin rather than after they've been running for hours. For small teams without a dedicated SRE, that kind of automatic context can cut the time from "something's wrong" to "here's why" significantly.

The fastest path to fixing a crash loop: find the last lines before each restart, check for OOM kills in dmesg, and reproduce the failure outside systemd where you can see everything.

Crash loops feel chaotic but they're almost always deterministic — the same thing is failing every time. Work backwards from the exit code, read across multiple crash cycles in the logs, and rule out environment and resource issues before assuming it's a code bug. Most of the time, the answer is already in the logs.