5 Linux Production Incidents That Look Simple — And Destroy Senior Engineers

It’s 2:47am. Your phone wakes you up.

The alert says load average has been above 18 for fifteen minutes. You open your laptop. CPU is at 24%. Memory is fine. The service is running. Everything looks fine.

And yet — users are timing out, the business is bleeding, and the dashboard is lying to you.

The most dangerous production incidents are not the ones where everything is obviously on fire. They are the ones where everything looks fine — and the root cause is hiding in a completely different layer than the symptom.

Here are five incident patterns that can derail senior interviews, stretch experienced engineers, and turn routine on-call work into a long outage. Every one has a symptom pointing somewhere — and a cause hiding somewhere else entirely.

Incident 1 — Load Average of 18. CPU at 24%.

Most Linux engineers are trained to associate high load with high CPU. That mental model is incomplete — and it can make you freeze in production.

Load average counts two things: processes actively using CPU, and processes blocked waiting on something else. When your wa column in vmstat is at 56% and you have 19 blocked processes, CPU utilization is not the primary problem.

Run vmstat 1 5. Look at the b column — that’s your blocked process count. Then look at wa — that’s your I/O wait percentage. If wa is above 30% with 10 or more blocked processes, disk I/O is a strong bottleneck candidate.

# Run these three commands in order
vmstat 1 5          # look at: b (blocked), wa (iowait)
iostat -xz 1 5      # find the device at 99%+ utilization
pidstat -d 1 5      # name the process doing the damage

In Lab LNX-01 from Infrastructure War Room Pro, a report-writer process is pushing 91,000 kB/s of writes to the root volume. The disk is saturated. The application sitting on the same volume cannot write normally. Load spikes to 18.72. CPU stays at 24%. Users see timeouts.

The safe response is to collect evidence first, then pause or throttle the offending workload with the appropriate owner approval. A reboot may erase the evidence and does not explain the failure mode.

Key lesson: High load does not automatically mean high CPU. Load average reflects runnable work and uninterruptible tasks. Check vmstat before assuming CPU saturation.

Related: Best Monitoring Tools for SysAdmins in 2026.

Incident 2 — EC2 Passes Every Health Check. Nobody Can Reach It.

A Terraform change was applied 20 minutes ago. EC2 system status: OK. Instance status: OK. You SSH in through the bastion — works fine. You curl localhost on port 8080 — HTTP 200.

The ALB says the target is unhealthy. Customers are getting 503s.

Most engineers at this point start looking inside the instance. They check application logs. They check whether the service is running. They restart it.

None of that helps.

EC2 instance health and application reachability are different questions. A healthy instance can still be unreachable from the load balancer if the security-group path is wrong.

# Check these in order — stay OUTSIDE the instance
aws elbv2 describe-target-health --target-group-arn <arn>
# → Reason: Target.Timeout

aws ec2 describe-security-groups --group-ids sg-app-id
# → TCP 8080 from sg-alb: MISSING
# → TCP 22  from sg-bastion: present

# Check the Terraform diff
git diff HEAD~1 -- terraform/security_groups.tf
# → ingress 8080 from sg-alb: removed

The Terraform change removed the inbound rule on TCP/8080 from the application security group. The ALB’s health check probes cannot reach the instance. The application is healthy. The network path is broken.

The fix is to restore TCP/8080 inbound on the application security group with the ALB security group as the source — not 0.0.0.0/0.

Key lesson: Never diagnose reachability only from inside the thing that cannot be reached. Check the full path between the load balancer and the instance.

Incident 3 — PostgreSQL CPU at 31%. Application Completely Frozen.

The orders API stops processing writes. Support tickets are flooding in. You check the database. CPU: 31%. It’s accepting connections. It’s running.

Most engineers assume the database is not the problem and start looking at the application.

Wrong direction.

PostgreSQL can look healthy by broad utilization metrics and still be the cause of the incident. CPU is not useful when the problem is lock contention.

-- Run this first — it shows you what is hiding in the database
SELECT pid, usename, state, wait_event_type,
       xact_start,
       now() - xact_start AS xact_age,
       left(query, 80) AS query
FROM pg_stat_activity
WHERE state != 'idle'
ORDER BY xact_start NULLS LAST;

-- Then map the blocking chain
SELECT blocked.pid  AS blocked_pid,
       blocking.pid AS blocking_pid,
       blocking.state,
       left(blocking.query, 60) AS blocking_query
FROM pg_stat_activity AS blocked
JOIN pg_stat_activity AS blocking
  ON blocking.pid = ANY(pg_blocking_pids(blocked.pid))
WHERE blocked.wait_event_type = 'Lock';

In DB-01, PID 4520 — a reporting session — opened a query on the orders table inside a transaction, got its data, and never committed or rolled back. A schema migration needs an AccessExclusiveLock to proceed. It queues behind the long-running transaction. Application writes that touch the table then queue behind the migration.

Database connections climb to 197 out of 200 while CPU sits at 31%.

Key lesson: Low CPU does not mean the database is not the problem. Lock contention can cripple a workload without high CPU. Inspect pg_stat_activity, transaction age, wait events, and blockers.

Incident 4 — The Application Restart That Fixes Everything. For Eight Minutes.

502 errors. ALB reporting target timeouts. Linux hosts at 38% CPU, 0% I/O wait, 62% idle. You restart the application. Things improve immediately.

Eight minutes later the 502s return.

You restart again. Eight minutes. 502s back. This is one of the most dangerous incident patterns because the restart feels like it is working. It is not. It is temporarily resetting state.

16:29:55 WARN  db acquire timeout pool=50 in_use=50 waiting=127
16:29:56 ERROR request failed: upstream database timeout
16:30:02 WARN  db acquire timeout pool=50 in_use=50 waiting=141

pool=50 in_use=50 waiting=127 means all 50 database connections are occupied and 127 more requests are queued. Each stalled request holds or waits on a connection. As the pool saturates, upstream requests time out and the ALB returns 502s.

Restarting the application resets the connection pool. Traffic briefly recovers. But if the PostgreSQL lock source is still present, the pool fills again and the 502s return.

This is a three-layer incident:

• Symptom: ALB 502 — cloud layer
• Amplifier: connection-pool exhaustion — application layer
• Root cause: long-lived PostgreSQL transaction / blocker — database layer

The restart told you the state was recoverable. It was not the root-cause fix.

Key lesson: The layer that shows the symptom may not contain the root cause. Treat restarts as diagnostic evidence, not proof that the problem is solved.

Incident 5 — Disk Full. df -h Says 85%.

Files stop being created on the server. Applications log “no space left on device.” You run df -h. The root filesystem is at 85%.

That is not block-space exhaustion.

You may be out of inodes instead.

Linux filesystems have independent limits for storage blocks and inodes. You can exhaust one without exhausting the other. When inodes hit 100%, you cannot create new files even if there are gigabytes of free block space.

df -h /   # checks storage blocks  → 85%  ← looks fine
df -i /   # checks inodes          → 100% ← this is the problem

# Find where file counts are exploding
for i in /var/*; do echo "$i"; find "$i" | wc -l; done | sort -rn | head -20

The fix is to identify what generated the huge file count — commonly failed log rotation, a backed-up mail queue, or an uncontrolled temp directory — then clean up safely and fix the process that caused the growth.

Key lesson: df -h and df -i measure different resources. Check both on “disk full” incidents.

The Skill That Separates All Five

These incidents share a pattern: the symptom appears in one layer while the useful evidence is hiding somewhere else.

All five require the same skill: cross-layer diagnostic reasoning. You look at a load spike, an ALB error, a frozen API, or a filesystem failure and trace it through Linux, network, application, cloud, and database evidence until the failure chain makes sense.

That skill is difficult to build from documentation alone. Tutorials and certifications can teach commands and concepts, but incident judgment comes from repeated troubleshooting against realistic evidence.

If you’re sharpening Linux troubleshooting skills for interviews as well as production work, see Best Linux SysAdmin Interview Questions in 2026 and Linux SysAdmin AI Prompts for Troubleshooting and Automation.

Stop Watching. Start Solving.

Founding price: $149 — planned retail price: $199.99.

Get Infrastructure War Room Pro founding access →

No subscription. Founding buyers receive the current product plus planned v1.x lab updates included with this edition.

Have you been burned by one of these scenarios? Drop it in the comments — which layer did you check first?

Image credits: featured image by Kevin Ache, network image by Scott Rodgerson, and code image by Chris Ried via Unsplash.