Container exit 137: debug OOMKilled and memory limits

Identifying an OOMKilled container

Container restarting constantly but no clear crash log? Check the exit code — if it's 137 (= 128 + SIGKILL), the Linux kernel OOM killer terminated the process because it exceeded the memory limit.

Confirming via logs

docker inspect <container_id> | jq '.[0].State'
# {
#   "OOMKilled": true,
#   "ExitCode": 137,
#   ...
# }

# Or
docker stats --no-stream

On Kubernetes:

kubectl describe pod <pod-name>
# Containers:
#   app:
#     Last State: Terminated
#       Reason: OOMKilled
#       Exit Code: 137

Common causes

1. Memory leak in application code
2. Node.js heap limit set lower than the container limit
3. Cache with no TTL or eviction policy
4. N+1 queries loading too much data into memory at once

How to fix

Set an appropriate memory limit

# docker-compose.yml
services:
  app:
    deploy:
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 256M

Tune Node.js heap size

ENV NODE_OPTIONS="--max-old-space-size=400"
# Keep Node heap at ~80% of the container memory limit
# Container 512M → Node heap 400M

Monitor memory usage

// Log memory usage periodically
setInterval(() => {
  const usage = process.memoryUsage();
  logger.log({
    heapUsed: Math.round(usage.heapUsed / 1024 / 1024) + 'MB',
    heapTotal: Math.round(usage.heapTotal / 1024 / 1024) + 'MB',
    rss: Math.round(usage.rss / 1024 / 1024) + 'MB',
  });
}, 60000);

Conclusion

OOMKilled typically has three root causes: a memory leak in application code, a Node.js heap limit not aligned with the container limit, or loading too much data into memory at once.

Troubleshooting checklist:

1. Confirm OOMKilled via docker inspect or kubectl describe pod
2. Set --max-old-space-size to ~80% of the container's memory limit
3. Add memory monitoring to track usage trends over time
4. If heap keeps growing without releasing → memory leak; use node --inspect + Chrome DevTools to capture a heap snapshot