Services, Logs and Disks

Introduction

Programs that must run all the time, such as a web server or database, are services (also called daemons). Most modern Linux distributions manage them with systemd. When a server misbehaves, you need to check the service, its logs and the machine's resources.

Managing services

systemctl status nginx          # is it running? recent log lines
sudo systemctl start nginx      # start now
sudo systemctl stop nginx       # stop now
sudo systemctl restart nginx    # stop and start
sudo systemctl reload nginx     # reread config without stopping, if supported
sudo systemctl enable nginx     # start automatically at boot
sudo systemctl disable nginx
systemctl list-units --type=service --state=failed

start and enable are different. One starts the service now, and the other makes it start at boot.

Writing a service

A minimal unit file, saved as /etc/systemd/system/myapp.service:

[Unit]
Description=My application
After=network.target

[Service]
ExecStart=/usr/bin/node /srv/myapp/server.js
Restart=on-failure
User=myapp

[Install]
WantedBy=multi-user.target

Then load and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp

Run services as a dedicated, unprivileged user, not as root.

Reading logs

On systemd systems, journalctl shows logs:

journalctl -u nginx              # logs for one service
journalctl -u nginx -f           # follow live
journalctl -u nginx --since "1 hour ago"
journalctl -p err -b             # errors since this boot

Traditional logs also live in /var/log/, such as /var/log/syslog or /var/log/auth.log. Combine them with the text tools you know: grep "error" /var/log/syslog | tail.

Disk space

df -h                    # space on each filesystem, human readable
du -sh /var/log          # size of one folder
du -h --max-depth=1 /var | sort -h   # biggest folders in /var

A full disk breaks many things, and logs are a common cause. Check df -h early when something odd happens.

Memory and load

free -h                  # memory and swap
uptime                   # load averages over 1, 5 and 15 minutes
top                      # what is using CPU and memory

Load average roughly counts processes waiting to run. As a rule of thumb, a load persistently above the number of CPU cores means the machine is overloaded.

A troubleshooting routine

1. Is the service running? systemctl status
2. What do the logs say? journalctl -u name -n 50
3. Is the disk full? df -h
4. Is memory or CPU exhausted? free -h, top
5. Is the network reachable? ping, curl
6. What changed recently? Think about deployments and updates.

Change one thing at a time, and note what you did.

Try it yourself

On a Linux machine or virtual machine, list failed services, view the last 20 lines of a service's log, and find the three largest folders under /var.

Summary

• systemctl starts, stops and enables services, and unit files define them.
• journalctl and /var/log hold logs, and grep helps search them.
• df, du, free and uptime show disk, memory and load for diagnosis.