Text Processing with Pipes

Introduction

Almost everything on a Linux system is text: logs, configuration files, command output. A handful of small tools, connected with pipes, can answer big questions fast. This is the real power of the command line.

Examples use a small file called access.log with lines like these:

10.0.0.5 GET /home 200
10.0.0.9 GET /about 200
10.0.0.5 POST /login 401
10.0.0.7 GET /home 200
10.0.0.5 POST /login 200

grep: find lines

grep "login" access.log          # lines containing login
grep -i "get" access.log         # ignore case
grep -v "200" access.log         # lines NOT containing 200
grep -c "POST" access.log        # count matching lines
grep -n "401" access.log         # show line numbers
grep -r "TODO" src/              # search a whole folder
grep -E "GET|POST" access.log    # extended regular expression: either word

sort, uniq and counting

uniq only collapses neighbouring duplicates, so sort first:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn

Reading it left to right: take field 1 (the IP address, with a space as delimiter), sort the addresses, count each unique one, then sort by the count from largest to smallest. The result:

      3 10.0.0.5
      1 10.0.0.9
      1 10.0.0.7

So 10.0.0.5 made the most requests.

cut, head, tail and wc

cut -d' ' -f3 access.log          # third field: the path
head -n 2 access.log              # first two lines
tail -f /var/log/syslog           # follow a growing log (Ctrl+C to stop)
wc -l access.log                  # number of lines

sed: find and replace

sed 's/GET/FETCH/' access.log     # replace the first GET on each line
sed 's/GET/FETCH/g' access.log    # replace all matches on each line
sed -n '2,3p' access.log          # print only lines 2 to 3

By default sed prints the result and leaves the file unchanged. To edit in place use -i, and keep a backup: sed -i.bak 's/old/new/g' file.

awk: work with columns

awk treats each line as fields, named $1, $2, and so on.

awk '{print $1, $4}' access.log                # fields 1 and 4
awk '$4 == 401 {print $1}' access.log          # IPs with a 401 status
awk '{count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log

The last one counts lines per IP using an array.

A bigger pipeline

How many failed logins (status 401) did each IP make?

grep "401" access.log | cut -d' ' -f1 | sort | uniq -c

Build pipelines one step at a time. Run the first command, check the output, then add the next.

Try it yourself

Create access.log with the lines above using echo and >>. Find the most requested path, and count how many requests returned 200.

Summary

• grep finds lines, cut and awk take fields, and sort plus uniq -c count things.
• sed replaces text, and does not change the file unless you use -i.
• Pipelines combine small tools. Build them one stage at a time.