Edit streams with sed and awk
Find-and-replace with sed, and process columns, totals and conditions with awk.
- Substitute, delete and print lines with sed
- Use capture groups to rearrange text
- Select fields, filter rows and compute totals with awk
sed is a stream editor: it applies editing commands to every line passing through. Its star is substitution:
1sed 's/old/new/' # replace the first match on each line
2sed 's/old/new/g' # replace every match (g = global)
3sed -E 's/(a+)b/\1/' # -E for extended regex; \1 is the 1st captured group
4sed 's|/usr|/opt|g' # any character can be the delimiter - handy for paths
5sed -n '2,4p' # -n: print nothing by default; 2,4p prints lines 2 to 4
6sed '/^#/d' # delete lines that start with #1cat > menu.txt <<'EOF'
2# Galley menu
3monday: soup, soup, bread
4tuesday: noodles
5wednesday: soup
6EOF
7sed 's/soup/stew/' menu.txt | sed -n '2p'
8sed 's/soup/stew/g' menu.txt | sed -n '2p'
9sed '/^#/d' menu.txt | sed -E 's/^([a-z]+): (.*)/\2 (\1)/'monday: stew, soup, bread monday: stew, stew, bread soup, soup, bread (monday) noodles (tuesday) soup (wednesday)
awk: a tiny language for columns
awk splits each line into fields - $1, $2, ... ($0 is the whole line, NF the number of fields, NR the line number) - and runs pattern { action } rules on every line. Leave out the pattern and the action runs on every line; leave out the action and matching lines are printed.
1awk '{ print $1 }' # first field (split on spaces by default)
2awk -F, '{ print $2, $1 }' # -F, splits on commas; the comma in print adds a space
3awk '$3 > 100' # lines whose 3rd field is over 100
4awk '{ sum += $2 } END { print sum }' # END runs after the last line
5awk '{ count[$1]++ } END { for (key in count) print key, count[key] }'Variables start at 0 or empty, so sum += $2 just works, and count[$1]++ builds a tally - like a dictionary - keyed by the first field.
1cat > cargo.csv <<'EOF'
2coffee,deck1,40
3oxygen,deck2,300
4water,deck2,500
5tools,deck3,120
6snacks,deck1,15
7EOF
8echo "Heavy items:"
9awk -F, '$3 >= 120 { print " " $1 " (" $3 " kg)" }' cargo.csv
10echo "Mass per deck:"
11awk -F, '{ mass[$2] += $3 } END { for (deck in mass) print deck, mass[deck] }' cargo.csv | sort
12awk -F, '{ total += $3 } END { printf "Total: %d kg in %d items\n", total, NR }' cargo.csvHeavy items: oxygen (300 kg) water (500 kg) tools (120 kg) Mass per deck: deck1 55 deck2 800 deck3 120 Total: 975 kg in 5 items
for (key in array) visits keys in no particular order, so pipe the output through sort when order matters.
Try it
Pick the right tool
grep, sed, awk and cut overlap a lot, but each has a sweet spot. Choose the simplest tool for each job.
“Show only the lines that mention “timeout””
“Replace every “http://” with “https://””
“Add up the numbers in the 4th column”
“Keep just the 2nd field of a colon-separated file”
“Print lines where the 3rd column is greater than 100”
“Delete blank lines”
“Count lines containing “ERROR””
Key takeaways
sed 's/old/new/g'replaces text;-Ewith\1rearranges captured groups.sed -n 'Np'prints specific lines andsed '/re/d'deletes matching ones.awk splits lines into
$1,$2, ... and runspattern { action }rules.awk’s variables, arrays and
ENDblock handle totals and tallies.
Lesson quiz
7 questions · pass with 5 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Bash scripts
Write a script in the editor and run it for real against sample input. Each run gets a fresh Linux sandbox with Bash 5.2 and the GNU tools on Wandbox, a free public service - so experiment freely, even with rm. Your script and test input are sent there.
Reformat the dates
stdin is the launch schedule, one launch per line, like 2026-10-02 cargo run. Rewrite each date from YYYY-MM-DD to DD/MM/YYYY and leave the rest of the line alone, using sed.
- Schedule
Your script runs with Bash 5.2 and GNU tools on Wandbox, a free public service, in a fresh sandbox each time. Your script and test input are sent to that service.
Mass per deck
stdin is the cargo manifest as CSV: item,deck,kg. Print each deck with its total mass (deck1 55), sorted by deck name.
- Manifest
- Single deck
Your script runs with Bash 5.2 and GNU tools on Wandbox, a free public service, in a fresh sandbox each time. Your script and test input are sent to that service.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…