Mastering Sed: The Stream Editor for Power Text Processing


What is sed?

Sed, short for stream editor, is a command-line tool used to parse, transform, and filter text in a programmatic and non-interactive manner. Originally developed in the 1970s for Unix systems, sed has stood the test of time and remains an essential utility for anyone working with large amounts of text, scripting, system administration, or data processing.

Unlike traditional text editors, sed reads input line by line (stream), processes it according to a set of instructions, and outputs the modified text. The powerful combination of regular expressions and a minimal scripting language enables users to handle complex text manipulation tasks efficiently.


Major Use Cases of Sed

1. Search and Replace

The most common and widely used feature of sed. With the s/pattern/replacement/ syntax, you can quickly replace text strings, even with regular expressions.

  • Example: sed 's/cat/dog/' pets.txt Replaces the first occurrence of “cat” with “dog” in each line of pets.txt.

2. Global Replacements

To replace all occurrences in a line:

sed 's/cat/dog/g' pets.txt

3. Delete Lines

Remove lines matching a specific pattern.

  • Example: sed '/DEBUG/d' logfile.txt Deletes all lines that contain the word “DEBUG”.

4. Insert or Append Lines

Insert content before or after specific lines.

  • Example: sed '/ERROR/i\Check this issue' log.txt Inserts “Check this issue” before every line that contains “ERROR”.
  • Append: sed '/ERROR/a\--End of Log Entry--' log.txt

5. Select and Print Specific Lines

You can extract a range of lines or specific matches:

sed -n '5,10p' file.txt
sed -n '/start/,/end/p' file.txt

6. Batch File Updates

Modify configuration files across a system or project with simple scripts.

7. Script Automation

sed is widely used in DevOps pipelines for automated deployments, where configuration files or environment variables are modified on-the-fly.

8. Log File Cleanup

Quickly clean up and format system logs or application output using one-liners or reusable scripts.


How Sed Works: Under-the-Hood Architecture

At its core, sed works through a pattern space and optionally a hold space, processing text input in a streaming fashion.

1. Input Stream

sed reads text line by line (or “record by record”) from standard input or a file.

2. Pattern Space

Each line is loaded into the pattern space — a working buffer where all sed commands act upon. Changes to the text are made here.

3. Execution Engine

The sed interpreter executes a list of commands on the pattern space. These can include substitutions, deletions, insertions, or text output commands.

4. Hold Space (Optional)

The hold space is a secondary buffer used to temporarily store and retrieve text between commands or lines. It is especially useful in multi-line editing or more complex sed scripts.

5. Output

Once all commands for the current line are applied, the result (unless suppressed) is sent to standard output. Then, the pattern space is cleared and the next line is read, repeating the process.


Basic Workflow of Sed

The flow of a sed operation can be broken into these stages:

  1. Initialize script or expression
  2. Read input (line-by-line)
  3. Load current line into pattern space
  4. Apply all sed commands
  5. Output (unless suppressed with -n)
  6. Repeat for all lines until EOF

Syntax Overview

sed [OPTIONS] 'COMMANDS' [FILE]
  • -e: add script to the commands to be executed
  • -n: suppress automatic output
  • -i: in-place file editing (with optional backup)

Step-by-Step Getting Started Guide for Sed

Step 1: Basic Substitution

Replace “foo” with “bar” in a file.

sed 's/foo/bar/' input.txt

Step 2: Replace All Occurrences in Each Line

sed 's/foo/bar/g' input.txt

Step 3: Case-Insensitive Replacement

sed 's/foo/bar/Ig' input.txt

Step 4: Save Output to a New File

sed 's/error/notice/g' log.txt > updated_log.txt

Step 5: In-Place Editing (GNU/Linux)

sed -i 's/localhost/127.0.0.1/g' config.cfg

On macOS:

sed -i '' 's/localhost/127.0.0.1/g' config.cfg

Step 6: Delete Lines

Delete all comments (lines starting with #):

sed '/^#/d' config.cfg

Step 7: Insert Text Before a Match

sed '/main()/i\// main function starts here' main.c

Step 8: Append Text After a Match

sed '/main()/a\// main function ends here' main.c

Step 9: Multiple sed Commands

sed -e 's/foo/bar/g' -e '/baz/d' file.txt

Step 10: Use sed Script Files

Create commands.sed:

s/red/green/g
/DELETE/d

Run it:

sed -f commands.sed colors.txt

Advanced Tips for Sed Users

  • Backreferences in Substitutions sed 's/\(.*\):\(.*\)/\2:\1/' file.txt Swaps text before and after the colon.
  • Using sed with Pipes cat logfile.txt | sed 's/ERROR/WARNING/g'
  • Transform Characters sed 'y/abc/ABC/' input.txt
  • Print Line Numbers with Matches sed -n '/pattern/=' file.txt
  • Print Only Modified Lines sed -n 's/foo/bar/p' input.txt