sed is a command, short for Stream Editor, which is a stream-based text editor. It can quickly read, modify, and output text without the need for manual line-by-line editing. This efficient text processing capability makes sed excel in handling large amounts of text data, especially suitable for automation scripts and batch processing tasks.
sed (stream editor) is a very powerful text processing tool in Linux and Unix systems. It is mainly used for filtering and transforming text data. sed can operate directly on input streams without opening files, and can output results to standard output or files.
Basic Syntax:
sed [options] 'script' [input [output]]- [options]: Command line options for
sed, such as -i for in-place file modification.
- script: Defines the script of editing operations to be executed.
- [input]: Input file, default is standard input (stdin).
- [output]: Output file, default is standard output (stdout).
Common sed Commands:
sed 's/pattern/replacement/': Replacement command, replaces pattern with replacement.
-
- g: Global replacement, replaces all matches in the line.
- p: Print, only prints matching lines.
- n: Print as is, no replacement.
sed -n 'pattern': Only prints lines matching pattern.
sed -e 'command1' -e 'command2': Executes multiple commands.
sed -i 's/old/new/' file: Directly modifies file content, replacing old with new.
sed 'N;N;s/\n//g': Merges consecutive empty lines.
sed '1,5 s/.*/Hello/': Performs replacement on lines 1 to 5 of the file.
sed '$!N;s/\n//g': Merges all lines except the last one, removing newline characters.
sed -n '1,5p;1,5!d': Prints lines 1 to 5 of the file and deletes those lines.
Regular Expressions:
sed uses Basic Regular Expressions (BRE) for pattern matching. When using regular expressions in sed, you need to escape special characters.
Examples:
- Replace all occurrences of old with new in the file input.txt and save the result to output.txt:
sed 's/old/new/' input.txt > output.txt- Directly replace all occurrences of old with new in the file file.txt:
sed -i 's/old/new/' file.txt- Print lines 1 to 3 of the file:
sed -n '1,3p' file.txt- Delete all empty lines in the file:
sed ‘/^$/d’ file.txt
Application Scenarios
sed (stream editor) is a stream editor widely used in various text processing tasks in Unix and Unix-like systems. Here are some typical application scenarios for sed:
1. Text Replacement:
sed 's/oldtext/newtext/g' inputfile > outputfileThis command searches for all occurrences of oldtext in inputfile and replaces them with newtext, outputting the result to outputfile.
2. Text Deletion:
sed '/pattern/d' inputfile > outputfileThis command deletes all lines containing pattern in inputfile and outputs the remaining content to outputfile.
3. Text Insertion:
sed '5i\ New text' inputfile > outputfileThis command inserts New text before line 5 in inputfile and outputs the result to outputfile.
4. Text Appending:
sed '$a\ Additional text' inputfile > outputfileThis command appends Additional text to the end of inputfile and outputs the result to outputfile.
5. Text Formatting:
sed '=; s/^/ /' inputfile > outputfileThis command indents each line in inputfile and adds line numbers at the top, outputting the result to outputfile.
6. Text Filtering:
sed -n '1,5p' inputfile > outputfileThis command extracts lines 1 to 5 from inputfile and outputs them to outputfile.
7. Log File Processing:
sed -n '/error/p' inputfile > outputfileThis command extracts all lines containing error from inputfile and outputs them to outputfile.
8. Text Conversion:
sed 's/\t/,/g' inputfile > outputfileThis command replaces all tabs (\t) in inputfile with commas (,) and outputs the result to outputfile.
9. Text Sorting:
sed '1d; s/^/ /' inputfile | sort > outputfileThis command deletes the first line of inputfile, adds a space before each line, sorts the result using the sort command, and outputs it to outputfile.
10. Automation Scripts:
sed_script.shYou can create a shell script sed_script.sh that contains multiple sed commands to automate text processing tasks.
11. Data Extraction and Report Generation:
sed -n '1,$p' inputfile | awk '{print $1}' > reportfileThis command extracts the first field from each line of inputfile and saves these fields to reportfile as part of a report.
12. Text Encryption and Decryption:
# Encryption (simple reverse order)
sed '1,$!G;h;$!d;$!N;s/\n//g' inputfile > encryptedfile
# Decryption
sed '1,$!G;s/$.*$$.*$/\2 \1/;s/ //g' encryptedfile > decryptedfileThis is a simple example of encryption and decryption by reversing the text.
13. Batch Renaming Files:
sed 's/\.txt$/renamed.txt/' filelist.txt | xargs -I {} mv {} {}This command reads from a filelist.txt containing a list of filenames and renames all files ending with .txt to renamed.txt.
Work Scenarios
sed (stream editor) has many applications in real work, especially useful in handling text files and automation tasks. Here are some examples of applications in real work:
1. Log File Analysis
Search for specific error messages in server log files or filter out activity records of specific users. For example, find all 404 errors:
sed -n '/404/' /var/log/apache2/access.log2. Configuration File Editing
Batch modify parameters in configuration files. For example, update all listening ports:
sed -i 's/listen 80/listen 8080/' /etc/httpd/conf/httpd.conf3. Text Replacement
Replace strings in large text files. For example, replace all foo with bar:
sed 's/foo/bar/g' input.txt > output.txt4. Data Cleaning
Delete or format data, such as removing empty lines or comment lines from text files:
sed '/^$/d' data.txt # Delete empty lines
sed '/^#/d' data.txt # Delete comment lines starting with #5. Batch File Renaming
Use sed to generate a new list of filenames, then use the mv command to batch rename files:
sed 's/\.txt$/_renamed.txt/' files.txt | xargs -I {} mv {} {}6. Text Stream Editing
Use sed in pipeline operations to process stream data. For example, find processes with CPU usage over 50%:
ps aux | sed -n '$p; /cpu/ p'7. Automatic Report Generation
Extract information from logs or other text files to generate reports. For example, extract access logs for a specific service:
sed -n '/myservice/' /var/log/messages > myservice.log8. Temporary Configuration Changes
Temporarily change parameters in configuration files without modifying the original file, for example, temporarily increase log level:
sed 's/LogLevelWarn/LogLevelInfo/' /etc/myapp/myapp.conf > temp.conf && source temp.conf9. Password Hiding
Hide sensitive information such as passwords in scripts:
sed 's/password: .*/password: ***/' /etc/passwd10. Conditional Command Execution
Execute specific commands based on matched text. For example, if a file contains a certain keyword, perform a backup operation:
sed -n '/critical issue/{=; /$/q;} p' /var/log/syslog | xargs -I {} tar -czf {}_backup.tar.gz /path/to/data