Sed linux add a character to the end of the line. Using the Linux stream text editor sed. Inserting text into a stream
Many of you probably used the stream text editor sed for some of your purposes, if not, I'll be glad to tell you about it, I'll try to get more details. Why is it called streaming? The answer is simple - imagine the input text Document which goes through the program and ends up with some other form of this fileprocessed by the program. A kind of meat grinder - you put meat on the basis of the grid - you get either minced meat or something else.
So, by default, it seems like this utility should already be on your system (in my case, in Debian 7.6 I already had it), if not, then -
With text:
the "s" parameter at the beginning indicates that the text should be replaced, g - at the end of the replaced text - what needs to be done globally (throughout the file)
For example, we want to replace the word Sergey with Andrey in our text.txt file and upload all this to the textout.txt file, proceed:
sed "s / Sergey / Andrey / g" text. txt\u003e textout. txt |
Result:
If you want to make replacements for special symbols - for example on the & symbol, then it is necessary before the special. put a backslash "\\" with a character, if it is necessary to indicate what sed needs to turn to the beginning of a line, use the special character "^". In addition, you can write 2 or more changes on one line, separating them with a semicolon - ";". For example, we torture the already changed textout.txt file. First, I'll show you the current content of the textout.txt file again:
root @ testhostname: ~ # cat textout.txt Test for Andrey Test 2 for Andrey Test 3 for Andrey |
Now we enter the command:
sed "s / for / \\ & / g; s / ^ Test / Sergey / g" textout. txt\u003e textout2. txt |
Thus, instead of the word for, we put the & icon (the special character is entered with the "\\" before the special character), then the separation sign (so that all changes can be written in one sed line -\u003e ";", instead of the word at the beginning of the line "Test" put the word Sergey, the result of what happened:
Everything as we wanted!
Also, sed is a good helper for viewing logs. For example, we need to unload all lines of today's date (let in our case it is October 10) from the log file / var / log / messages to the testlog.txt file, let's proceed:
sed - n "/ ^ Oct 10 / p" / var / log / messages\u003e testlog. txt |
here we have added the -n parameter, and then - '/ ^ Oct 10 / - meaning that the line should start from the date of October 10, then the p parameter - meaning print (print the content when given condition), then the source file and the file where we drop the results according to our filter condition, run it, see what the testlog.txt file contains exclusively on October 10:
Excellent! If you do not need many lines, but conditionally there is a need to take only from the first to the fifth line, we separate our current query with the "|" removing the dump to the testlog.txt file and write sed -n 1,5p - which means that we need to print (p - print at the end of the expression) from the first "1" to (separated by commas) the fifth "5" line. In total, we get something like this:
sed - n "/ ^ Oct 10 / p" / var / log / messages | sed - n 1, 5p\u003e testlog - 5strok.txt |
Once again, I draw your attention to the fact that the file where we upload the results is moved to the end (testlog-5strok.txt), we see the result of our actions:
Last time we talked about functions in bash scripts, in particular, how to call them from command line... Our topic today is a very useful tool for processing string data - a Linux utility called sed. It is often used to work with texts in the form of log files, configuration files and other files.
If you manipulate data in some way in bash scripts, you should be familiar with the sed and gawk tools. Here we will focus on sed and text manipulation, as this is a very important step on our journey through the vast expanses of bash script development.
Now we will cover the basics of working with sed, as well as look at more than three dozen examples of using this tool.
Sed basics
The sed utility is called a stream text editor. In interactive text editors, like nano, they work with texts using the keyboard, editing files, adding, removing or changing texts. Sed allows you to edit data streams based on developer-defined sets of rules. This is how this command is called: $ sed options file
By default, sed applies the specified command set rules to STDIN when invoked. This allows data to be passed directly to sed.
For example, like this:
$ echo "This is a test" | sed "s / test / another test /"
Here's what you get when you run this command.
Simple sed call example
In this case, sed replaces the word "test" in the line passed in for processing with the words "another test". For formatting the rule for processing text enclosed in quotes, forward slashes are used. In our case, we used a command like s / pattern1 / pattern2 /. The letter "s" is an abbreviation of the word "substitute", that is, we are facing a replacement command. Sed, executing this command, will scan the transferred text and replace the fragments found in it (about which ones, we'll talk below), corresponding to pattern1, with pattern2.
Above is a primitive sed example to get you up to speed. In fact, sed can be used in much more complex text processing scenarios, such as working with files.
Below is a file that contains a piece of text, and the results of processing it with this command:
$ sed "s / test / another test" ./myfile
Text file and the results of its processing
This is the same approach we used above, but sed now processes the text stored in the file. At the same time, if the file is large enough, you will notice that sed processes the data in chunks and displays what has been processed on the screen, without waiting for the entire file to be processed.
Sed does not change data in the processed file. The editor reads the file, processes the read, and sends the output to STDOUT. To make sure that the original file hasn't changed, it is enough to open it after it has been passed to sed. If necessary, the output of sed can be redirected to a file, possibly overwriting the old file. If you are familiar with one of the previous articles in this series, which deals with redirecting input and output streams, you can do it.
Executing command sets when invoking sed
To perform multiple data manipulations, use the -e switch when invoking sed. For example, here's how to organize the replacement of two pieces of text: $ sed -e "s / This / That /; s / test / another test /" ./myfile
Using the -e switch when invoking sed
Both commands are applied to each line of text in the file. Separate them with a semicolon, with no space between the end of the command and the semicolon.
To enter several patterns of text processing when invoking sed, you can, after entering the first single quote, press Enter, and then enter each rule on a new line, not forgetting the closing quote:
$ sed -e "\u003e s / This / That /\u003e s / test / another test /" ./myfile
This is what happens after the command presented in this form is executed.
Another way to work with sed
Reading commands from a file
If you have many sed commands to process text with, it is usually best to write them to a file first. In order to specify a file containing commands to sed, use the -f switch:Here is the content of the mycommands file:
S / This / That / s / test / another test /
Let's call sed, passing the command file and the file to be processed to the editor:
$ sed -f mycommands myfile
The result when calling such a command is the same as in the previous examples.
Using a file with commands when invoking sed
Replacement command flags
Take a close look at the following example. $ sed "s / test / another test /" myfile
This is what is in the file and what sed will get after processing it.
The original file and the results of its processing
The replace command normally processes a file consisting of several lines, but only the first occurrences of the search text on each line are replaced. In order to replace all occurrences of a pattern, use the corresponding flag.
The scheme for writing the replacement command when using flags looks like this:
S / pattern / replacement / flags
The execution of this command can be modified in several ways.
- When transferring the number, serial number occurrences of a pattern in a string, exactly this occurrence will be replaced.
- The g flag indicates to process all occurrences of the pattern in the string.
- The p flag indicates to print the contents of the original string.
- A flag like w file tells the command to write the text processing results to a file.
$ sed "s / test / another test / 2" myfile
Calling the replace command indicating the position of the fragment to be replaced
Here we indicated the number 2. as the replacement flag. This led to the fact that only the second occurrence of the required pattern in each line was replaced. Now let's try out the global replace flag - g:
$ sed "s / test / another test / g" myfile
As you can see from the output, this command replaced all occurrences of the pattern in the text.
Global replacement
The replacement command flag p allows you to print the lines that match, but the -n option specified when invoking sed suppresses normal output:
$ sed -n "s / test / another test / p" myfile
As a result, when sed is run in this configuration, only the lines (in our case, one line) are displayed in which the specified text fragment was found.
Using the p replace command flag
Let's use the w flag, which allows us to save the text processing results to a file:
$ sed "s / test / another test / w output" myfile
Saving text processing results to a file
It is clearly seen that during the operation of the command, the data is output to STDOUT, while the processed lines are written to the file whose name is specified after w.
Separators
Imagine replacing / bin / bash with / bin / csh in / etc / passwd. The task is not so difficult: $ sed "s / \\ / bin \\ / bash / \\ / bin \\ / csh /" / etc / passwd
However, all this does not look very good. The fact is that since forward slashes are used as separator characters, the same characters in the lines passed to sed must be escaped. As a result, the readability of the command suffers.
Fortunately, sed allows us to define delimiter characters ourselves for use in the replacement command. The delimiter is the first character that is encountered after s:
$ sed "s! / bin / bash! / bin / csh!" / etc / passwd
In this case, the separator used is exclamation pointAs a result, the code is easier to read and looks much cleaner than before.
Selecting text fragments for processing
So far, we've called sed to process the entire data stream passed to the editor. In some cases, sed only needs to process a portion of the text — a particular line or group of lines. To achieve this goal, you can take two approaches:- Set a limit on the numbers of processed lines.
- Specify the filter for which rows are to be processed.
$ sed "2s / test / another test /" myfile
Processing only one line, the number given when calling sed
The second option is a range of strings:
$ sed "2,3s / test / another test /" myfile
Handling a range of strings
In addition, you can call the replace command so that the file is processed from a certain line to the end:
$ sed "2, $ s / test / another test /" myfile
File processing from second line to end
In order to process only lines matching the specified filter using the replace command, the command must be called like this:
$ sed "/ likegeeks / s / bash / csh /" / etc / passwd
Similar to what was discussed above, the pattern is passed before the command name s.
Processing rows matching a filter
We used a very simple filter here. In order to fully reveal the capabilities of this approach, you can use regular expressions. We will talk about them in one of the next materials in this series.
Deleting lines
Sed is good for more than just replacing one sequence of characters in strings with another. With its help, namely, using the d command, you can remove lines from the text stream.The command call looks like this:
$ sed "3d" myfile
We want the third line to be removed from the text. Please note that this is not a file. The file will remain unchanged, the deletion will only affect the output that sed will generate.
Removing the third line
If you do not specify the line number to be deleted when calling the d command, all lines in the stream will be deleted.
Here's how to apply the d command to a range of strings:
$ sed "2,3d" myfile
Deleting a Range of Rows
And here's how to delete lines starting with the given one and up to the end of the file:
$ sed "3, $ d" myfile
Delete lines to the end of the file
Lines can also be deleted by pattern:
$ sed "/ test / d" myfile
Deleting lines by pattern
When calling d, you can specify a couple of patterns - the lines in which the pattern is encountered, and those lines that are between them will be deleted:
$ sed "/ second /, / fourth / d" myfile
Deleting a Range of Rows Using Templates
Inserting text into a stream
With sed, you can insert data into a text stream using the i and a commands:- The i command adds new line before the given one.
- The a command adds a new line after the given one.
$ echo "Another test" | sed "i \\ First test"
Team i
Now let's take a look at command a:
$ echo "Another test" | sed "a \\ First test"
Command a
As you can see, these commands add text before or after the data from the stream. What if you need to add a line somewhere in the middle?
Here we will be helped by specifying the number of the reference line in the stream, or template. Note that addressing strings as a range will not work here. Let's call the i command, specifying the line number before which we need to insert a new line:
$ sed "2i \\ This is the inserted line." myfile
I command with reference line number
Let's do the same with the a command:
$ sed "2a \\ This is the appended line." myfile
A command with reference line number
Note the difference between the i and a commands. The first one inserts a new line before the specified one, the second one after.
Replacing strings
The c command allows you to change the content of an entire line of text in a data stream. When calling it, you need to specify the line number, instead of which you need to add new data to the stream: $ sed "3c \\ This is a modified line." myfile
Replacing an entire string
If you use a plain text or regular expression pattern when invoking a command, all lines matching the pattern will be replaced:
$ sed "/ This is / c This is a changed line of text." myfile
Replacing strings by pattern
Replacing characters
The y command works on individual characters, replacing them according to the data passed to it when called: $ sed "y / 123/567 /" myfile
Replacing characters
When using this command, you must take into account that it applies to the entire text stream, you cannot limit it to specific occurrences of characters.
Displaying line numbers
If you invoke sed using the \u003d command, the utility will print the line numbers in the data stream: $ sed "\u003d" myfile
Displaying line numbers
The streaming editor printed line numbers in front of their content.
If you pass a pattern to this command and use the sed -n switch, only line numbers that match the pattern are printed:
$ sed -n "/ test / \u003d" myfile
Displaying line numbers matching a pattern
Reading data to insert from a file
Above, we looked at techniques for inserting data into a stream, specifying what to insert, right on the call to sed. You can also use a file as a data source. To do this, use the r command, which allows you to insert data from the specified file into the stream. When calling it, you can specify the line number after which you need to insert the file content, or a template.Let's consider an example:
$ sed "3r newfile" myfile
Inserting file content into a stream
Here the content of newfile was inserted after the third line of myfile.
Here's what happens if you apply the pattern when you invoke r:
$ sed "/ test / r newfile" myfile
Using a template when calling the r command
The file content will be inserted after each line matching the pattern.
Example
Let's imagine such a task. There is a file in which there is a certain sequence of characters, in itself meaningless, which must be replaced with data taken from another file. Namely, let it be the file newfile, in which the DATA character sequence acts as a placeholder. The data to be substituted for DATA is stored in the data.You can solve this problem using the r and d commands of the stream editor sed:
$ Sed "/ DATA\u003e / (r newfile d)" myfile
Replacing the place indicator with real data
As you can see, instead of the DATA placeholder, sed added two lines from data.
Outcome
Today we covered the basics of working with the sed stream editor. In fact, sed is a huge topic. Learning it can be compared to learning a new programming language, however, once you understand the basics, you can master sed at any level you need. As a result, your ability to process texts with it will be limited only by your imagination.That's all for today. Next time, let's talk about awk data processing language.
Dear Readers! Do you use sed in your day to day work? If so, please share your experience.
Last time we talked about functions in bash scripts, in particular how to call them from the command line. Our topic today is a very useful tool for processing string data - a Linux utility called sed. It is often used to work with texts in the form of log files, configuration files and other files.
If you manipulate data in some way in bash scripts, you should be familiar with the sed and gawk tools. Here we will focus on sed and text manipulation, as this is a very important step on our journey through the vast expanses of bash script development.
Now we will cover the basics of working with sed, as well as look at more than three dozen examples of using this tool.
Sed basics
The sed utility is called a stream text editor. In interactive text editors like nano, texts are manipulated using the keyboard, editing files, adding, deleting, or changing texts. Sed allows you to edit data streams based on developer-defined sets of rules. This is how this command is called: $ sed options file
By default, sed applies the specified command set rules to STDIN when invoked. This allows data to be passed directly to sed.
For example, like this:
$ echo "This is a test" | sed "s / test / another test /"
Here's what you get when you run this command.
Simple sed call example
In this case, sed replaces the word "test" in the line passed in for processing with the words "another test". For formatting the rule for processing text enclosed in quotes, forward slashes are used. In our case, we used a command like s / pattern1 / pattern2 /. The letter "s" is an abbreviation of the word "substitute", that is, we are facing a replacement command. Sed, executing this command, will scan the transferred text and replace the fragments found in it (about which ones, we'll talk below), corresponding to pattern1, with pattern2.
Above is a primitive sed example to get you up to speed. In fact, sed can be used in much more complex text processing scenarios, such as working with files.
Below is a file that contains a piece of text, and the results of processing it with this command:
$ sed "s / test / another test" ./myfile
Text file and results of its processing
This is the same approach we used above, but sed now processes the text stored in the file. At the same time, if the file is large enough, you will notice that sed processes the data in chunks and displays what has been processed on the screen, without waiting for the entire file to be processed.
Sed does not change data in the processed file. The editor reads the file, processes the read, and sends the output to STDOUT. To make sure that the original file hasn't changed, it is enough to open it after it has been passed to sed. If necessary, the output of sed can be redirected to a file, possibly overwriting the old file. If you are familiar with one of the previous articles in this series, which deals with redirecting input and output streams, you can do it.
Executing command sets when invoking sed
To perform multiple data manipulations, use the -e switch when invoking sed. For example, here's how to organize the replacement of two pieces of text: $ sed -e "s / This / That /; s / test / another test /" ./myfile
Using the -e switch when invoking sed
Both commands are applied to each line of text in the file. Separate them with a semicolon, with no space between the end of the command and the semicolon.
To enter several patterns of text processing when invoking sed, you can, after entering the first single quote, press Enter, and then enter each rule on a new line, not forgetting the closing quote:
$ sed -e "\u003e s / This / That /\u003e s / test / another test /" ./myfile
This is what happens after the command presented in this form is executed.
Another way to work with sed
Reading commands from a file
If you have many sed commands to process text with, it is usually best to write them to a file first. In order to specify a file containing commands to sed, use the -f switch:Here is the content of the mycommands file:
S / This / That / s / test / another test /
Let's call sed, passing the command file and the file to be processed to the editor:
$ sed -f mycommands myfile
The result when calling such a command is the same as in the previous examples.
Using a file with commands when invoking sed
Replacement command flags
Take a close look at the following example. $ sed "s / test / another test /" myfile
This is what is in the file and what sed will get after processing it.
The original file and the results of its processing
The replace command normally processes a file consisting of several lines, but only the first occurrences of the search text on each line are replaced. In order to replace all occurrences of a pattern, use the corresponding flag.
The scheme for writing the replacement command when using flags looks like this:
S / pattern / replacement / flags
The execution of this command can be modified in several ways.
- When transmitting the number, the sequence number of the occurrence of the pattern in the string is taken into account; this particular occurrence will be replaced.
- The g flag indicates to process all occurrences of the pattern in the string.
- The p flag indicates to print the contents of the original string.
- A flag like w file tells the command to write the text processing results to a file.
$ sed "s / test / another test / 2" myfile
Calling the replace command indicating the position of the fragment to be replaced
Here we indicated the number 2. as the replacement flag. This led to the fact that only the second occurrence of the required pattern in each line was replaced. Now let's try out the global replace flag - g:
$ sed "s / test / another test / g" myfile
As you can see from the output, this command replaced all occurrences of the pattern in the text.
Global replacement
The replacement command flag p allows you to print the lines that match, but the -n option specified when invoking sed suppresses normal output:
$ sed -n "s / test / another test / p" myfile
As a result, when sed is run in this configuration, only the lines (in our case, one line) are displayed in which the specified text fragment was found.
Using the p replace command flag
Let's use the w flag, which allows us to save the text processing results to a file:
$ sed "s / test / another test / w output" myfile
Saving text processing results to a file
It is clearly seen that during the operation of the command, the data is output to STDOUT, while the processed lines are written to the file whose name is specified after w.
Separators
Imagine replacing / bin / bash with / bin / csh in / etc / passwd. The task is not so difficult: $ sed "s / \\ / bin \\ / bash / \\ / bin \\ / csh /" / etc / passwd
However, all this does not look very good. The fact is that since forward slashes are used as separator characters, the same characters in the lines passed to sed must be escaped. As a result, the readability of the command suffers.
Fortunately, sed allows us to define delimiter characters ourselves for use in the replacement command. The delimiter is the first character that is encountered after s:
$ sed "s! / bin / bash! / bin / csh!" / etc / passwd
In this case, an exclamation point is used as a separator, as a result the code is easier to read and looks much cleaner than before.
Selecting text fragments for processing
So far, we've called sed to process the entire data stream passed to the editor. In some cases, sed only needs to process a portion of the text — a particular line or group of lines. To achieve this goal, you can take two approaches:- Set a limit on the numbers of processed lines.
- Specify the filter for which rows are to be processed.
$ sed "2s / test / another test /" myfile
Processing only one line, the number given when calling sed
The second option is a range of strings:
$ sed "2,3s / test / another test /" myfile
Handling a range of strings
In addition, you can call the replace command so that the file is processed from a certain line to the end:
$ sed "2, $ s / test / another test /" myfile
File processing from second line to end
In order to process only lines matching the specified filter using the replace command, the command must be called like this:
$ sed "/ likegeeks / s / bash / csh /" / etc / passwd
Similar to what was discussed above, the pattern is passed before the command name s.
Processing rows matching a filter
We used a very simple filter here. In order to fully reveal the capabilities of this approach, you can use regular expressions. We will talk about them in one of the next materials in this series.
Deleting lines
Sed is good for more than just replacing one sequence of characters in strings with another. With its help, namely, using the d command, you can remove lines from the text stream.The command call looks like this:
$ sed "3d" myfile
We want the third line to be removed from the text. Please note that this is not a file. The file will remain unchanged, the deletion will only affect the output that sed will generate.
Removing the third line
If you do not specify the line number to be deleted when calling the d command, all lines in the stream will be deleted.
Here's how to apply the d command to a range of strings:
$ sed "2,3d" myfile
Deleting a Range of Rows
And here's how to delete lines starting with the given one and up to the end of the file:
$ sed "3, $ d" myfile
Delete lines to the end of the file
Lines can also be deleted by pattern:
$ sed "/ test / d" myfile
Deleting lines by pattern
When calling d, you can specify a couple of patterns - the lines in which the pattern is encountered, and those lines that are between them will be deleted:
$ sed "/ second /, / fourth / d" myfile
Deleting a Range of Rows Using Templates
Inserting text into a stream
With sed, you can insert data into a text stream using the i and a commands:- The i command adds a new line before the given one.
- The a command adds a new line after the given one.
$ echo "Another test" | sed "i \\ First test"
Team i
Now let's take a look at command a:
$ echo "Another test" | sed "a \\ First test"
Command a
As you can see, these commands add text before or after the data from the stream. What if you need to add a line somewhere in the middle?
Here we will be helped by specifying the number of the reference line in the stream, or template. Note that addressing strings as a range will not work here. Let's call the i command, specifying the line number before which we need to insert a new line:
$ sed "2i \\ This is the inserted line." myfile
I command with reference line number
Let's do the same with the a command:
$ sed "2a \\ This is the appended line." myfile
A command with reference line number
Note the difference between the i and a commands. The first one inserts a new line before the specified one, the second one after.
Replacing strings
The c command allows you to change the content of an entire line of text in a data stream. When calling it, you need to specify the line number, instead of which you need to add new data to the stream: $ sed "3c \\ This is a modified line." myfile
Replacing an entire string
If you use a plain text or regular expression pattern when invoking a command, all lines matching the pattern will be replaced:
$ sed "/ This is / c This is a changed line of text." myfile
Replacing strings by pattern
Replacing characters
The y command works on individual characters, replacing them according to the data passed to it when called: $ sed "y / 123/567 /" myfile
Replacing characters
When using this command, you must take into account that it applies to the entire text stream, you cannot limit it to specific occurrences of characters.
Displaying line numbers
If you invoke sed using the \u003d command, the utility will print the line numbers in the data stream: $ sed "\u003d" myfile
Displaying line numbers
The streaming editor printed line numbers in front of their content.
If you pass a pattern to this command and use the sed -n switch, only line numbers that match the pattern are printed:
$ sed -n "/ test / \u003d" myfile
Displaying line numbers matching a pattern
Reading data to insert from a file
Above, we looked at techniques for inserting data into a stream, specifying what to insert, right on the call to sed. You can also use a file as a data source. To do this, use the r command, which allows you to insert data from the specified file into the stream. When calling it, you can specify the line number after which you need to insert the file content, or a template.Let's consider an example:
$ sed "3r newfile" myfile
Inserting file content into a stream
Here the content of newfile was inserted after the third line of myfile.
Here's what happens if you apply the pattern when you invoke r:
$ sed "/ test / r newfile" myfile
Using a template when calling the r command
The file content will be inserted after each line matching the pattern.
Example
Let's imagine such a task. There is a file in which there is a certain sequence of characters, in itself meaningless, which must be replaced with data taken from another file. Namely, let it be the file newfile, in which the DATA character sequence acts as a placeholder. The data to be substituted for DATA is stored in the data.You can solve this problem using the r and d commands of the stream editor sed:
$ Sed "/ DATA\u003e / (r newfile d)" myfile
Replacing the place indicator with real data
As you can see, instead of the DATA placeholder, sed added two lines from data.
Outcome
Today we covered the basics of working with the sed stream editor. In fact, sed is a huge topic. Learning it can be compared to learning a new programming language, however, once you understand the basics, you can master sed at any level you need. As a result, your ability to process texts with it will be limited only by your imagination.That's all for today. Next time, let's talk about awk data processing language.
Dear Readers! Do you use sed in your day to day work? If so, please share your experience.
Author: Rares Aioanei
Publication date: November 19, 2011
Translation: A. Krivoshey
Date of transfer: July 2012
1. Introduction
Welcome to the second part of our series on sed, the GNU version. There are several versions of sed that are available on different platforms, but we will focus on GNU sed 4.x. Many of you have heard of, or have already used sed, most likely as a replacement tool. But this is just one of the uses of sed, and we will try to show you all the aspects of using sed. Its name stands for "Stream EDitor" and the word "stream" in this case can mean a file, a channel, or simply stdin. We hope that you already have a basic knowledge of Linux, and if you have already worked with regular expressions, or at least know what it is, then everything will be much easier for you. The volume of the article does not allow to include in it complete guide on regular expressions, instead we'll go over the basic concepts and give lots of examples of sed usage.
2. Installation
There is not much to tell here. Most likely you already have sed installed, since it is used by various system scripts, as well as linux userswho want to improve their efficiency. You can find out what version of sed you have installed with the command:
$ sed --version
On my system, this command shows that I have GNU sed 4.2.1 installed plus gives a link to the program's home page and other useful information. The package is named "sed" regardless of distribution, except in Gentoo where it is implicitly present.
3. Concepts
Before going any further, we think it is important to focus on what sed does, since the word "stream editor" says little about its purpose. sed accepts text as input, performs the specified operations on each line (unless otherwise specified), and prints the modified text. The specified operations can be add, insert, delete or replace. This is not as easy as it sounds: be warned that there are a large number of options and their combinations that can make the sed command very difficult to understand. Therefore, we recommend that you learn the basics of regular expressions to understand how it works. Before starting the tutorial, we would like to thank Eric Pement and others for the inspiration and what he has done for everyone who wants to learn and use sed.
4. Regular expressions
Since sed commands (scripts) remain a mystery to many, we feel that our readers should understand the basic concepts rather than blindly copy and paste commands whose meaning they do not understand. When a person wants to understand what regular expressions are, keyword is a "match", or more precisely, a "match pattern". For example, in a report for your department, you wrote the name Nick to a network architect. But Nick is gone and John is in his place, so now you have to replace Nick with John. If the report file is named report.txt, you must run the following command:
$ cat report.txt / sed "s / Nick / John / g"\u003e report_new.txt
Sed uses stdout by default, you can use the output redirection operator as shown in the example above. This is a very simple example, but we have illustrated a few points: we look for all matches to the pattern "Nick" and replace in all cases with "John". Note that sed does case sensitive searches, so be careful and check the output file to make sure any replacements have been made. The above example could have been written like this:
$ sed "s / Nick / John / g" report.txt\u003e report_new.txt
Okay, you say, but where are the regular expressions here? Yes, we wanted to show an example first, and now the fun part begins.
If you are not sure if you wrote "nick" or "Nick" and want to accommodate both, you must use the sed command "s / Nick / nick / John / g". The vertical bar has a meaning that you should know if you have studied C, that is, your expression will match "nick" or "Nick". As you will see below, the channel can be used in other ways, but the meaning is the same. Other operators commonly used in regular expressions are "?", Which matches zero or one repetition of the preceding character (that is, flavou? R will match flavor and flavor), "*" zero or more times, "+" one or more times. "^" matches the beginning of a line, and "$" does the opposite. If you are a vi or vim user, many things will sound familiar to you. After all, these utilities, along with awk and C, have their roots in the early days of UNIX. We will not talk more on this topic, since it is easier to understand the meaning of these symbols with examples, but you should be aware that there are different implementations of regular expressions: POSIX, POSIX Extended, Perl, as well as different implementations of fuzzy regular expressions that guarantee you a headache ...
5. Examples of sed usage
| Command syntax | Description |
| sed "s / Nick / John / g" report.txt | Replaces every occurrence of Nick with John in report.txt |
| sed "s / Nick / nick / John / g" report.txt | Replaces each occurrence of Nick or nick with John. |
| sed "s / ^ / /" file.txt\u003e file_new.txt | Adds 8 spaces to the left of the text to improve print quality. |
| sed -n "/ Of course /, / attention you \\ pay / p "myfile |
Outputs one paragraph starting with "Of course" and ending with "attention you pay" |
| sed -n 12.18p file.txt | Only prints lines 12-18 of file.txt |
| sed 12,18d file.txt | Outputs the entire file file.txt except for lines 12 to 18 |
| sed G file.txt | Doubles spaces in file.txt |
| sed -f script.sed file.txt | Writes all commands to script.sed and executes them. |
| sed "5! s / ham / cheese /" file.txt | Replaces ham with cheese in file.txt except for line 5 |
| sed "$ d" file.txt | Removes the last line |
| sed "/ \\ (3 \\) / p" file.txt | Prints only lines with three consecutive digits |
| sed "/ boom /! s / aaa / bb /" file.txt | If "boom" is found, replace aaa with bb |
| sed "17, / disk / d" file.txt | Deletes all lines from line 17 to "disk" |
| echo ONE TWO / sed "s / one / unos / I" | Replaces one with unos case-insensitively, so "unos TWO" will be printed |
| sed "G; G" file.txt | Triple spaces in a file |
| sed "s /.$//" file.txt | A way to replace dos2unix :) |
| sed "s / ^ [^ t] * //" file.txt | Removes all spaces before every line in file.txt |
| sed "s / [^ t] * $ //" file.txt | Removes all spaces at the end of each line in file.txt |
| sed "s / ^ [^ t] * //; s / [^] * $ //" file.txt | Removes all spaces at the beginning and end of each line in file.txt |
| sed "s / foo / bar /" file.txt | Replaces foo with bar only at the first occurrence of a line. |
| sed "s / foo / bar / 4" file.txt | Replaces foo with bar only at the fourth occurrence of a line. |
| sed "s / foo / bar / g" file.txt | Replaces foo with bar for all occurrences of the string. |
| sed "/ baz / s / foo / bar / g" file.txt | Replace foo with bar only if the string contains baz. |
| sed "/./,/^$/!d" file.txt | Remove all consecutive blank lines except EOF |
| sed "/ ^ $ / N; / \\ n $ / D" file.txt | Remove all consecutive blank lines, but leave the top blank line. |
| sed "/./,$!d" file.txt | Remove all leading blank lines |
| sed -e: a -e "/ ^ \\ n * $ / ($ d; N;); / \\ n $ / ba" \\ file.txt |
Remove all trailing blank lines |
| sed -e: a -e "/ \\\\ $ / N; s / \\\\\\ n //; ta" \\ file.txt |
If the file ends with a backsplash, concatenate it with the following (useful for shell scripts) |
| sed "/ regex /, + 5 / expr /" | Matches regex plus 5 next lines |
| sed "1 ~ 3d" file.txt | Delete every third line starting from the first. |
| sed -n "2 ~ 5p" file.txt | Print every fifth line, starting with the second. |
| sed "s / ick / John / g" report.txt | Another way to write some of the above examples. Can you suggest yours? |
| sed -n "/ RE / (p; q;)" file.txt | Print only the first RE (regular expression) match |
| sed "0, / RE / (// d;)" file.txt | Removes only the first match |
| sed "0, / RE / s // to_that /" file.txt | Changes only the first match |
| sed "s / ^ [^,] *, / 9999, /" file.csv | Replaces the first field with 9999 in the CSV file |
| s / ^ * \\ (. * [^] \\) * $ // \\ 1 //; s / "*, * /" // g; : loop s // * \\ ([^ ", /] [^, /] * \\) *, * // \\ 1 // g; s // *, * // \\ 1 // g; t loop s / * //// g; s // * /// g; s / ^ / \\ (. * \\) / $ / \\ 1 /; | A sed script to convert a CSV file to a pipe delimited file (only works with some CSV types, with embedded quotes and commas). |
| sed ": a; s / \\ (^ \\ / [^ 0-9.] \\) \\ (\\ + \\) \\ (\\ (3 \\) \\) / \\ 1 \\ 2, \\ 3 / g; ta" file .txt | Changes the format of numbers in file.txt from 1234.56 to 1.234.56 |
| sed -r "s / \\<(reg/exp)+/\U&/g" | Converts any word beginning with reg or exp to uppercase. |
| sed "1.20 s / Johnson / White / g" file.txt | Replace Johnson with White on lines 1 - 20 only. |
| sed "1,20! s / Johnson / White / g" file.txt | The previous example is the other way around (replaces everywhere except lines 1-20) |
| sed "/ from /, / until / (s / \\ / magenta / g; \\ s / \\ / cyan / g;)" file.txt | Replaces only between "from" and "until" |
| sed "/ ENDNOTES: /, $ (s / Schaff / Herzog / g; \\ s / Kraft / Ebbing / g;)" file.txt | Replaces only from the word "ENDNOTES:" to EOF |
| sed "/./(H;$!d;);x;/regex/!d" file.txt | Print a paragraph only if it contains a regex |
| sed -e "/./(H;$!d;)" -e "x; / RE1 /! d; / RE2 /! d; / RE3 /! d" file.txt | Prints paragraphs only if they contain RE1, RE2 and RE3 |
| sed "s / 14" / fourteen inches / g "file.txt | This way you can use double quotes |
| sed "s / \\ / some \\ / UNIX \\ / path / \\ / a \\ / new \\ / path / g" file.txt | Working with Unix paths |
| sed "s /// g" file.txt | Removes all characters from a to g from file.txt |
| sed "s / \\ (. * \\) foo / \\ 1bar /" file.txt | Replaces only the last match of foo with bar |
| sed "1! G; h; $! d" | Replacing the tac command |
| sed "/\\n/!G;s/\\(.\\)\\(.*\\n\\)/&\\2\\1/;//D;s/.//" | Replacing the rev command |
| sed 10q file.txt | Head command replacement |
| sed -e: a -e "$ q; N; 11, $ D; ba" \\ file.txt | Tail command replacement |
| sed "$! N; /^\\(.*\\)\\n\\1$/!P; D" \\ file.txt | Replacing the uniq command |
| sed "$! N; s / ^ \\ (. * \\) \\ n \\ 1 $ / \\ 1 /; \\ t; D" file.txt | Reverse command (equivalent to uniq -d) |
| sed "$! N; $! D" file.txt | Equivalent to tail -n 2 |
| sed -n "$ p" file.txt | ... tail -n 1 (or tail -1) |
| sed "/ regexp /! d" file.txt | Equivalent to grep |
| sed -n "/ regexp / (g; 1! p;); h" file.txt | Prints the line before the first match of the regular expression, but not including the match |
| sed -n "/ regexp / (n; p;)" file.txt | Prints the line after the first match of the regular expression, but does not include the match itself |
| sed "/ pattern / d" file.txt | Deletes lines matching pattern |
| sed "/./!d" file.txt | Removes all blank lines from the file |
| sed "/ ^ $ / N; / \\ n $ / N; // D" file.txt | Deletes all consecutive blank lines except the first two |
| sed -n "/ ^ $ / (p; h;); /./ (x; /./ p;)" \\ file.txt | Removes the last line of each paragraph |
| sed "/ ^ $ / q" | Gets the email header |
| sed "1, / ^ $ / d" | Gets the body of the email |
| sed "/ ^ Subject: * /! d; s ///; q" | Gets the subject of the email |
| sed "s / ^ /\u003e /" | Quotes the message by inserting "\u003e" in front of each line |
| sed "s / ^\u003e //" | Reverse command (removes the quote from the message) |
| sed -e: a -e "s /<[^>] *\u003e // g; / | Removes HTML tags |
| sed "/./(H;d;);x;s/\\n/\u003d(NL)\u003d/g" file.txt / sort \\ / sed "1s / \u003d (NL) \u003d //; s / \u003d ( NL) \u003d / \\ n / g " | Sorts paragraphs in file.txt alphabetically |
| sed " [email protected]/ usr / [email protected]&/[email protected]"path.txt | Replaces / usr / bin with / usr / bin / local in path.txt |
| sed " [email protected]^.*[email protected]<<<&>\u003e\u003e @ g "path.txt | Try it and see :) |
| sed "s / \\ (\\ / [^:] * \\). * / \\ 1 / g" path.txt | Provided path.txt contains $ PATH, only prints the first path on each line |
| sed "s / \\ ([^:] * \\). * / \\ 1 /" / etc / passwd | Replacement awk - only show users from passwd file |
| echo "Welcome To The Geek Stuff" / sed \\ "s / \\ (\\ b \\) / \\ (\\ 1 \\) / g" (W) elcome (T) o (T) he (G) eek (S) tuff | Understandable without explanation |
| sed -e "/ ^ $ /, / ^ END / s / hills / \\ mountains / g" file.txt | Replaces "hills" with "mountains", but only in blocks of text starting with an empty line and ending with a line with three "END" characters, inclusive. |
| sed -e "/ ^ # / d" / etc / services / more | Shows the services file with no lines commented out |
| sed " [email protected]\([^:]*\):\([^:]*\):\([^:]*\)@\3:\2:\[email protected]"path.txt | Reverse the order of the elements in the last line of path.txt |
| sed "/ regex / (x; p; x;)" file.txt | Inserts a new line above each line that matches a regular expression |
| sed "/ AAA /! d; / BBB /! d; / CCC /! d" file.txt | Searches for AAA, BBB and CCC matches in any order |
| sed "/AAA.*BBB.*CCC/!d" file.txt | Searches for AAA, BBB and CCC matches in a given order |
| sed -n "/^.\\(65\\)/p" file.txt | Prints lines of 65 characters or more |
| sed -n "/^.\\(65\\)/!p" file.txt | Prints lines of 65 characters or less |
| sed "/ regex / G" file.txt | Inserts a blank line below each line |
| sed "/ regex / (x; p; x; G;)" file.txt | Inserts a blank line above and below each line |
| sed \u003d file.txt / sed "N; s / \\ n / \\ t /" | Numbers lines in file.txt |
| sed -e: a -e "s / ^. \\ (1.78 \\) $ / & /; ta" file.txt | Align text to the right |
| sed -e: a -e "s / ^. \\ (1,77 \\) $ / & /; ta" -e "s / \\ (* \\) \\ 1 / \\ 1 /" file.txt | Align text to center |
6. Conclusion
This is only a fraction of what could be said about sed, but this series of articles is more of a practical guide that we hope will help you appreciate the power of Unix utilities and make you more efficient.
