1. Preface

• As we all know, in Linux, everything is a file, such as configuration file, log file, startup file and so on. If we edit and query these files, we may think of some commands such as VI, VIM, cat and more. But these commands are not efficient. It's like a space to build a house. 10 masters were asked to dig the foundation with a shovel. It took a month to dig out. And another empty land was excavated. Three, five, and two were fix. That's efficiency. There are three models of "excavator" in Linux: awk on the top, sed in the middle and grep as standard. Using these tools, we can save a lot of repetitive work and improve efficiency on the premise of achieving the same effect.
• Next, let's take a look at the detailed description of sed
• Sed is the abbreviation of Stream Editor, which is called Stream Editor for short. What is flow? You can imagine the following assembly line. Sed is like a workshop. Each line of characters in the document are raw materials. They are transported to sed workshop, and then processed through a series of processing. Finally, they become goods from the assembly line.

 

 

• Of course, the contents of the file in the figure above can be from the file or directly from standard inputs such as keyboard or pipeline. The final result is displayed on the screen of the terminal by default, but it can also be output to the file.
• In the past, when there was no assembly line in the factory, the production of a commodity needed more than a dozen types of work to cooperate with each other, so the profit was too low. Later, there was an assembly line. Although there were still more than a dozen processes for the production of a commodity, they were mechanized production, and the workers only played an auxiliary role, so the profit and output were greatly improved.
• The same is true for editing files. In the past, when we modified a configuration file, we needed to move the cursor to one line, then add some text, and then move the cursor to another line to comment... It may take tens of minutes to modify a configuration file, and we may have to rework the wrong configuration file. This is also a configuration file. What if there are dozens or hundreds? Therefore, when you learn the SED command, you will find it very useful to use it to deal with a series of changes in the file. As long as you think that processing 20 different editing operations in about 100 files can be completed in a few minutes, you will know the power of sed.

2. Software function and version

• The Sed command is a powerful tool for manipulating, filtering and converting text content. Common functions include adding, deleting, modifying and querying (adding, deleting, modifying and querying). The two most commonly used functions of query are filtering (filtering specified strings) and fetching rows (fetching specified rows).
• The sed version we are going to learn now is the open source version of GNU. My experimental environment is CentOS 6.8 system, and the kernel version is 2.6.32-642.el6.x86_ sixty-four

[root@chensiqi1 ~]# cat /etc/redhat-release 
CentOS release 6.8 (Final)
[root@chensiqi1 ~]# uname -r
2.6.32-642.el6.x86_64
[root@chensiqi1 ~]# sed --version   # View sed software version
GNU sed version 4.2.1

3. Syntax format

sed [options] [sed -commands][input -file]
sed [option]  [sed Command [input file]

explain:
1. Pay attention to sed software and the following options, sed command and input file. There is at least one space between each element.
2. In order to avoid confusion, sed is called sed software in this paper. Sed - commands are built-in command options in sed software. In order to distinguish them from the previous options, they are called sed commands
3. sed -commands can be either a single sed command or a combination of multiple sed commands.
4. input -file is optional. sed can also obtain input from standard input such as pipeline.

4. Command execution process

Outline the process:

Sed software reads one line from the file or pipeline, processes one line and outputs one line; Read one more line, process one more line, and output one more line

Tips:

The design of one line at a time makes the performance of SED software very high. Sed won't appear Caton's imagination when reading very large files. Everyone has used the vi command. If you use the vi command to open a file of tens of M or larger, you will find a jamming phenomenon. This is because the vi command loads the file into memory at one time and then opens it again. Therefore, the length of jamming depends on the reading speed from disk to memory. And if the file is too large, it will also cause memory overflow. Sed software avoids this situation very well. It opens very fast and executes very fast.

Detailed process:

There is a file person.txt with five lines of text. The sed command reads the first line "101, chensiqi,CEO" of the file person.txt and stores this line of text in the mode space (a temporary cache in the memory of the sed software is used to store the read content, which is compared to the conveyor belt of the factory assembly line.)

The complete processing flow of the file person.txt in the pattern space

1. Judge whether line 1 is the line to be processed. If it is not the line to be processed, read the next line from the file again. If it is the line to be processed, then go down.
2. Execute the sed command on the contents of the pattern space, such as a (append), i (insert), s (replace)
3. Output the content processed by sed command in the mode space to the screen, and then clear the mode space
4. Read the next line of text, and then re execute the above process until the end of the file

The above process is summarized as follows:

 

  

Sed software has two built-in storage spaces:

• Pattern space: it is a buffer that the sed software reads a line of text from the text and stores it (this buffer is in memory), and then uses the sed command to operate the contents of the pattern space.
• Hold space: it is another buffer of SED software, which is used to store temporary data. It is also in memory, but the purpose of mode space and hold space is different. Sed can exchange data in the holding space and the mode space, but it cannot execute the ordinary sed command in the holding space, that is, we can store data in the holding space.

5. Description of options

6. Use examples

6.1 unified experimental text

In order to better test the usage of the sed command, we need to prepare the following test files.

[root@chensiqi1 ~]# cat >person.txt<<KOF
> 101,chensiqi,CEO
> 102,zhangyang,CTO
> 103,Alex,COO
> 104,yy,CFO
> 105,feixue,CIO
> KOF          #Kofs must appear in pairs to indicate termination of input
 Command description: use one cat Command to create multiline text. The file contains the above contents. This file will be used for subsequent operations.

6.2 common functions - add, delete, modify and query

Next, we begin to learn the "four axes" of sed command - > add, delete, change and check. Don't worry, let's learn one by one, and eat hot tofu slowly.

6.2.1 add

• Next, I'll teach you the first move -- > append or insert the specified text to the specified position of the file.
• This function is very useful. For example, we usually write a few lines of text to the configuration file. The most commonly used commands are vi or vim commands, but these two commands are interactive commands. We also need to enter a string in the vi/vim editor interface, save and exit. The operation is cumbersome, but it can be used. But when we learn Shell scripts, we will find that vi or vim commands cannot be used normally in scripts. Why? Please experience by yourself.
• We learn Shell script mainly to liberate our hands, execute a script, and then automatically write data to the file. We don't need to do it again. So we thought of sed software, which can help us achieve our goals.
• Here we need to use two sed commands:
  "A": after appending the text to the specified line, memory method: the full spelling of a is apend, which means appending.
  "i": before inserting text into the specified line, memory method: the full spelling of i is insert, which means insert.

6.2.1.1 single line addition

First, let's take a look at the usage of adding a single line. To put it bluntly, it means adding a line of text to the file. We learned the echo command before. You can add text at the end of the file. It's relatively simple, but we also have other complex requirements, such as inserting a line of numbers in line 10. Here we need sed. Let's take a look at the following examples, and students follow the examples Practice together. Commands will only be familiar if you practice more. You can never learn by watching alone.

[root@chensiqi1 ~]# sed '2a 106,dandan,CSO' person.txt
101,chensiqi,CEO
102,zhangyang,CTO
106,dandan,CSO   #This is the new sentence
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command line explanation:

First, let's look at the result of the command. We can see that a new line "106, dandan, CSO" is inserted before the original second line "102, Zhang Yang, CTO", and the original second line becomes the third line.
Next, let's interpret the structure of the SED statement, starting with sed and then followed by a space (there is no limit to the number of spaces, but at least one!). After the space, we first type a pair of single quotation marks (") and then backspace to negotiate '106, dandan, CSO' in the single quotation marks.

• 2 specifies the operation on line 2, and other lines are ignored
• i stands for insertion, 2i is to insert text before line 2
• 2i followed by a space, and then follow the text you want to insert

 

 

Finally, connect the file person.txt you want to process

6.2.1.2 summary of differences between quotation marks

In the process of teaching, some students asked me whether I use single quotation marks or double quotation marks in sed? Here I'll tell you the difference between quotation marks in detail.

1. Double quotation marks: output the contents of double quotation marks; if there are commands, variables, etc. in the contents, the results of commands and variables will be parsed first, and then the final contents will be output. The writing method of commands or variables in double quotation marks is ` command or variable 'or $(command or variable)
2. Single quotation mark: WYSIWYG. Output the contents in the single quotation mark as they are, and prevent the escape of all characters
3. No quotation marks: the string containing spaces will not be regarded as an overall output. If there are commands and variables in the content, the command and variables will be parsed out first, and then the final content will be output. If the string contains special characters such as spaces, it cannot be completely output, so double quotation marks need to be added.
4. Inverted quotation mark (below the back quotation mark Esc key): replace the command. The shell command inside the inverted quotation mark will be executed, and the result output will replace the text enclosed in inverted quotation marks.

Why did Sed use single quotation marks?

[root@chensiqi1 ~]# cat person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@chensiqi1 ~]# sed '2i $PATH' person.txt  #Single quotation mark -- text content is inserted intact
101,chensiqi,CEO
$PATH
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@chensiqi1 ~]# sed 2i $PATH person.txt   #Without quotation marks, linux cannot recognize spaces and will not execute commands with spaces as a command
sed: -e expression #1, char 2: expected \ after `a', `c' or `i'
[root@chensiqi1 ~]# sed "2i $PATH" person.txt #Double quotation marks -- the variable $PATH is inserted as text after being parsed
101,chensiqi,CEO
/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Conclusion 1:

• If there is an ordinary string in quotation marks, you can use single quotation marks or double quotation marks at will:
• If there are variables or commands with backquotes in quotation marks, if you want the result of variable parsing or command execution, use double quotation marks; if you want the original string in quotation marks, use single quotation marks.
• As long as you understand the above usage and what you want, you can use whatever quotation marks you want.
• In fact, choose one of two, either or. If this doesn't work, try that.

Conclusion 2:

1. Many times, the conclusion may not be suitable for all situations. However, summarizing small conclusions is convenient for us to learn knowledge and deeply understand knowledge. Therefore, we recommend you to make more summaries.
2. Don't be too obsessed with the conclusion. After all, the conclusion is not the truth, but the product of our learning process. Just like the parallel line in mathematics, in junior high school, the definition is that the two disjoint lines are parallel lines. In senior high school, the conclusion is wrong.

6.2.1.3 adding multiple lines

The previous learning has realized the addition or insertion of single line text to the file, but there is also the need to insert multiple lines of text. We also learned that the cat command can add multiple lines of text to the file.

For example preparation, execute the following complete command to generate the contents of test.txt file:

[root@chensiqi1 ~]# cat > test.txt <<EOF
> welcome to my blog.http://www.cnblogs.com/chensiqiqi/
> 
> if you like my blog\'s contents,pls support me.
> 
> 
> 
> bye!boys and girls.
> EOF
 Command description: the above is cat Please do not ignore the blank line above for the common methods of generating or adding content to files in the production environment of. In addition, if there are single quotation marks in the content, $Symbols and other special symbols shall be used as content“\"Escape.

Execution process and results of the above commands:

[root@chensiqi1 ~]# cat > test.txt <<EOF

> welcome to my blog.http://www.cnblogs.com/chensiqiqi/
> #This line is empty
> if you like my blog\'s contents,pls support me.
> #This line is empty
> #This line is empty
> #This line is empty
> bye!boys and girls.
> EOF #-->In addition, EOF must appear in pairs, but it can also be replaced with other paired labels, such as chensiqi character label.
Command description: the leftmost part of the command result above“>"yes Linux The system generates it automatically, not manually.

• ok, we have briefly reviewed the usage of cat command. If you want to know more about the detailed usage of cat command, please continue to pay attention to my blog.
• Previously, we implemented the cat command to add multiple lines of text to the file, but the cat command and echo command have the same disadvantages. They can only add content to the end of the text and cannot operate on the specified line. Therefore, we still need our sed command.
• First, we need to understand a special symbol "\ n", which is called line feed, as the name suggests. Let's take a few examples to understand this symbol.

[root@chensiqi1 ~]# echo "chensiqi";echo "chensiqi"
chensiqi
chensiqi

Command description: the above command connects two commands with a semicolon ";, and the output result is 2 lines chensiqi，But is there a simpler way?
[root@chensiqi1 ~]# echo -e "chensiqi\nchensiqi"
chensiqi
chensiqi
 Command description: here it is“\n"It comes in handy. There is a difference between lines“\n"As a separator, so“ chensiqi\nchensiqi"It is equivalent to 2 lines chensiqi. Next we use echo Command experiment, in which-e Parameter indicates if the following special characters appear in the string(\n For line feed,\t representative Tab Key, etc.) will be specially processed instead of being output as general text. If you want to know more about it echo For the detailed usage of the command, please continue to follow my blog.

After learning about line breaks, let's take a look at how the sed command is used
!

[root@chensiqi1 ~]# sed '2a 106,dandan,CSO\n107,bingbing,CCO' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
106,dandan,CSO  #Append multiple lines at the same time
107,bingbing,CCO  #Append multiple lines at the same time
103,Alex,COO
104,yy,CFO
105,feixue,CIO
EOF

Command line analysis:

• We first glance at the command statement. We can find that the command structure is almost the same as adding text in a single line.
• Then let's look at the result of the command. We can see that two lines of text "106, dandan, CSO" and "107, bingbing, CCO" are added after the original second line "102, Zhang Yang, CTO".
• Next, let's interpret the structure of the sed statement. The sed software starts with a blank space. After the blank space, we first type a pair of single quotation marks ("), then back off and write '2a, 106, Dandan, CSO \ n107, Bingbing, CCO' in the single quotation marks.

1. 2 means to specify the operation on line 2, and other lines are ignored;
2. A stands for addition, 2a is to add text after line 2;
3. 2a followed by a space, and then keep up with the multiple lines of text you want to insert. Each line of text here can be written into one line by using "\ n" connection.

 

 

Finally, connect the file person.txt you want to process

Of course, there is another way to add multiline text, but this method is not as convenient as "\ n", so let's talk about it briefly here. This method uses "\" , it also means line feed. If you write on one line when executing a long command, it's too ugly and difficult to understand. Therefore, you can use this symbol to divide a complete command into multiple lines. For example:

[root@chensiqi1 ~]# echo chensiqi
chensiqi
[root@chensiqi1 ~]# echo \
> chensiqi
chensiqi
[root@chensiqi1 ~]# echo \ 
> chensiqi \
> is \
> me
chensiqi is me

Command description: a command can be\Multi line output of symbols

Next, the SED command uses a backslash. First, enter "sed '2a, 106, Dandan, CSO \", and then press enter. In this way, a symbol ">" will be displayed in the window. After this symbol, we will write the rest of the command "107, bingbing, CCO' person.txt".

[root@chensiqi1 ~]# sed '2a 106,dandan,CSO \ 
> 107,bingibng,CCO' \  
> person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
106,dandan,CSO 
107,bingibng,CCO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
EOF
 Command description: the result is the same as the first method, but we also find this method more troublesome, so we suggest you use the first method.

• sed software uses the command i to insert multiple lines of text. The usage of the command and a is the same, so it is not listed in detail here. You can practice by replacing "a" with "i" in the previous command.
• After learning so much, there is always a place to be useful, so don't look at the answers. First complete the following enterprise cases alone!

Enterprise case 1: optimize SSH configuration (add several parameters with one click)

When learning about CentOS6 system optimization, there is an optimization point: change the remote login configuration of ssh service. The main operation is to add the following 5 lines of text to the ssh configuration file / etc/ssh/sshd_config. (see other courses for the specific meaning of the following parameters.)

Port 52113
PermitRootLogin no
PermitEmptyPasswords no
UseDNS no
GSSAPIAuthentication no

Of course, we can use the vi/vim command to edit this text, but it is more troublesome. Now we want to add 5 lines of text to the front of line 13 with a command?
Note: do not forget to back up the configuration file before modifying: cp /etc/ssh/sshd_config{,.bak}
If you don't understand the above command knowledge, please read the wildcard chapter

This enterprise interview question can be solved with the multi line addition function of sed command we have learned.

[root@chensiqi1 ~]# sed -i '13i Port 52113\nPermitRootLogin no\nPermitEmptyPasswords no\nUseDNS no\nGSSAPIAuthentication no' /etc/ssh/sshd_config

Command description: if the title requires to be inserted before line 13, you need to use command 13 i. When some students do a topic, they think so. Before line 13, isn't that after line 12, line 12 a Yes, that's right. This can be regarded as the second method.
The last 5 lines inserted are used“\n"It can become a line.
There is another option not mentioned above"-i",This option can actually modify the file content, which can be removed during practice to prevent changing the configuration file. If it is used-i，You can restore with a backup file. Of course, if you modify the configuration file in the production environment, you need to use-i Options.
[root@chensiqi1 ~]# sed -n '13,17p' /etc/ssh/sshd_config
Port 52113
PermitRootLogin no
PermitEmptyPasswords no
UseDNS no
GSSAPIAuthentication no

Command description: view the added text content and select options-n And command p See 6 below for the specific usage of.2.4 Check.

6.2.2 delete

• I showed you the first move. Is it easy to use? If you don't understand it, please read it again. If you learn it, follow me to learn the second move -- > delete the specified line of text.
• This function is also very useful. For example, we want to delete some lines in the file. In the past, the most commonly used command was vi or vim, but now that we know the sed command, we should use this high-level command to complete the task.
• Here we need to use a sed command;
• "d": delete the text. Memory method: the complete spelling of d is delete, which means delete.
• Because the deletion function is relatively simple, we will explain it together with the address range. We learned earlier that the sed command can target a line of text (add one or more lines of text before and after a single line). Next, let's see how to target multiple lines of text.

6.2.2.1 address range specified for execution

sed software can process single line or multi line text. If you do not specify an address range before the sed command, all lines are matched by default.
Usage: n1[,n2]{sed -commands}
Addresses are separated by commas. N1 and N2 can be represented by numbers, regular expressions, or a combination of the two.

Let's use a specific example to demonstrate that the test file is still person.txt

[root@chensiqi1 ~]# sed 'd' person.txt 
[root@chensiqi1 ~]# 

Command description: if sed If no address range is specified before the command, all lines will be matched by default, and then d The command delete function will delete all the contents of this file

 

 

 

 

[root@chensiqi1 ~]# sed '2d' person.txt 
101,chensiqi,CEO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: this single line deletion must be understood. Specify to delete the text 102 in line 2, zhangyang，CTO

 

 

[root@chensiqi1 ~]# sed '2,5d' person.txt 
101,chensiqi,CEO

Command description:'2，5d'Specify to delete lines 2 to 5, d Represents a delete operation.

 

 

ok, we finished experimenting with the digital address range above. Next, we experimented with the address range of regular expressions. Although regular expressions can be used, we are still used to writing complete matching strings to achieve the purpose of accurate matching.

[root@chensiqi1 ~]# sed '/zhangyang/d' person.txt 
101,chensiqi,CEO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description:
    stay sed In the software, regular formats and awk Same, use 2 "/"Contains the specified regular expression, i.e“/regular expression /". 

 

 

Of course, you can also use two regular expressions, as shown in the following example.

[root@chensiqi1 ~]# sed '/chensiqi/,/Alex/d' person.txt 
104,yy,CFO
105,feixue,CIO

Command description: This is a multi line deletion in the form of regular expression, and two addresses are separated by commas. The final result is to delete the address containing“ chensiqi"Rows to include“ Alex"Line of

 

 

[root@chensiqi1 ~]# sed '3,$d' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO

Command description: after learning regular expressions, we know“ $"Represents the end of the line, but in sed There are some changes in the“ $"stay sed Therefore, the meaning of this example is to delete the text from line 3 to the last line, including line 3 and the last line, so the contents of lines 1 and 2 are left.

 

 

Next, let's look at some special cases.

[root@chensiqi1 ~]# sed '/chensiqi/,3d' person.txt 
104,yy,CFO
105,feixue,CIO

Command description: this example is to delete the“ chensiqi"But this combination has a special case. If there is this in addition to the first three lines“ chensiqi"Words, sed The software will still find him "trouble". Please see the following example.

[root@chensiqi1 ~]# sed '$a 106,chensiqi,CMO' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
106,chensiqi,CMO

Command description: in order not to cause the students to disagree with the experimental text, I just temporarily modify the memory data with the above command statement, and then pass it to the user through the pipeline symbol sed Software.

[root@chensiqi1 ~]# sed '$a 106,chensiqi,CMO' person.txt | sed '/chensiqi/,3d'
104,yy,CFO
105,feixue,CIO

Command description: from the command result, we can see that not only the first line (101, chensiqi，CEO)To line 3 (103, ALex，COO)Deleted, and the last line (106, chensiqi，CMO)Also deleted. So we can draw a small conclusion, sed The software uses regular expressions to find all matching lines, even if there are numeric address restrictions.

Let's look at a special case.

[root@chensiqi1 ~]# sed '2,/O/d' person.txt 
101,chensiqi,CEO
104,yy,CFO
105,feixue,CIO

Command description:
    Delete from line 2 to include letters O The line ends, but we find that lines 3, 4 and 5 contain letters O，The command results show that only lines 2 and 3 are deleted, which belongs to the shortest deletion. How do you understand this?
    It can also be understood from the above command execution flowchart that the loop starts from line 2, sed The first time the software encounters a letter O(The third line) thinks the loop is over.
[root@chensiqi1 ~]# sed '2,/o/d' person.txt 
101,chensiqi,CEO

Command description:
    Delete from line 2, but there are no letters after the text O，So the loop goes on until the end of the text, sed The software terminates automatically.

After reading the above example, it's actually very simple. In our work, we most often use the exact matching method of digital address. Fuzzy matching such as regular address or mixed address above is rarely used. Just understand it.

6.2.2.2 analysis of special symbols ~ (step size)

Format: "First to step" means to start from and increase by step. This is called an equal difference sequence in mathematics

example:

• 1 ~ 2 matches 1, 3, 5, 7..... #-- > to output only odd lines. Please carefully observe the difference of each number.
• 2 ~ 2 matches 2, 4, 6, 8.... #-- > used to output only even rows
• 1 ~ 3 match 1, 4, 7, 10
• 2 ~ 3 match 2, 5, 8, 11

[root@chensiqi1 ~]# seq 10
1
2
3
4
5
6
7
8
9
10

Command description: seq The command can generate a sequence of numbers from 1 to 10.

[root@chensiqi1 ~]# seq 10 | sed -n '1~2p'
1
3
5
7
9

Command description:
    The above command mainly validates special symbols“～"Other effects sed Command usage n and p Please refer to the following text for details. You only need to know that this command can change "1"～2"The specified line is displayed.
    The above example tests "1"～2"You can also test the effect of "2" manually～2"，"1～3"，"2～3"，See if their results match the arithmetic sequence.

Supplementary knowledge:

If you want to generate an odd sequence, in fact, the above method is for example, which is not a good method, because the seq command has this function.

[root@chensiqi1 ~]# seq 1 2 10
1
3
5
7
9

Command description: seq Command format seq Start tolerance end value

Another example:

[root@chensiqi1 ~]# sed '1~2d' person.txt 
102,zhangyang,CTO
104,yy,CFO

Command description: "1～2"This is another format for specifying the number of rows. Starting from the first row, the rows (1, 3, 5) are incremented by step 2. Therefore, delete rows 1, 3, 5, that is, all odd rows.

6.2.2.3 special symbols + analysis

[root@chensiqi1 ~]# sed '1,+2d' person.txt  
104,yy,CFO
105,feixue,CIO

Command description: This is actually an addition operation,'1，+2d'==>Delete lines 1 to 3 (1+2)Line text.

The above is the usage of the special symbol "+", as you know.

6.2.2.4 analysis of special symbols

Exclamation point "!" we have been in contact with many commands, most of which are negative, and sed is no exception.

[root@chensiqi1 ~]# sed '2,3!d' person.txt 
102,zhangyang,CTO
103,Alex,COO

Command description: add after address range "2, 3"“
!"，If you do not add "!" to delete line 2 and line 3, the result is as shown in the following example. Then add "!" to delete everything except line 2 and line 3. This method can be used as a supplementary method to display the title of line 2 and line 3 of the file.

[root@chensiqi1 ~]# sed '2,3d' person.txt 
101,chensiqi,CEO
104,yy,CFO
105,feixue,CIO

Enterprise case 2: print the contents of the file without chensiqi

This is an enterprise interview question. It's very simple. Just delete the line containing chensiqi string. We can use "grep -v" to get the desired result, but here we use sed software.

[root@chensiqi1 ~]# sed '/chensiqi/d' person.txt 
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: delete include“ chensiqi"The regular matching string is used directly chensiqi that will do

6.2.3 modification

Now you have learned the two moves of sed software, isn't it very powerful! Of course, there are more powerful -- > changes behind. When learning linux, our most common operations are to change the configuration file, parameters, etc., and what's more, the addition and deletion we learned earlier can be realized in disguise with the modifications we are going to learn.

6.2.3.1 replace by line

First, let's talk about replacing by line. This function is rarely used, so you can understand it. The sed command used here is:
"c": replace the old line with the new line. Memory method: the complete spelling of c is change, which means replacement.

example:

[root@chensiqi1 ~]# sed '2c 106,dandan,CSO' person.txt 
101,chensiqi,CEO
106,dandan,CSO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: Using sed command c Replace the original line 2 with "102, zhangyang，CTO"Replace with "106, dandan,CSO",Whole line replacement

6.2.3.2 text substitution

• Students with working experience should be very familiar with this function, because 80% of the scenes using sed software use the replacement function.
• The sed command used here has the following options:
  "s": use -- > alone to replace the first matching string in each line = = > sed command
  "g": replace all lines -- > one of the replacement flags of SED command s (global replacement), non sed command.
  "- i": modify the file content -- > the option of SED software. Note the difference from sed command i.

sed software replacement model

Sed - I's / target content / replacement content / g 'chensiqi.log
Sed - I's # target content # replace content #g '

Observation characteristics

1. There are quotation marks on both sides. The two sides in the quotation mark are s and g respectively. In the middle are three same characters / or # as delimiters. The character # can contain characters in the replacement content / help distinguish. The delimiter can be any character, such as: or | etc., but when the replacement content contains delimiters, it needs to escape \: or |. After long-term practice, it is recommended that you use # as delimiters.
2. Delimiter / or #, between the first and the second is the replaced content, and between the second and the third is the replaced content.
3. s# target content # replacement content # the "target content" can use regular expressions, but the replacement content cannot be used. It must be specific. Because if the replacement content uses regular expressions, the sed software will not know what to do, and it does not know what content you want to replace.
4. The default sed software operates on the mode space (data in memory), and the - i option will change the file content on the disk.

example:

[root@chensiqi1 ~]# sed 's#zhangyang#dandan#g' person.txt 
101,chensiqi,CEO
102,dandan,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: text to be replaced“ zhangyang"On the first and second“#"Between, the replaced text“ dandan"Put it in the second core and the third“#”Replace "Zhang Yang" in the second line with "Dan Dan".

Modify file:

[root@chensiqi1 ~]# cat person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: we know from the result of the above command sed The command does not modify the contents of the file by default

[root@chensiqi1 ~]# sed -i 's#zhangyang#dandan#g' person.txt

Command description: if you want to really modify the contents of the file, you need to use the option“-i"，This should be with sed Command“ i"At the same time, we can find that there is no output after the command is executed.

[root@chensiqi1 ~]# cat person.txt 
101,chensiqi,CEO
102,dandan,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: when we check this file again, we will find that the file has been modified successfully. Therefore, if you use the replacement function in the future, you should not use the option first“-i"Test it to ensure correct operation, and finally use it“-i"Modify file

Restore test files:

[root@chensiqi1 ~]# sed -i 's#dandan#zhangyang#g' person.txt

Command description: restore the test file. Don't forget to execute this step, otherwise you won't be able to follow the steps

Enterprise case 3: specifying a line to modify a profile

The model we learned earlier can operate all the qualified text in the file, but we will also encounter the need to accurately modify the configuration file on the specified line, because it can avoid too many modifications.

[root@chensiqi1 ~]# sed '3s#0#9#' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
193,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description:
    The previous examples are sed Command“ s"There is no address range specified before, so all rows are operated by default.
    In this case, only 0 in line 3 is required to be replaced by 9. Here we use the address range knowledge we learned earlier. In sed Command“ s"Add "3" before it to replace line 3

6.2.3.3 variable replacement

Variable substitution is actually the same as the previous text substitution, that is, the specific text becomes a variable, and everyone is required to have a clear understanding of the use of quotation marks. Therefore, students who do not understand the difference between double quotation marks and single quotation marks, please turn the page to [summary of quotation marks] and review it again.

[root@chensiqi1 ~]# cat >test.txt<<KOF #Create a new simple test text
> a
> b
> a
> KOF
[root@chensiqi1 ~]# cat test.txt 
a
b
a
[root@chensiqi1 ~]# x=a #->Set the variable x and assign a value
[root@chensiqi1 ~]# y=b #->Set variable y and assign value b
[root@chensiqi1 ~]# echo $x
a
[root@chensiqi1 ~]# echo $x $y
a b

Command description: print variables x，y Verify that you need to use $Symbolic reference variable

Do not use quotation marks:

[root@chensiqi1 ~]# sed s#$x#$y#g test.txt 
b
b
b

Command description: use variables to replace. From the execution results, we can find that the replacement is successful, test.txt All in the file a All replaced with b. At the same time, we can find s#$x#$y#g does not use quotation marks. Of course, this writing method is not a special standard.

[root@chensiqi1 ~]# sed 's#'$x'#'$y'#g' test.txt 
b
b
b

Command description: it seems that single quotation marks can be used on the surface, but in fact, a blind trick is used here to segment in your eyes'$x'and'$y',But in fact's#'and'#'and'#g ', so $x and $y are not enclosed in quotation marks, just like the above example.

Use the eval command:

[root@chensiqi1 ~]# eval sed 's#$x#$y#g' test.txt 
b
b
b

Command description: here is an extension for you Linux Built in command eval，This command can read in variables, combine them into a new command, and then execute it eval Can resolve variables $x And variables $y，The final effect is the same as that of double quotation marks.

Extension: the fastest way to obtain IP address

[root@chensiqi1 ~]# hostname -I
192.168.197.133 

6.2.3.4 instructions for group replacement () and \ 1

The () function of sed software can remember part of regular expressions. Among them, \ 1 is the first remembered pattern, that is, the matching content in the first parenthesis, \ 2 is the second remembered pattern, that is, the matching content in the second parenthesis. sed can remember up to 9.

Example: echo "I am chensiqi teacher." if you want to keep the word chensiqi in this line, delete the rest and mark the part you want to keep with parentheses.

[root@chensiqi1 ~]# echo "I am chensiqi teacher." | sed 's#^.*am \([a-z]\+\) tea.*$#\1#g'
chensiqi

[root@chensiqi1 ~]# echo "I am chensiqi teacher." | sed -r 's#^.*am ([a-z]+) tea.*$#\1#g'
chensiqi

[root@chensiqi1 ~]# echo "I am chensiqi teacher." | sed -r 's#I (.*) (.*) teacher.#\1\2#g'
amchensiqi

Command description:
sed If not-r Suffix, extension of regular expressions is not supported by default, and\The function of parentheses is to group the matching contents in parentheses so that they can be escaped in the 2nd and 3rd#The reverse reference of sed between the numbers, \ 1 refers to the first group, and 2 refers to the second group

Let's look at another topic: please execute the command to get the IP address of eth0 in linux?

[root@chensiqi1 ~]# ifconfig eth0 | sed -n '2p'
          inet addr:192.168.197.133  Bcast:192.168.197.255  Mask:255.255.255.0
[root@chensiqi1 ~]# ifconfig eth0 | sed -n '2p' | sed -r 's#^.*addr:(.*) Bcast:.*$#\1#g'
192.168.197.133 

It can also be combined

[root@chensiqi1 ~]# ifconfig eth0 | sed -rn '2s#^.*addr:(.*) Bcast:.*$#\1#gp'
192.168.197.133 

Command description:
This problem needs to be solved ifconfig eth0 Of line 2 of the execution result IP Take out the address. The idea of the above answer is to use IP Address to replace line 2.

Enterprise case 4: system startup item optimization (using sed)

[root@chensiqi1 ~]# chkconfig --list | egrep -v  "sshd|crond|rsyslog|sysstat|network" | awk '{print $1}' |sed -r 's#^(.*)#chkconfig \1 off#g' |bash

This question can also be applied awk Directly in one step
[root@chensiqi1 ~]# chkconfig --list | egrep -v "sshd|crond|network|rsyslog|sysstat" | awk '{print "chkconfig",$1,"off"}' | bash

6.2.3.5 special symbols & represent the replaced content

This is a special skill, which is especially convenient to use in suitable scenes. The special symbol "&" is used together with grouping replacement for comparison.

[root@chensiqi1 ~]# sed -r 's#(.*),(.*),(.*)#& ----- \1 \2 \3#' person.txt 
101,chensiqi,CEO ----- 101 chensiqi CEO
102,zhangyang,CTO ----- 102 zhangyang CTO
103,Alex,COO ----- 103 Alex COO
104,yy,CFO ----- 104 yy CFO
105,feixue,CIO ----- 105 feixue CIO

Command description:
    1，Here we will group and replace&Put the symbols together for comparison
    2，The grouping substitution in the command uses three parentheses, each representing each row and each column separated by a comma.
    3，Ordered above&The symbol represents each line, that is, in the model's#Target content#replace content#Target content of g '.

Enterprise case 5: batch renaming files
The files in the current directory are as follows:

[root@chensiqi1 chen]# find ./ -name "*_finished.jpg"
./stu_102999_1_finished.jpg
./stu_102999_5_finished.jpg
./stu_102999_3_finished.jpg
./stu_102999_2_finished.jpg
./stu_102999_4_finished.jpg

It is required to rename with the sed command. The effect is
stu_ 102999_ 1_ finished.jpg==>stu_ 102999_ 1. JPG, that is, delete the file name_ finished

Problem solving idea: because this is a file name, it cannot be directly replaced by yongsed command. Therefore, it also needs to be renamed with the help of mv command. The format is mv stu_102999_1_finished.jpg stu_102999_1.jpg. We need to piece together such a format, and then use the bash command to execute it.

[root@chensiqi1 chen]# find ./ -name "*_finished.jpg" | sed -r 's#^(.*)_finished(.*)#\1\2#g'
./stu_102999_1.jpg
./stu_102999_5.jpg
./stu_102999_3.jpg
./stu_102999_2.jpg
[root@chensiqi1 chen]# find ./ -name "*_finished.jpg" | sed -r 's#^(.*)_finished(.*)#& \1\2#g' 
./stu_102999_1_finished.jpg ./stu_102999_1.jpg
./stu_102999_5_finished.jpg ./stu_102999_5.jpg
./stu_102999_3_finished.jpg ./stu_102999_3.jpg
./stu_102999_2_finished.jpg ./stu_102999_2.jpg
./stu_102999_4_finished.jpg ./stu_102999_4.jpg
[root@chensiqi1 chen]# find ./ -name "*_finished.jpg" | sed -r 's#^(.*)_finished(.*)#mv & \1\2#g' 
mv ./stu_102999_1_finished.jpg ./stu_102999_1.jpg
mv ./stu_102999_5_finished.jpg ./stu_102999_5.jpg
mv ./stu_102999_3_finished.jpg ./stu_102999_3.jpg
mv ./stu_102999_2_finished.jpg ./stu_102999_2.jpg
mv ./stu_102999_4_finished.jpg ./stu_102999_4.jpg
[root@chensiqi1 chen]# find ./ -name "*_finished.jpg" | sed -r 's#^(.*)_finished(.*)#mv & \1\2#g' |bash
[root@chensiqi1 chen]# ls
stu_102999_1.jpg  stu_102999_2.jpg  stu_102999_3.jpg  stu_102999_4.jpg  stu_102999_5.jpg

Command description:
1."\1"Represents the front“(^.*)"Matching content“&"Represent“ s# #”For the replaced content in, the complete file name is matched here.
2.use bash Command execution, bash The command executes the standard input statement, just as we hit enter after entering the statement on the command line.

6.2.4 check

• After learning this, you can take a breath, because we have learned the most commonly used and important in sed. Next, we can easily learn the last move -- > view the text
• This function is also very useful. For example, we want to view some lines in the file. In the past, the most commonly used commands were cat, more or less commands, but these commands have some disadvantages, that is, they can't view the specified lines. We have this function with the SED command which has been used for a long time. Moreover, we said earlier that using sed is faster than other commands, such as vim!
• Here we need to use a sed command
• "p": output the specified content, but the matching results will be output twice by default. Therefore, use the - n option to cancel the default output. Memory method: the full spelling of p is print, which means print.

6.2.4.1 query by line

[root@chensiqi1 ~]# sed '2p' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@chensiqi1 ~]# sed -n '2p' person.txt 
102,zhangyang,CTO

Command description: Options-n Cancel the default output and only output the matching text. You only need to remember to use the command p Required option-n. 

[root@chensiqi1 ~]# sed -n '2,3p' person.txt 
102,zhangyang,CTO
103,Alex,COO

Command description: View lines 2 to 3 of the file and use the address range "2,3". Just take the line sed，Simplest

[root@chensiqi1 ~]# sed -n '1~2p' person.txt 
101,chensiqi,CEO
103,Alex,COO
105,feixue,CIO

Command description: print lines 1, 3 and 5 of the file.～Representative step size

[root@chensiqi1 ~]# sed -n 'p' person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: do not specify the address range, and print all contents by default.

6.2.4.2 query by string

[root@chensiqi1 ~]# sed -n '/CTO/p' person.txt 
102,zhangyang,CTO

Command description: print with CTO Line of

[root@chensiqi1 ~]# sed -n '/CTO/,/CFO/p' person.txt 
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO

Command description: print with CTO Row to include CFO OK.

6.2.4.3 hybrid query

[root@chensiqi1 ~]# sed -n '2,/CFO/p' person.txt 
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO

Command description: print line 2 to including CFO OK.

[root@chensiqi1 ~]# sed -n '/feixue/,2p' person.txt 
105,feixue,CIO

Command description: under special circumstances, the first two lines do not match feixue，Just match back, if you match feixue Just print this line. So this mixed address is not recommended.

6.2.4.4 filter multiple characters

[root@chensiqi1 ~]# sed -rn '/chensiqi|yy/p' person.txt 
101,chensiqi,CEO
104,yy,CFO

Command description:
    Use extended regular“|"，In order not to use escape symbols“\"，So use-r Option to turn on extended regular expression mode

7. sed command application knowledge extension

7.1 sed make backup while modifying the file

[root@chensiqi1 ~]# cat person.txt 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@chensiqi1 ~]# sed -i.bak 's#zhangyang#NB#g' person.txt 
[root@chensiqi1 ~]# cat person.txt
101,chensiqi,CEO
102,NB,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO
[root@chensiqi1 ~]# cat person.txt.bak 
101,chensiqi,CEO
102,zhangyang,CTO
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command line description:
    stay-i Add after parameter.bak(.Any character), sed The file will be backed up before modification

7.2 special symbol = get line number

[root@chensiqi1 ~]# sed '=' person.txt
1
101,chensiqi,CEO
2
102,NB,CTO
3
103,Alex,COO
4
104,yy,CFO
5
105,feixue,CIO

Command description: use special symbols“="You can get the line number of the file. This is a special usage. Just remember. From the above command results, we also found a bad place: the line number and line are not on the same line.

[root@chensiqi1 ~]# sed '1,3=' person.txt
1
101,chensiqi,CEO
2
102,NB,CTO
3
103,Alex,COO
104,yy,CFO
105,feixue,CIO

Command description: only print the line numbers of lines 1, 2 and 3, and print the contents of the output file at the same time

[root@chensiqi1 ~]# sed '/yy/=' person.txt
101,chensiqi,CEO
102,NB,CTO
103,Alex,COO
4
104,yy,CFO
105,feixue,CIO

Command description:
    Print only the line numbers of regular matching lines, and output the contents of the file at the same time

[root@chensiqi1 ~]# sed -n '/yy/=' person.txt
4

Command description: displays only the line number but not the contents of the line, that is, cancels the default output.

[root@chensiqi1 ~]# sed -n '$=' person.txt
5

Command description:
    "$"Represents the last line, so the line number of the last line is displayed to obtain the total number of rows of the file in a disguised form.

Method improvement:

[root@chensiqi1 ~]# sed '=' person.txt | sed 'N;s#\n# #'
1 101,chensiqi,CEO
2 102,NB,CTO
3 103,Alex,COO
4 104,yy,CFO
5 105,feixue,CIO

Command description: Front sed There is a disadvantage in obtaining the line number of the file, which we use here Sed command N To compensate for this shortcoming. Sed command N The next row of data is read and appended to the schema space.

7.3 sed how to get discontinuous lines

[root@chensiqi1 ~]# sed -n '1p;3p;5p' person.txt
101,chensiqi,CEO
103,Alex,COO
105,feixue,CIO

7.4 use of special symbols {}

[root@chensiqi1 ~]# sed -n '2,4p;=' person.txt
1
102,NB,CTO
2
103,Alex,COO
3
104,yy,CFO
4
5

Command description:-n Remove the default output, 2,4p，Output 2 to 4 lines,=Output line numbers of all lines

[root@chensiqi1 ~]# sed -n '2,4{p;=}' person.txt
102,NB,CTO
2
103,Alex,COO
3
104,yy,CFO
4

Command description:
    '2，4{p;=}'Represents the line number and content of 2 and 4 lines of unified output