1, Regular expression overview

linux text processing tool

Text processor	Basic regular expression	Extended regular expression
vi editor	support
grep	support
egrep	support	support
sed	support
awk	support	support

1. Basic regular expression

Metacharacter	effect
\	Escape character, used to cancel the meaning of special symbols
^	Start position of matching string
$	End of matching string
.	Matches any character except \ n (line feed)
*	Matches the previous subexpression 0 or more times
[list]	Match a character in the list
[^list]	One of the characters in the list does not match
\{n\}	Match the previous sub expression n times
\{n,\}	Match the previous subexpression no less than n times
\{n,m\}	Match the previous subexpression n to m times

[root@localhost ~]# grep root /etc/passwd / / filter the lines containing root in the file
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin   
[root@localhost ~]# grep ^root /etc/passwd / / filter lines starting with root
root:x:0:0:root:/root:/bin/bash
[root@localhost ~]# grep '[^s]bin' /etc/passwd / / filter the rows that are not s before bin
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin
tvxq:x:1000:1000:tvxq:/home/tvxq:/bin/bash   
zhangsan:x:1001:1001::/home/zhangsan:/bin/bash
wangwu:x:1003:1003::/home/wangwu:/bin/bash

2. Extended regular expression

Metacharacter	effect
+	Match the previous subexpression more than 1 time
？	Matches the previous subexpression 0 or 1 times
()	Take the string in () as a whole
|	Match string in or

[root@localhost ~]# egrep '0+' /etc/passwd / / matches at least one 0 line
root:x:0:0:root:/root:/bin/bash
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
[root@localhost ~]# egrep '(root|ntp)' /etc/passwd / / match the line containing root or ntp
root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin
ntp:x:38:38::/etc/ntp:/sbin/nologin
[root@localhost ~]# egrep 'ro? T '/ etc / passwd / / match the line containing rt or rot
abrt:x:173:173::/etc/abrt:/sbin/nologin
rtkit:x:172:172:RealtimeKit:/proc:/sbin/nologin

2, Sed tool overview

1.Sed basic syntax

• Command syntax:
  sed -e 'edit instruction' file 1 file 2
  sed -n -e 'edit instruction' file 1 file 2
  sed -i -e 'edit instruction' file 1 file 2
• Common options:
  -e specifies the command to be executed, and only one editing command can be omitted
  -n only the processed rows are output
  -i edit the file directly without outputting the results
• Edit instruction format: [address 1] [address 2] operation [parameter]
  "Operation" is used to specify the action behavior of file operation, that is, the command of sed. Generally, it is in the format of "[n1[,n2]]" operation parameter. n1 and n2 are optional and represent the number of lines selected for operation. If the operation needs to be carried out between 5 and 20 lines, it is expressed as "5,20 action behavior". Common operations include the following.
  a: Add: add a line of specified content under the current line.
  c: Replace to replace the selected line with the specified content.
  d: Delete, delete the selected row.
  i: Insert, inserts a row of specified content above the selected row.
  p: Print: if a line is specified at the same time, it means to print the specified line; If no line is specified, all contents will be printed; If there are non printing characters, they are output in ASCII code. It is usually used with the "- n" option.
  s: Replace, replace the specified character.
  y: Character conversion.

2.Sed usage example

Example 1 delete the first character of each line of a file

[root@localhost ~]# head /etc/passwd > test.txt   
[root@localhost ~]# sed  's/^.//' test.txt
oot:x:0:0:root:/root:/bin/bash
in:x:1:1:bin:/bin:/sbin/nologin
aemon:x:2:2:daemon:/sbin:/sbin/nologin
dm:x:3:4:adm:/var/adm:/sbin/nologin
p:x:4:7:lp:/var/spool/lpd:/sbin/nologin
ync:x:5:0:sync:/sbin:/bin/sync
hutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
alt:x:7:0:halt:/sbin:/sbin/halt
ail:x:8:12:mail:/var/spool/mail:/sbin/nologin
perator:x:11:0:operator:/root:/sbin/nologin
[root@localhost ~]# 

Example 2 delete all numbers in each line of the file

[root@localhost ~]# sed  's/[0-9]//g' test.txt
root:x:::root:/root:/bin/bash
bin:x:::bin:/bin:/sbin/nologin
daemon:x:::daemon:/sbin:/sbin/nologin
adm:x:::adm:/var/adm:/sbin/nologin
lp:x:::lp:/var/spool/lpd:/sbin/nologin
sync:x:::sync:/sbin:/bin/sync
shutdown:x:::shutdown:/sbin:/sbin/shutdown
halt:x:::halt:/sbin:/sbin/halt
mail:x:::mail:/var/spool/mail:/sbin/nologin
operator:x:::operator:/root:/sbin/nologin
[root@localhost ~]# 

Example 3 delete test All special characters in TXT

[root@localhost ~]# sed  's/[^a-Z0-9]//g' test.txt
rootx00rootrootbinbash
binx11binbinsbinnologin
daemonx22daemonsbinsbinnologin
admx34admvaradmsbinnologin
lpx47lpvarspoollpdsbinnologin
syncx50syncsbinbinsync
shutdownx60shutdownsbinsbinshutdown
haltx70haltsbinsbinhalt
mailx812mailvarspoolmailsbinnologin
operatorx110operatorrootsbinnologin

Example 4 modify the ssh service configuration file with sed, remove all lines beginning with # and empty lines, change the port to 2222, do not allow root user to log in and connect, and modify UseDNS to no

[root@localhost ~]# sed  -e 's/#Port 22/Port 2222/' -e 's/#PermitRootlogin yes/PermitRootlogin no/' -e 's/#UseDNS yes/UseDNS no/' -e '/^#/d' -e '/^$/d' sshd.config
Port 2222
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile	.ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
UseDNS no
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem	sftp	/usr/libexec/openssh/sftp-server

Example 5 call script

[root@localhost test]# head /etc/passwd > pass.txt / / extract the first ten lines from the passwd file to pass txt
[root@localhost test]# cat sed.sh 
#!/bin/sed  -f

s/root/ROOT/g     //Replace all roots with root
s/\sbin\/nologin/xxx/g   
1d               //Delete first row
[root@localhost test]# sed -i -f sed.sh pass.txt 

[root@localhost test]# cat  pass.txt 
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/ROOT:/sbin/nologin
[root@localhost test]# 

3, Awk tool introduction

1.Awk basic syntax

awk [[options] 'Mode or condition{Edit instruction}' File 1 file 2   //Filter and output the qualified contents in the file
awk -f Script file 1 file 2     //Invoke edit instruction from script, filter and output content.

Patterns can be conditional statements, compound statements, or regular expressions
In the process of using the awk command, you can use logical operators
"& &" means "and" and "|" means "or" and "!" Means "not";
It can also carry out simple mathematical operations, such as +, -, *, /,%, ^ respectively representing addition, subtraction, multiplication, division, remainder and power

Single quotation marks plus braces "{}" are used to set the processing action of data
awk can directly process the target file or process the target file through "- f" reading script

Common built-in variables

variable	describe
$0	The entire line content of the currently processed line
$n	The nth field of the current processing line (column n)
NF	Number of fields in the currently processed row
NR	The line number of the currently processed line
FS	Specifies the field separator for each line of text, which defaults to spaces or tab stops
OFS	Output field separator, which defaults to space or tab stop
RS	Enter the record separator. The default is space or tab stop
ORS	The output record separator, which defaults to space or tab stop

2. Example of awk usage

Example 1 lists the users of bash environment in the system

[root@localhost ~]# awk -F: '$7~/bash/{print $1}' /etc/passwd
root
tvxq
[root@localhost ~]# 

Example 2: check the file / etc/passwd to filter out users, UIDs and home directories, and count the total number of users in a

[root@localhost ~]# awk -F: 'BEGIN{print "user\t\t\t","UID\t\t\t","home"}{print $1"\t\t\t",$3"\t\t\t",$6}END{print "total is "NR}' /etc/passwd
user			 UID			 home
root			 0			     /root
bin		     	 1		     	 /bin
daemon			 2		    	 /sbin
adm			     3		    	 /var/adm
...
tcpdump			 72			     /
tvxq			 1000			 /home/tvxq
total is 44
[root@localhost ~]# 

Example 3 output all users in the system and their encrypted passwords

[root@localhost ~]# awk -F: '{if($2!="*"&&$2!="!!") {print "user name" $1, "password" $2} '/ etc / shadow
 user name root The password is $6$yJ3dKevUaX8gSdu1$d66CL1ODS7fh9t7JZqfq0Hrb5qeHGFQ.xSy7DT84u5ej0sUmO7FvjR90VcNHzW0JmDLiYIcxkaMNoEnPT5Ibc0
 user name tvxq The password is $6$ofGRDt4gApgPrawP$KlCkKsKzLrANar59653IK/V0MvSfDji/wlEN3mC6Qos1XyscPLMaio1BNJCciJQKNvzuPB4BxB70uUZ8LG2SS0
[root@localhost ~]#