1. Text Processing Tools

1. grep tools

Grammar and Options

Grammar:

# grep [options]'keyword'file name

Common options:

OPTIONS:
    -i: Case insensitive
    -v: Find rows that do not contain the specified content,Reverse selection
    -w: Search by word
    -o: Print matching keywords
    -c: Count matched rows
    -n: set number
    -r: Walk through directories layer by layer to find
    -A: Show matching rows and how many rows follow 
    -B: Show matching rows and how many preceding rows
    -C: Show how many lines match before and after
    -l: List only matching file names
    -L: List mismatched file names
    -e: Use regular matching
    -E:Using extended regular matching
    ^key:Start with a keyword
    key$:End with keyword
    ^$:Match blank lines
    --color=auto : You can color the part of the keyword you find

Color display (alias setting):

Temporary settings:
# alias grep='grep --color=auto'// only valid for current terminal and current user

Permanent settings:
1) Global (effective for all users)
vim /etc/bashrc
alias grep='grep --color=auto'
source /etc/bashrc

2) Local (for a specific user)
vim ~/.bashrc
alias grep='grep --color=auto'
source ~/.bashrc

Examples:

==Instructions: Do not use / etc/passwd files directly, copy them to / tmp for experimentation!==

# grep-i root passwd ignores case matches for rows containing roots
 # grep-w ftp passwd exactly matches ftp words
 # grep-w Hello passwd exactly matches the Hello word; add your own line containing hello to the file
 # grep-wo ftp passwd prints the matched keyword ftp
 # grep-n root passwd prints good lines matching the root keyword
 # grep-ni root passwd ignores case matching statistics for rows containing the keyword root
 # grep-nic root passwd ignores case matching statistics for the number of rows containing the keyword root
 # grep-i ^ root passwd ignores matching rows that start with root
 # grep bash$passwd matches rows ending in Bash
 # grep-n ^$passwd matches blank lines and prints line numbers
 # grep ^#/etc/vsftpd/vsftpd.conf matches lines starting with #
# grep-v ^#/etc/vsftpd/vsftpd.conf matches lines that do not start with #
# grep-A 5 mail passwd match contains mail keyword and the next 5 lines
 # grep-B 5 mail passwd match contains mail keyword and its first 5 lines
 # grep-C 5 mail passwd matches contain mail keywords and their top and bottom 5 lines

2. cut tool

Grammar and Options

Grammar:

# cut option file name

Common options:

-c: split in characters, intercept
 -d: Custom delimiter, default to tab\t
 -f: Used with-d to specify which area to intercept

Examples:

# cut-d: -f1.txt is divided by: colon, truncating column 1
 # cut-d: -f1,6,7 1.txt separated by: colon, intercepts column 1,6,7
 # cut -c4 1.txt intercepts the fourth character per line in the file
 # cut-c1-4 1.txt intercepts 1-4 characters per line in a file
 # cut-c4-10 1.txt intercepts 4-10 characters per line in a file
 # cut-c5-1.txt intercepts all subsequent characters from the fifth character

3. sort tool

Grammar and Options

-u: remove duplicate rows
 -r: descending, ascending by default
 -o: Output the sorting result to a file, like a redirection symbol>
-n: sorted by number, by default by character
 -t: separator
 -k: Column N
 -b: Ignore leading spaces.
-R: Random sorting with different results for each run

Example

# Sort-n-t: -k3 1.txt is sorted in ascending order by user's uid
# Sort-nr-t: -k3 1.txt in descending order by user's uid
# Sort-n 2.txt sorted by number
# Sort-nu 2.txt sorted numerically and weighted
# sort -nr 2.txt 
# sort -nru 2.txt 
# sort -nru 2.txt 
# Sort-n 2.txt-o 3.txt Sorts the results numerically and redirects them to a file
# sort -R 2.txt 
# sort -u 2.txt 

4.uniq Tools

Common options:
-i: Ignore case
 -c: Count the number of repeated rows
 -d: Show only duplicate rows

Examples:
# uniq 2.txt 
# uniq -d 2.txt 
# uniq -dc 2.txt 

5.tee Tools

Options:
-a Bidirectional pursuit aggravation orientation

# echo hello world
# echo hello world|tee file1
# cat file1 
# echo 999|tee -a file1
# cat file1 

6.diff tool

Note: diff describes two files differently by telling us==how to change the first==after the file==to match the second file==.

Grammar and Options

Grammar:

diff [options] file 1 file 2

Common options:

Examples:

• Compare two==Normal file==Same and Same, File preparation:

[root@MissHou ~]# cat file1
aaaa
111
hello world
222
333
bbb
[root@MissHou ~]#
[root@MissHou ~]# cat file2
aaa
hello
111
222
bbb
333
world

1) Normal display

diff Purpose: file1 How to change ability and file2 matching
[root@MissHou ~]# diff file1 file2
1c1,2                   The first line of the first file needs to be changed(c=change)To match lines 1 to 2 of the second file         
< aaaa              Less than sign"<"Represents the left file(file1)File Content
---                 ---Representation separator
> aaa                   Greater than sign">"Represents the right file(file2)File Content
> hello
3d3                 Delete line 3 of the first file(d=delete)To match the third line of the second file
< hello world
5d4                 Line 5 of the first file must be deleted to match line 4 of the second file
< 333
6a6,7                   Add Line 6 to First File(a=add)Content must match lines 6 to 7 of the second file
> 333                   What needs to be added in the second file is 333 and world
> world

2) Context format display

[root@MissHou ~]# diff -c file1 file2
 The first two lines mainly list the filenames and timestamps of the files to be compared; the symbol *** before the filename indicates file1, - - file2
*** file1       2019-04-16 16:26:05.748650262 +0800
--- file2       2019-04-16 16:26:30.470646030 +0800
 ***************I am a separator
 *** 1,6 **** file1 files begin with *** and 1,6 lines 1 to 6
 ! aaaa! Indicates that the line needs to be modified to match the second file
  111
 - hello world - Indicates that the line needs to be deleted to match the second file
  222
 - 333 - Indicates that the line needs to be deleted to match the second file
  bbb
 --- 1,7 - - file2 file at the beginning of -, 1,7 line 1 to 7
 ! aaa indicates that the first file needs to be modified to match the second file
 ! hello indicates that the first file needs to be modified to match the second file
  111
  222
  bbb
 + 333 means the first file needs to be added to match the second file
 + world means that the first file needs to be added to match the second file

3) Merge format display

[root@MissHou ~]# diff -u file1 file2
 The first two lines mainly list the filenames and timestamps of the files to be compared; the symbols before the filenames - - for file1, +++ for file2
--- file1       2019-04-16 16:26:05.748650262 +0800
+++ file2       2019-04-16 16:26:30.470646030 +0800
@@ -1,6 +1,7 @@
-aaaa
+aaa
+hello
 111
-hello world
 222
-333
 bbb
+333
+world

• Compare two==Directories are different==

By default, the contents of the same file in two directories are also compared
[root@MissHou  tmp]# diff dir1 dir2
diff dir1/file1 dir2/file1
0a1
> hello
Only in dir1: file3
Only in dir2: test1
//If you only need to compare the files in the two directories, you don't need to further compare the contents of the files, you need to add the -q option
[root@MissHou  tmp]# diff -q dir1 dir2
Files dir1/file1 and dir2/file1 differ
Only in dir1: file3
Only in dir2: test1

Other tips:

Sometimes we need to modify other files based on one file, and when there are more modifications, we can fix them.

1)Find out the difference between files and output to a file
[root@MissHou ~]# diff -uN file1 file2 > file.patch
-u:Context mode
-N:Treat non-existent files as empty
2)Patch different contents to file
[root@MissHou ~]# patch file1 file.patch
patching file file1
3)Test Validation
[root@MissHou ~]# diff file1 file2
[root@MissHou ~]#

7. Pase tool

Common options:
-d: Custom interval character, tab by default
 -s: Serial processing, non-parallel

##8. tr Tool

Grammar and Options

Grammar:

Usage 1: The execution result of the command is handled by tr, where string1 is used for query and string2 is used for conversion processing
# commands|tr  'string1'  'string2'
Usage 2:tr handles content from a file, remember to use'<'standard input
# tr  'string1'  'string2' < filename

Usage 3: Match string1 for actions such as deletion
# tr options 'string1' < filename

Common options:

-d Deletes all input characters from string 1.
-s Deletes all repeating character sequences, leaving only the first; compresses the repeating string into a single string

Frequently matched strings:

\r Ctrl-M carriage return
\t Ctrl-I tab key|

Examples:

[root@MissHou  shell01]# cat 3.txt creates the file itself for testing
ROOT:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
boss02:x:516:511::/home/boss02:/bin/bash
vip:x:517:517::/home/vip:/bin/bash
stu1:x:518:518::/home/stu1:/bin/bash
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
aaaaaaaaaaaaaaaaaaaa
bbbbbb111111122222222222233333333cccccccc
hello world 888
666
777
999

# Tr-d'[:/]'< 3.txt Delete: and/or
# Cat 3.txt | tr-d'[:/]'Delete: and/or
# Tr'[0-9]''@' < 3.txt Replace numbers in files with @ symbols
# Tr'[a-z]''[A-Z]' < 3.txt Replaces lower case letters with upper case letters in a file
# Tr-s'[a-z]'< 3.txt Matches lower case letters and compresses duplicates into one
# Tr-s'[a-z0-9]'< 3.txt Matches lowercase letters and numbers and compresses duplicates into one
# Tr-d'[: digit:]'< 3.txt Delete numbers from file
# Tr-d'[: blank:]'< 3.txt Delete horizontal white space
# Tr-d'[: space:]'< 3.txt Deletes all horizontal and vertical whitespace

Calf knife

1. Use widgets to intercept the current host IP; NETMASK; broadcast address; MAC address

# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4
10.1.1.1:10.1.1.255:255.255.255.0
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4|tr ':' '\n'
10.1.1.1
10.1.1.255
255.255.255.0
# ifconfig eth0|grep 'HWaddr'|cut -d: -f2-|cut -d' ' -f4
00:0C:29:25:AE:54
# ifconfig eth0|grep 'HW'|tr -s ' '|cut -d' ' -f5
00:0C:29:B4:9E:4E

# ifconfig eth1|grep Bcast|cut -d: -f2|cut -d' ' -f1
# ifconfig eth1|grep Bcast|cut -d: -f2|tr -d '[ a-zA-Z]'
# ifconfig eth1|grep Bcast|tr -d '[:a-zA-Z]'|tr ' ' '@'|tr -s '@'|tr '@' '\n'|grep -v ^$
# ifconfig eth0|grep 'Bcast'|tr -d [:alpha:]|tr '[ :]' '\n'|grep -v ^$
# ifconfig eth1|grep HWaddr|cut -d ' ' -f11
# ifconfig eth0|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth1|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth0|grep 'Bcast'|tr -d 'a-zA-Z:'|tr ' ' '\n'|grep -v '^$'

1. Save the username, password, and default shell of all normal users in the system to a file, requiring tab keys to separate the username password from the default shell

# grep 'bash$' passwd |grep -v 'root'|cut -d: -f1,2,7|tr ':' '\t' |tee abc.txt

2. Characteristics of bash

1. Auto-Completion of Commands and Files

Tab can only complete== commands and files== (RHEL6/Centos6)

##2, Common shortcuts

^c Terminate foreground running programs
 ^z Suspends programs running in the foreground to the background
 ^d exits the equivalent exit
 ^l Clear Screen 
^a |home cursor moved to the front of the command line
 ^e |end cursor moved to the back of the command line
 ^u Remove all characters before the cursor
 ^k Remove all characters after cursor
 ^r Search History Command

2. Common wildcards

*:  Match 0 or more arbitrary characters
?:  Match any single character
[list]: matching[list]Any single character in,Or a set of individual characters   [a-z]
[!list]: Match Division list Any single character in
{string1,string2,...}: matching string1,string2 Or more strings

# rm -f file*
# cp *.conf  /dir1
# touch file{1..5}

3. Quotes in bash

• Double quotation mark'': takes the quotation mark as a whole and allows it to refer to other variable values through the $symbol
• Single quotation mark'': takes the quotation mark as a whole, prohibits other variable values, and treats special symbols in the shell as common characters
• Reverse Apostrophe``: As with $(), commands in quotation marks or parentheses take precedence, and if nesting exists, reverse apostrophes cannot be used

[root@MissHou  dir1]# echo "$(hostname)"
server
[root@MissHou  dir1]# echo '$(hostname)'
$(hostname)
[root@MissHou  dir1]# echo "hello world"
hello world
[root@MissHou  dir1]# echo 'hello world'
hello world

[root@MissHou  dir1]# echo $(date +%F)
2018-11-22
[root@MissHou  dir1]# echo `echo $(date +%F)`
2018-11-22
[root@MissHou  dir1]# echo `date +%F`
2018-11-22
[root@MissHou  dir1]# echo `echo `date +%F``
date +%F
[root@MissHou  dir1]# echo $(echo `date +%F`)
2018-11-22

To the end of this article, thank you for reading -----------