Text gadget for shell programming

Posted by released on Thu, 20 Feb 2020 19:20:16 +0100

1. Text Processing Tools
1. grep tools
2. cut tool
3. sort tool
4. uniq Tools
5. tee tools
6. diff tools
7. Pase Tool
2. Characteristics of bash
1. Auto-Completion of Commands and Files
2. Common wildcards
3. Quotes in bash

1. Text Processing Tools

1. grep tools

grep is a==row==filter tool; used to filter rows based on keywords

Grammar and Options

Grammar:

# grep [options]'keyword'file name

Common options:

OPTIONS:
    -i: Case insensitive
    -v: Find rows that do not contain the specified content,Reverse selection
    -w: Search by word
    -o: Print matching keywords
    -c: Count matched rows
    -n: set number
    -r: Walk through directories layer by layer to find
    -A: Show matching rows and how many rows follow 
    -B: Show matching rows and how many preceding rows
    -C: Show how many lines match before and after
    -l: List only matching file names
    -L: List mismatched file names
    -e: Use regular matching
    -E:Using extended regular matching
    ^key:Start with a keyword
    key$:End with keyword
    ^$:Match blank lines
    --color=auto : You can color the part of the keyword you find

Color display (alias setting):

Temporary settings:
# alias grep='grep --color=auto'// only valid for current terminal and current user

Permanent settings:
1) Global (effective for all users)
vim /etc/bashrc
alias grep='grep --color=auto'
source /etc/bashrc

2) Local (for a specific user)
vim ~/.bashrc
alias grep='grep --color=auto'
source ~/.bashrc

Examples:

==Instructions: Do not use / etc/passwd files directly, copy them to / tmp for experimentation!==

# grep-i root passwd ignores case matches for rows containing roots
 # grep-w ftp passwd exactly matches ftp words
 # grep-w Hello passwd exactly matches the Hello word; add your own line containing hello to the file
 # grep-wo ftp passwd prints the matched keyword ftp
 # grep-n root passwd prints good lines matching the root keyword
 # grep-ni root passwd ignores case matching statistics for rows containing the keyword root
 # grep-nic root passwd ignores case matching statistics for the number of rows containing the keyword root
 # grep-i ^ root passwd ignores matching rows that start with root
 # grep bash$passwd matches rows ending in Bash
 # grep-n ^$passwd matches blank lines and prints line numbers
 # grep ^#/etc/vsftpd/vsftpd.conf matches lines starting with #
# grep-v ^#/etc/vsftpd/vsftpd.conf matches lines that do not start with #
# grep-A 5 mail passwd match contains mail keyword and the next 5 lines
 # grep-B 5 mail passwd match contains mail keyword and its first 5 lines
 # grep-C 5 mail passwd matches contain mail keywords and their top and bottom 5 lines

2. cut tool

cut is ==column==intercept tool, used to intercept columns

Grammar and Options

Grammar:

# cut option file name

Common options:

-c: split in characters, intercept
 -d: Custom delimiter, default to tab\t
 -f: Used with-d to specify which area to intercept

Examples:

# cut-d: -f1.txt is divided by: colon, truncating column 1
 # cut-d: -f1,6,7 1.txt separated by: colon, intercepts column 1,6,7
 # cut -c4 1.txt intercepts the fourth character per line in the file
 # cut-c1-4 1.txt intercepts 1-4 characters per line in a file
 # cut-c4-10 1.txt intercepts 4-10 characters per line in a file
 # cut-c5-1.txt intercepts all subsequent characters from the fifth character

3. sort tool

The sort tool is used to sort; it takes each line of a file as a unit, compares them by ASCII code values from the first character to the next, and outputs them in ascending order.

Grammar and Options

-u: remove duplicate rows
 -r: descending, ascending by default
 -o: Output the sorting result to a file, like a redirection symbol>
-n: sorted by number, by default by character
 -t: separator
 -k: Column N
 -b: Ignore leading spaces.
-R: Random sorting with different results for each run

Example

# Sort-n-t: -k3 1.txt is sorted in ascending order by user's uid
# Sort-nr-t: -k3 1.txt in descending order by user's uid
# Sort-n 2.txt sorted by number
# Sort-nu 2.txt sorted numerically and weighted
# sort -nr 2.txt 
# sort -nru 2.txt 
# sort -nru 2.txt 
# Sort-n 2.txt-o 3.txt Sorts the results numerically and redirects them to a file
# sort -R 2.txt 
# sort -u 2.txt 

4.uniq Tools

uniq is used to remove==consecutive==repeat==rows

Common options:
-i: Ignore case
 -c: Count the number of repeated rows
 -d: Show only duplicate rows

Examples:
# uniq 2.txt 
# uniq -d 2.txt 
# uniq -dc 2.txt 

5.tee Tools

The tee tool reads and writes from standard input to standard output and files, i.e., bi-directional override redirection (screen output | text input)

Options:
-a Bidirectional pursuit aggravation orientation

# echo hello world
# echo hello world|tee file1
# cat file1 
# echo 999|tee -a file1
# cat file1 

6.diff tool

The diff tool is used to compare file differences line by line

Note: diff describes two files differently by telling us==how to change the first==after the file==to match the second file==.

Grammar and Options

Grammar:

diff [options] file 1 file 2

Common options:

option Meaning Remarks
-b Do not check spaces
-B Do not check blank lines
-i Do not check case
-w Ignore all spaces
--normal Normal format display (default)
-c Context format display
-u Merge Format Display

Examples:

  • Compare two==Normal file==Same and Same, File preparation:
[root@MissHou ~]# cat file1
aaaa
111
hello world
222
333
bbb
[root@MissHou ~]#
[root@MissHou ~]# cat file2
aaa
hello
111
222
bbb
333
world

1) Normal display

diff Purpose: file1 How to change ability and file2 matching
[root@MissHou ~]# diff file1 file2
1c1,2                   The first line of the first file needs to be changed(c=change)To match lines 1 to 2 of the second file         
< aaaa              Less than sign"<"Represents the left file(file1)File Content
---                 ---Representation separator
> aaa                   Greater than sign">"Represents the right file(file2)File Content
> hello
3d3                 Delete line 3 of the first file(d=delete)To match the third line of the second file
< hello world
5d4                 Line 5 of the first file must be deleted to match line 4 of the second file
< 333
6a6,7                   Add Line 6 to First File(a=add)Content must match lines 6 to 7 of the second file
> 333                   What needs to be added in the second file is 333 and world
> world

2) Context format display

[root@MissHou ~]# diff -c file1 file2
 The first two lines mainly list the filenames and timestamps of the files to be compared; the symbol *** before the filename indicates file1, - - file2
*** file1       2019-04-16 16:26:05.748650262 +0800
--- file2       2019-04-16 16:26:30.470646030 +0800
 ***************I am a separator
 *** 1,6 **** file1 files begin with *** and 1,6 lines 1 to 6
 ! aaaa! Indicates that the line needs to be modified to match the second file
  111
 - hello world - Indicates that the line needs to be deleted to match the second file
  222
 - 333 - Indicates that the line needs to be deleted to match the second file
  bbb
 --- 1,7 - - file2 file at the beginning of -, 1,7 line 1 to 7
 ! aaa indicates that the first file needs to be modified to match the second file
 ! hello indicates that the first file needs to be modified to match the second file
  111
  222
  bbb
 + 333 means the first file needs to be added to match the second file
 + world means that the first file needs to be added to match the second file

3) Merge format display

[root@MissHou ~]# diff -u file1 file2
 The first two lines mainly list the filenames and timestamps of the files to be compared; the symbols before the filenames - - for file1, +++ for file2
--- file1       2019-04-16 16:26:05.748650262 +0800
+++ file2       2019-04-16 16:26:30.470646030 +0800
@@ -1,6 +1,7 @@
-aaaa
+aaa
+hello
 111
-hello world
 222
-333
 bbb
+333
+world
  • Compare two==Directories are different==
By default, the contents of the same file in two directories are also compared
[root@MissHou  tmp]# diff dir1 dir2
diff dir1/file1 dir2/file1
0a1
> hello
Only in dir1: file3
Only in dir2: test1
//If you only need to compare the files in the two directories, you don't need to further compare the contents of the files, you need to add the -q option
[root@MissHou  tmp]# diff -q dir1 dir2
Files dir1/file1 and dir2/file1 differ
Only in dir1: file3
Only in dir2: test1

Other tips:

Sometimes we need to modify other files based on one file, and when there are more modifications, we can fix them.

1)Find out the difference between files and output to a file
[root@MissHou ~]# diff -uN file1 file2 > file.patch
-u:Context mode
-N:Treat non-existent files as empty
2)Patch different contents to file
[root@MissHou ~]# patch file1 file.patch
patching file file1
3)Test Validation
[root@MissHou ~]# diff file1 file2
[root@MissHou ~]#

7. Pase tool

Pase tool for merging file lines

Common options:
-d: Custom interval character, tab by default
 -s: Serial processing, non-parallel

##8. tr Tool

tr is used for character conversion, replacement, and deletion; it is mainly used to==delete control characters in files==or perform==character conversion==

Grammar and Options

Grammar:

Usage 1: The execution result of the command is handled by tr, where string1 is used for query and string2 is used for conversion processing
# commands|tr  'string1'  'string2'
Usage 2:tr handles content from a file, remember to use'<'standard input
# tr  'string1'  'string2' < filename

Usage 3: Match string1 for actions such as deletion
# tr options 'string1' < filename

Common options:

-d Deletes all input characters from string 1.
-s Deletes all repeating character sequences, leaving only the first; compresses the repeating string into a single string

Frequently matched strings:

Character string Meaning Remarks
==a-z==or [:lower:] Match all lowercase letters [a-zA-Z0-9]
==A-Z==or[:upper:] Match all uppercase letters
==0-9==or [:digit:] Match all numbers
[:alnum:] Match all letters and numbers
[:alpha:] Match all letters
[:blank:] All horizontal blanks
[:punct:] Match all punctuation symbols
[:space:] All horizontal or vertical spaces
[:cntrl:] All control characters \f Ctrl-L Line Break <br/>\n Ctrl-J Line Break

\r Ctrl-M carriage return
\t Ctrl-I tab key|

Examples:

[root@MissHou  shell01]# cat 3.txt creates the file itself for testing
ROOT:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
boss02:x:516:511::/home/boss02:/bin/bash
vip:x:517:517::/home/vip:/bin/bash
stu1:x:518:518::/home/stu1:/bin/bash
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
aaaaaaaaaaaaaaaaaaaa
bbbbbb111111122222222222233333333cccccccc
hello world 888
666
777
999

# Tr-d'[:/]'< 3.txt Delete: and/or
# Cat 3.txt | tr-d'[:/]'Delete: and/or
# Tr'[0-9]''@' < 3.txt Replace numbers in files with @ symbols
# Tr'[a-z]''[A-Z]' < 3.txt Replaces lower case letters with upper case letters in a file
# Tr-s'[a-z]'< 3.txt Matches lower case letters and compresses duplicates into one
# Tr-s'[a-z0-9]'< 3.txt Matches lowercase letters and numbers and compresses duplicates into one
# Tr-d'[: digit:]'< 3.txt Delete numbers from file
# Tr-d'[: blank:]'< 3.txt Delete horizontal white space
# Tr-d'[: space:]'< 3.txt Deletes all horizontal and vertical whitespace

Calf knife

  1. Use widgets to intercept the current host IP; NETMASK; broadcast address; MAC address
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4
10.1.1.1:10.1.1.255:255.255.255.0
# ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4|tr ':' '\n'
10.1.1.1
10.1.1.255
255.255.255.0
# ifconfig eth0|grep 'HWaddr'|cut -d: -f2-|cut -d' ' -f4
00:0C:29:25:AE:54
# ifconfig eth0|grep 'HW'|tr -s ' '|cut -d' ' -f5
00:0C:29:B4:9E:4E

# ifconfig eth1|grep Bcast|cut -d: -f2|cut -d' ' -f1
# ifconfig eth1|grep Bcast|cut -d: -f2|tr -d '[ a-zA-Z]'
# ifconfig eth1|grep Bcast|tr -d '[:a-zA-Z]'|tr ' ' '@'|tr -s '@'|tr '@' '\n'|grep -v ^$
# ifconfig eth0|grep 'Bcast'|tr -d [:alpha:]|tr '[ :]' '\n'|grep -v ^$
# ifconfig eth1|grep HWaddr|cut -d ' ' -f11
# ifconfig eth0|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth1|grep HWaddr|tr -s ' '|cut -d' ' -f5
# ifconfig eth0|grep 'Bcast'|tr -d 'a-zA-Z:'|tr ' ' '\n'|grep -v '^$'
  1. Save the username, password, and default shell of all normal users in the system to a file, requiring tab keys to separate the username password from the default shell
# grep 'bash$' passwd |grep -v 'root'|cut -d: -f1,2,7|tr ':' '\t' |tee abc.txt

2. Characteristics of bash

1. Auto-Completion of Commands and Files

Tab can only complete== commands and files== (RHEL6/Centos6)

##2, Common shortcuts

^c Terminate foreground running programs
 ^z Suspends programs running in the foreground to the background
 ^d exits the equivalent exit
 ^l Clear Screen 
^a |home cursor moved to the front of the command line
 ^e |end cursor moved to the back of the command line
 ^u Remove all characters before the cursor
 ^k Remove all characters after cursor
 ^r Search History Command

2. Common wildcards

*:  Match 0 or more arbitrary characters
?:  Match any single character
[list]: matching[list]Any single character in,Or a set of individual characters   [a-z]
[!list]: Match Division list Any single character in
{string1,string2,...}: matching string1,string2 Or more strings

# rm -f file*
# cp *.conf  /dir1
# touch file{1..5}

3. Quotes in bash

  • Double quotation mark'': takes the quotation mark as a whole and allows it to refer to other variable values through the $symbol
  • Single quotation mark'': takes the quotation mark as a whole, prohibits other variable values, and treats special symbols in the shell as common characters
  • Reverse Apostrophe``: As with $(), commands in quotation marks or parentheses take precedence, and if nesting exists, reverse apostrophes cannot be used
[root@MissHou  dir1]# echo "$(hostname)"
server
[root@MissHou  dir1]# echo '$(hostname)'
$(hostname)
[root@MissHou  dir1]# echo "hello world"
hello world
[root@MissHou  dir1]# echo 'hello world'
hello world

[root@MissHou  dir1]# echo $(date +%F)
2018-11-22
[root@MissHou  dir1]# echo `echo $(date +%F)`
2018-11-22
[root@MissHou  dir1]# echo `date +%F`
2018-11-22
[root@MissHou  dir1]# echo `echo `date +%F``
date +%F
[root@MissHou  dir1]# echo $(echo `date +%F`)
2018-11-22

To the end of this article, thank you for reading -----------

Topics: Linux ftp vsftpd shell vim