Text Processing Tools
1,cut
- -d Specifies the separator.For example, -d: -d''
- -f Specifies which column to choose.For example, -f1,3
- --output-delimiter specifies the separator to display
Tr-s compression-d deletion-c except
2. Percentage of disks fetched using tr and cut
[root@glowing-bliss-1 data]# df -h | tr -s ' ' | cut -d' ' -f5 | tr -dc '[0-9]\n' 0 0 5 0 32 67 32 0
3. Take IP
[root@glowing-bliss-1 data]# ifconfig lo | head -2 | tail -1 | tr -s ' ' | cut -d' ' -f3 127.0.0.1 [root@glowing-bliss-1 data]# ifconfig lo | head -2 | tail -1 | tr -s ' ' : | cut -d: -f3 127.0.0.1
4. paste merge file
Merge two files. -d Specifies the splitter
Following is a merge of a.txt and 1.txt with the separator:
qqq@qqq:~$ cat a.txt a b c d qqq@qqq:~$ cat 1.txt 1 2 3 4 qqq@qqq:~$ paste 1.txt a.txt -d: 1:a 2:b 3:c 4:d
5. wc command
Count rows, words, bytes
-m Characters - The widest line of the L file
[root@k8s-master ~]# wc /etc/passwd 20 28 874 /etc/passwd [root@k8s-master ~]# wc -l /etc/passwd 20 /etc/passwd [root@k8s-master ~]# wc -c /etc/passwd 874 /etc/passwd [root@k8s-master ~]# wc -w /etc/passwd 28 /etc/passwd [root@k8s-master ~]# wc -lwc /etc/passwd 20 28 874 /etc/passwd [root@k8s-master ~]# wc -clw /etc/passwd 20 28 874 /etc/passwd
6. wc shows the length of the file with the longest filename in the current directory
[root@k8s-master ~]# ls -1 /etc/ | wc -L 24
7. sort command
Alphabetical sort by default
-u Deletes duplicate rows in the output;
-t c uses c as the field delimiter (that is, the delimiter, as with cut-d);
-k x Sorts using column x as the base
7.1 Sort passwd files based on UID
[root@k8s-master ~]# cat /etc/passwd | sort -t: -k3 -nr qqq:x:1000:1000:qqq:/home/qqq:/bin/bash polkitd:x:999:998:User for polkitd:/:/sbin/nologin systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin nobody:x:99:99:Nobody:/:/sbin/nologin ...
7.2 df Disk Utilization Sorting
[root@glowing-bliss-1 data]# df -h | tr -s ' ' | cut -d' ' -f5 | tr -dc '[0-9]\n' | sort -n 0 0 0 0 5 32 32 67
8,uniq
Remove overlapping lines from output
-d Show only rows of duplicate results
-u only shows rows that have not been repeated
8.1 [Interview Question] Count the same list of files in two different directories
qqq@qqq:~$ ls . /tmp -1 | sort | uniq -d 10.txt 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt qqq@qqq:~$ (ls /tmp -1;ls . -1) | sort | uniq -d #This shows no directories 10.txt 1.txt 2.txt 3.txt 4.txt 5.txt 6.txt 7.txt 8.txt 9.txt
8.2 [Interview Question] Count the different file lists in two different directories
qqq@qqq:~$ (ls /tmp -1;ls . -1) | sort | uniq -u #This shows no directories
{1..txt}
a.txt
gems
myblogs
systemd-private-b8854b73c0e2479db1d56e43d8995bec-systemd-resolved.service-AcZjbU
systemd-private-b8854b73c0e2479db1d56e43d8995bec-systemd-timesyncd.service-sngpw9
vmware-root_643-39797085159. lastb displays btmb file contents
[root@glowing-bliss-1 data]# lastb -f /var/log/btmp
10. [Interview Question] Write a script to do nginx log statistics, statistic the top 10 most visited IP
[root@glowing-bliss-1 data]# awk '{print $1}' /var/log/nginx/access.log | sort -nr | uniq -c | sort -nr | head11. diff compares the differences between the two files
-u Display uniform diff format files
qqq@qqq:~$ diff 1.txt 2.txt -u --- 1.txt 2019-08-04 10:07:47.384154832 +0000 +++ 2.txt 2019-08-04 10:08:01.620224554 +0000 @@ -1,3 +1,4 @@ 1 2 3 +4
13,patch
Copy changed files in other files (use with caution)
-b option to automatically backup changed files
Regular Expressions (Standards and Extensions)
1. Basic Grammar
Characters that need to be escaped in standard regular expressions are: (,), {,}, |,+,.
Expand the beginning, end, grouped reference backslashes under regular expressions
- * means to match any number of times, including 0 times
- * Any character of any length
- Represents matching the previous character 0 or once
- +means to match the previous character one or more times
- {n,m} means matching the previous characters n to m times
- ^ denotes the beginning of a line
- $means end of line
- ^$denotes a blank line ^[[:space:]]*$denotes a blank line
- \<Initial\>End of Line
- \<hello\> Hello word (word composition: alphanumeric underscore, all other characters are separators)
- (. *) Brackets are grouped, \1 \2 are content that uses the preceding bracket
2. Display disk partition usage
[root@glowing-bliss-1 data]# df -h | grep '^/dev/sd' | tr -s ' ' | cut -d' ' -f5 | cut -d% -f1 | sort -n 32 67
3. Replace root with rooter
:%s@\(root\)@\1er@g
4. Regular expression to get IP
qqq@qqq:~$ ifconfig ens33 | grep -Eo '([0-9]{,3}\.){3}([0-9]{,3})'
192.168.38.148
255.255.255.0
192.168.38.2555. Find the line in the ntestat-atn output where LISTEN is followed by any white space character
qqq@qqq:~$ netstat -atn | grep -E 'LISTEN[[:space:]]*$' tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp6 0 0 :::22 :::* LISTEN
6. View the number of connections established per IP
qqq@qqq:~$ cat ss.log | grep ESTAB | tr -s ' ' : | cut -d: -f6 | sort | uniq -c | sort -nr | head -3
44 127.0.0.1
10 113.234.28.244
8 124.64.18.1357. Get the version number of CentOS 7
[root@glowing-bliss-1 data]# cat /etc/centos-release | grep -Eo '[0-9]+' | head -1 7
8,grep
-A #How many lines after -B #How many lines before - #How many lines before and after each -i Ignore case -e is equivalent to or in logic and can be used multiple times -w matches the whole word
9. misc directory
The disc will be automatically mounted to the / misc/cd directory The role of autofs, if not, install and start one
10. nmap Scan LAN Machine
See which IP s are currently in use on your local area network
[root@centos7 ~]# nmap -v -sn 192.168.10.1/24 | grep -B1 "Host is up" | grep report | cut -d' ' -f5 192.168.10.1 192.168.10.6 192.168.10.12 192.168.10.14 192.168.10.16 192.168.10.19 192.168.10.20 192.168.10.28 ...
11. Basic Regular Expression Metacharacters
Character Matching
- Match any single character
- [] Matches any single character within a specified range.[.?] Indicates a match. Or \ or question mark
- Any single character outside the matching range
- [: digit:] and so on
12,cat -A
You can see something you can't see: Tab, carriage return, line change, etc.
[root@centos7 ~]# cat -A 1.txt a b^Ic$ dd$ $ d$ $ $
13. Procedure Execution Mode
Advanced Language->Compiler->Machine Code->Execution
14,.vimrc vim profile
root@qqq:~# vim ~/.vimrc
root@qqq:~# cat ~/.vimrc
set ignorecase
set cursorline
set autoindent
set ai
autocmd BufNewFile *.yaml exec ":call SetTitle()"
func SetTitle()
if expand("%:e") == 'yaml'
call setline(1,"#**************************************************************")
call setline(2,"#Author: uscwifi")
call setline(3,"#QQ: 2*******1")
call setline(4,"#Date: ".strftime("%Y-%m-%d"))
call setline(5,"#FileName: ".expand("%"))
call setline(6,"#URL: http://www.baidu.com")
call setline(7,"#Description: The test script")
call setline(8,"#Copyright (C): ".strftime ("%Y")." Copyright_Site Name Copyright ")
call setline(9,"#************************************************************")
call setline(10,"")
endif
endfunc
autocmd BufNewFile * normal G15. Put the directory path of the script into the PATH path and execute the script directly
How to execute the script
- 1,bash date.sh
- 2. Relative path method. /date.sh
- 3. Absolute Path Method
- 4. PATH variable method, the directory of scripts adds PATH variable (write bashrc or profile, then source)
- 5,cat date.sh|bash
16. Two errors in scripting
Both shell and python belong to interpretive syntax, execute while interpreting; C language, JAVA, etc. are compiled languages, all code can be compiled before execution
- 1. Grammatical errors, it is best to use bash-n date.sh check to reduce grammatical errors, commands after grammatical errors will not execute, but the previous commands will execute
- 2. Other errors: command does not exist, command error, etc.Does not affect subsequent code execution.Adding set -e at the beginning of the script stops execution immediately when an error occurs.
- Bash-n Checks for grammatical errors, make sure to check after writing the script.
- Bash-x is used to debug scripts, view the execution of scripts, and execute logic.
17. Several methods for calculating the sum of shell s, etc.
17.1, let command, more common
[root@centos7 ~]# x=10 [root@centos7 ~]# y=17 [root@centos7 ~]# let z=$x+$y [root@centos7 ~]# echo $z 27
17.2, $()) are more common
[root@centos7 ~]# x=100 [root@centos7 ~]# y=302 [root@centos7 ~]# z=$(($x+$y)) [root@centos7 ~]# echo $z 402
17.3, bc sum, more common
[root@centos7 ~]# x=123 [root@centos7 ~]# y=986 [root@centos7 ~]# echo $x+$y | bc 1109
18. Color of shell scripts
echo must have-e parameter to use color
[root@glowing-bliss-1 ~]# cat /data/docker_stats.sh
#!/bin/bash
RED="\e[31;1m"
GREEN="\e[32;1m"
YELLOW="\e[33;1m"
END_COLOR="\e[0m"
# usage
echo -e "\n${YELLOW}################################${END_COLOR}\n"19,cat /proc/partitions
20. Single, Double, Reverse Quotes
- Single quote: strong reference, no command, no variable
- Double quotation marks: do not recognize commands, but separate variables;
- Reverse single quotation mark, all recognized
[root@glowing-bliss-1 ~]# echo '$HOSTNAME' $HOSTNAME [root@glowing-bliss-1 ~]# echo "$HOSTNAME" glowing-bliss-1.localdomain [root@glowing-bliss-1 ~]# ls `pwd` virt-sysprep-firstboot.log [root@glowing-bliss-1 ~]# echo `hostname` -bash: echglowing-bliss-1.localdomain: command not found
21. Programs have parent and child processes
21.1. View parent-child relationships between processes
pstree
21.2. View current process ID
echo $BASHPID
Echo $$echo $$is often used in scripts to see the process ID of the script when it runs
21.3. View parent process ID
echo $PPID
22,set
unset delete variable
set view
Set-C cannot overwrite an existing file after execution
shell Scripting Basics
[.?]Express. ? One of the
grep -E "(bash|nologin)$" /etc/passwd grep -E "bash$|nologin$" /etc/passwd
1. Declarations of environment variables
- export EDITOR=vim defines the default editor to write to the profile
- Declare-x EDITOR=vim is equivalent to above
- Declare-r declares that a read-only variable is equal to readonly name
2,set --
Empty all location variables
3. Use curly brackets for variable references
- ${10} ${DATE} ${PWD}