[100 day learning notes of Python stack] Day33Linux utility

Posted by kabucek on Sat, 12 Feb 2022 05:29:21 +0100

Utility program

File and folder operations

  1. Create / delete empty directory - mkdir / rmdir.

    [root ~]# mkdir abc
[root ~]# mkdir -p xyz/abc
[root ~]# rmdir abc

  2. Create / delete file - touch / rm.

    [root ~]# touch readme.txt
[root ~]# touch error.txt
[root ~]# rm error.txt
rm: remove regular empty file 'error.txt'? y
[root ~]# rm -rf xyz
    • The touch command is used to create a blank file or modify the file time. In Linux system, a file has three times:
      • Time of change - mtime.
      • Time to change permissions - ctime.
      • Last visit time - atime.
    • Several important parameters of rm:
      • -i: Interactive deletion, each deleted item will be asked.
      • -r: Delete the directory and recursively delete the files and directories in the directory.
      • -f: Forced deletion, ignoring nonexistent files without any prompt.

  3. Switch and view the current working directory - cd / pwd.

    Note: the cd command can be followed by a relative path (with the current path as a reference) or an absolute path (starting with /) to switch to the specified directory, or cd To return to the previous directory. Please think about what parameters should be added to the cd command if you want to return to the upper directory?

  4. View directory contents - ls.

    • -l: View files and directories in long format.
    • -a: Displays files and directories that begin with a dot (hidden files).
    • -R: Recursively expand the directory (continue to list the files and directories under the directory).
    • -d: Only the contents are listed, and nothing else is listed.
    • -S / -t: sort by size / time.

  5. View the contents of the file - cat / tac / head / tail / more / less / rev / od.

    [root ~]# wget http://www.sohu.com/ -O sohu.html
--2018-06-20 18:42:34--  http://www.sohu.com/
Resolving www.sohu.com (www.sohu.com)... 14.18.240.6
Connecting to www.sohu.com (www.sohu.com)|14.18.240.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 212527 (208K) [text/html]
Saving to: 'sohu.html'
100%[==================================================>] 212,527     --.-K/s   in 0.03s
2018-06-20 18:42:34 (7.48 MB/s) - 'sohu.html' saved [212527/212527]
[root ~]# cat sohu.html
...
[root ~]# head -10 sohu.html
<!DOCTYPE html>
<html>
<head>
<title>Sohu</title>
<meta name="Keywords" content="Sohu,portal site,new media,Network media,Journalism,Finance and Economics,Sports,entertainment,fashion,automobile,house property,science and technology,picture,forum,micro-blog,Blog,video,film,TV play"/>
<meta name="Description" content="Sohu provides users with the latest information 24 hours a day, as well as search, e-mail and other network services. The content includes global hot events, breaking news, current affairs comments, popular film and television dramas, sports events, industry trends, life service information, as well as interactive spaces such as forums, blogs, microblogs and my Sohu." />
<meta name="shenma-site-verification" content="1237e4d02a3d8d73e96cbd97b699e9c3_1504254750">
<meta charset="utf-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1"/>
[root ~]# tail -2 sohu.html
</body>
</html>
[root ~]# less sohu.html
...
[root ~]# cat -n sohu.html | more
...

    Note: a command named wget is used above. It is a network downloader program that can download resources from the specified URL.

  6. Copy / move files - cp / mv.

    [root ~]# mkdir backup
[root ~]# cp sohu.html backup/
[root ~]# cd backup
[root backup]# ls
sohu.html
[root backup]# mv sohu.html sohu_index.html
[root backup]# ls
sohu_index.html

  7. File rename - Rename.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# rename .htm .html *.htm

  8. Find files and find content - find / grep.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# find / -name "*.html"
/root/sohu.html
/root/backup/sohu_index.html
[root@izwz97tbgo9lkabnat2lo8z ~]# find . -atime 7 -type f -print
[root@izwz97tbgo9lkabnat2lo8z ~]# find . -type f -size +2k
[root@izwz97tbgo9lkabnat2lo8z ~]# find . -type f -name "*.swp" -delete
[root@iZwz97tbgo9lkabnat2lo8Z ~]# grep "<script>" sohu.html -n
20:<script>
[root@iZwz97tbgo9lkabnat2lo8Z ~]# grep -E \<\/?script.*\> sohu.html -n
20:<script>
22:</script>
24:<script src="//statics.itc.cn/web/v3/static/js/es5-shim-08e41cfc3e.min.js"></script>
25:<script src="//statics.itc.cn/web/v3/static/js/es5-sham-1d5fa1124b.min.js"></script>
26:<script src="//statics.itc.cn/web/v3/static/js/html5shiv-21fc8c2ba6.js"></script>
29:<script type="text/javascript">
52:</script>
...

    Note: grep can use regular expressions when searching strings. If you need to use regular expressions, you can use grep -E or egrep directly.

  9. Create links and view links - ln / readlink.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# ls -l sohu.html
-rw-r--r-- 1 root root 212131 Jun 20 19:15 sohu.html
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ln /root/sohu.html /root/backup/sohu_backup
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls -l sohu.html
-rw-r--r-- 2 root root 212131 Jun 20 19:15 sohu.html
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ln /root/sohu.html /root/backup/sohu_backup2
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls -l sohu.html
-rw-r--r-- 3 root root 212131 Jun 20 19:15 sohu.html
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ln -s /etc/centos-release sysinfo
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls -l sysinfo
lrwxrwxrwx 1 root root 19 Jun 20 19:21 sysinfo -> /etc/centos-release
[root@iZwz97tbgo9lkabnat2lo8Z ~]# cat sysinfo
CentOS Linux release 7.4.1708 (Core)
[root@iZwz97tbgo9lkabnat2lo8Z ~]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

    Note: links can be divided into hard links and soft links (symbolic links). A hard link can be regarded as a pointer to file data, just like the reference count of objects in Python. Each time a hard link is added, the number of links corresponding to the file increases by 1. Only when the number of links of the file is 0, the storage space corresponding to the file can be overwritten by other files. In fact, when we delete a file, we do not delete the data on the hard disk. What we delete is only a pointer, or a usage record of the data. Therefore, when "shredding" a file, software like "file shredder" will not only delete the file pointer, but also fill in the data in the corresponding storage area of the file to ensure that the file can no longer be recovered. The soft link is similar to the shortcut under Windows system. When the file linked by the soft link is deleted, the soft link will become invalid.

  10. Compress / decompress and archive / de Archive - gzip / gunzip / xz.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# wget http://download.redis.io/releases/redis-4.0.10.tar.gz
--2018-06-20 19:29:59--  http://download.redis.io/releases/redis-4.0.10.tar.gz
Resolving download.redis.io (download.redis.io)... 109.74.203.151
Connecting to download.redis.io (download.redis.io)|109.74.203.151|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1738465 (1.7M) [application/x-gzip]
Saving to: 'redis-4.0.10.tar.gz'
100%[==================================================>] 1,738,465   70.1KB/s   in 74s
2018-06-20 19:31:14 (22.9 KB/s) - 'redis-4.0.10.tar.gz' saved [1738465/1738465]
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls redis*
redis-4.0.10.tar.gz
[root@iZwz97tbgo9lkabnat2lo8Z ~]# gunzip redis-4.0.10.tar.gz
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls redis*
redis-4.0.10.tar

  11. Archiving and de archiving - tar.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# tar -xvf redis-4.0.10.tar
redis-4.0.10/
redis-4.0.10/.gitignore
redis-4.0.10/00-RELEASENOTES
redis-4.0.10/BUGS
redis-4.0.10/CONTRIBUTING
redis-4.0.10/COPYING
redis-4.0.10/INSTALL
redis-4.0.10/MANIFESTO
redis-4.0.10/Makefile
redis-4.0.10/README.md
redis-4.0.10/deps/
redis-4.0.10/deps/Makefile
redis-4.0.10/deps/README.md
...

    Note: the tar command is used for archiving (also known as creating archiving) and de archiving. Generally, creating archiving requires - cvf three parameters, where c represents create, v represents verbose, and f represents the file specified for archiving; The - xvf parameter needs to be added to de archive, where x represents extract, and the other two parameters are the same as creating archive.

  12. Convert standard input to command line arguments - xargs.

    The following command will find the html files in the current path, and then pass these files as parameters to the rm command through xargs to realize the operation of finding and deleting files.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# find . -type f -name "*.html" | xargs rm -f

    The following command turns multiple lines in the a.txt file into one line and outputs it to the b.txt file, where < means to read the input from a.txt and > means to output the execution result of the command to b.txt.

    [root@iZwz97tbgo9lkabnat2lo8Z ~]# xargs < a.txt > b.txt

    Note: as demonstrated above, this command is often used in pipeline (a way to realize inter process communication) and redirection (reassigning the location of input and output). Pipeline operation and input and output redirection will be discussed later.

  13. Displays the file or directory - basename / dirname.

  14. Other relevant tools.

    • Sort - sort content
    • uniq - remove adjacent duplicates
    • tr - replace specified content with new content
    • Cut / paste - cut / paste content
    • Split - split file
    • File - determine file type
    • wc - count the number of lines, words and bytes of the file
    • iconv - code conversion
    [root ~]# cat foo.txt
grape
apple
pitaya
[root ~]# cat bar.txt
100
200
300
400
[root ~]# paste foo.txt bar.txt
grape   100
apple   200
pitaya  300
        400
[root ~]# paste foo.txt bar.txt > hello.txt
[root ~]# cut -b 4-8 hello.txt
pe      10
le      20
aya     3
0
[root ~]# cat hello.txt | tr '\t' ','
grape,100
apple,200
pitaya,300
,400
[root ~]# split -l 100 sohu.html hello
[root ~]# wget https://www.baidu.com/img/bd_logo1.png
[root ~]# file bd_logo1.png
bd_logo1.png: PNG image data, 540 x 258, 8-bit colormap, non-interlaced
[root ~]# wc sohu.html
  2979   6355 212527 sohu.html
[root ~]# wc -l sohu.html
2979 sohu.html
[root ~]# wget http://www.qq.com -O qq.html
[root ~]# iconv -f gb2312 -t utf-8 qq.html

Pipes and Redirection

  1. Use of pipes - |.

    Example: find the number of files in the current directory.

    [root ~]# find ./ | wc -l
6152

    Example: list the files and folders under the current path and add a number to each item.

    [root ~]# ls | cat -n
     1  dump.rdb
     2  mongodb-3.6.5
     3  Python-3.6.5
     4  redis-3.2.11
     5  redis.conf

    Example: find record The total number of records that contain AAA but not BBB in the log

    [root ~]# cat record.log | grep AAA | grep -v BBB | wc -l

  2. Output redirection and error redirection - > / > > / 2 >.

    [root ~]# cat readme.txt
banana
apple
grape
apple
grape
watermelon
pear
pitaya
[root ~]# cat readme.txt | sort | uniq > result.txt
[root ~]# cat result.txt
apple
banana
grape
pear
pitaya
watermelon

  3. Enter redirection - <.

    [root ~]# echo 'hello, world!' > hello.txt
[root ~]# wall < hello.txt
[root ~]#
Broadcast message from root (Wed Jun 20 19:43:05 2018):
hello, world!
[root ~]# echo 'I will show you some code.' >> hello.txt
[root ~]# wall < hello.txt
[root ~]#
Broadcast message from root (Wed Jun 20 19:43:55 2018):
hello, world!
I will show you some code.

  4. Multiple orientation - tee.

    In addition to displaying the results of the command ls on the terminal, the following command will also be appended to ls Txt file.

    [root ~]# ls | tee -a ls.txt

alias

  1. alias

    [root ~]# alias ll='ls -l'
[root ~]# alias frm='rm -rf'
[root ~]# ll
...
drwxr-xr-x  2 root       root   4096 Jun 20 12:52 abc
...
[root ~]# frm abc

  2. unalias

    [root ~]# unalias frm
[root ~]# frm sohu.html
-bash: frm: command not found

text processing

  1. Character stream editor - sed.

    sed is a tool for manipulating, filtering and transforming text content. Suppose there is a named fruit Txt file, as shown below.

    [root ~]# cat -n fruit.txt 
     1  banana
     2  grape
     3  apple
     4  watermelon
     5  orange

    Next, we add a pitaya after line 2.

    [root ~]# sed '2a pitaya' fruit.txt 
banana
grape
pitaya
apple
watermelon
orange

    Note: the command just now, like many commands we talked about before, does not change fruit Txt file, but output the content added with a new line to the terminal. If you want to save it to fruit Txt, you can use the output redirection operation.

    Insert a waxberry before line 2.

    [root ~]# sed '2i waxberry' fruit.txt
banana
waxberry
grape
apple
watermelon
orange

    Delete line 3.

    [root ~]# sed '3d' fruit.txt
banana
grape
watermelon
orange

    Delete lines 2 to 4.

    [root ~]# sed '2,4d' fruit.txt
banana
orange

    Replace the character a in the text with @.

    [root ~]# sed 's#a#@#' fruit.txt 
b@nana
gr@pe
@pple
w@termelon
or@nge

    Replace the character a in the text with @, using global mode.

    [root ~]# sed 's#a#@#g' fruit.txt 
b@n@n@
gr@pe
@pple
w@termelon
or@nge

  2. Pattern matching and processing language - awk.

    awk is a programming language and the most powerful tool for processing text in Linux system. One of its authors and current maintainers is Brian Kernighan (ken and dmr's closest partner) mentioned earlier. Through this command, we can extract the specified column from the text, use regular expression to extract the content we want from the text, display the specified row, and carry out statistics and operations. In short, it is very powerful.

    Suppose there is a named fruit 2 Txt file, as shown below.

    [root ~]# cat fruit2.txt 
1       banana      120
2       grape       500
3       apple       1230
4       watermelon  80
5       orange      400

    Displays line 3 of the file.

    [root ~]# awk 'NR==3' fruit2.txt 
3       apple       1230

    Displays the second column of the file.

    [root ~]# awk '{print $2}' fruit2.txt 
banana
grape
apple
watermelon
orange

    Displays the last column of the file.

    [root ~]# awk '{print $NF}' fruit2.txt 
120
500
1230
80
400

    Output lines with the end number greater than or equal to 300.

    [root ~]# awk '{if($3 >= 300) {print $0}}' fruit2.txt 
2       grape       500
3       apple       1230
5       orange      400

    What is shown above is only the tip of the iceberg of awk command. More content is left for readers to explore in practice.

Topics: Python Linux

Hot Topics

©2024 Programmer Think