Although most of my work is Java related development, I am exposed to Linux every day, especially after using Mac. I work in the command line environment with black background every day I don't have a good memory. I can't remember many useful linux commands very well. Now I'll summarize them gradually for later viewing.
basic operation
Linux shutdown, restart
#Shut down shutdown -h now #Restart shutdown -r now
View system and CPU Information
#View system kernel information uname -a #View system kernel version cat /proc/version #View current user environment variables env cat /proc/cpuinfo #See how many logical CPUs are available, including cpu models cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c #Check how many CPUs are there, and how many cores are each cat /proc/cpuinfo | grep physical | uniq -c #Check whether the current CPU is running in 32bit or 64bit mode. If it is running in 32bit mode, it does not mean that the CPU does not support 64bit getconf LONG_BIT #The result is greater than 0, indicating that 64 bit calculation is supported lm refers to long # mode, while supporting lm is 64bit cat /proc/cpuinfo | grep flags | grep ' lm ' | wc -l
Establish soft connection
ln -s /usr/local/jdk1.8/ jdk
rpm correlation
#Check whether the software is installed through rpm rpm -qa | grep Software name
sshkey
#Create sshkey ssh-keygen -t rsa -C your_email@example.com #id_ rsa. Copy the contents of pub to the "home / username /. Of the server to be controlled ssh/authorized_ Keys ， if not, create a new one (. SSH permission is 700, authorized_keys permission is 600)
Command rename
#In the of each user bash_ Add rename configuration in profile alias ll='ls -alF'
Synchronize server time
sudo ntpdate -u ntp.api.bz
Background run command
#Run in the background and have nohup Out output nohup xxx & #Run in the background without outputting any logs nohup xxx > /dev/null & #Run in the background and output the error information to the log as a standard nohup xxx >out.log 2>&1 &
Force active user exit
#Command to complete the force active user exit Where TTY represents the terminal name pkill -kill -t [TTY]
View command path
which <command>
View the maximum fd number of all open processes
ulimit -n
Configure dns
vim /etc/resolv.conf
nslookup to view the domain name routing table
nslookup google.com
last, list of recent login information
#5 recently logged in accounts last -n 5
Set fixed ip
ifconfig em1 192.168.5.177 netmask 255.255.255.0
View environment variables loaded in process
#You can also go to the cd / proc directory to view the things loaded in the process memory ps eww -p XXXXX(Process number)
View the process tree to find the server process
ps auwxf
View process startup path
cd /proc/xxx(Process number) ls -all #cwd corresponds to the startup path
Add users and configure sudo permissions
#New user useradd user name passwd user name #Add sudo permission vim /etc/sudoers #Modification of documents inside # root ALL=(ALL) ALL #User name: ALL=(ALL) ALL
Force the shutdown of all processes whose process name contains xxx
ps aux|grep xxx | grep -v grep | awk '{print $2}' | xargs kill -9
Disk, file and directory related operations
vim operation
#In normal mode, g represents the global, x represents the searched content, and y represents the replaced content :%s/x/y/g #In normal mode 0 #Move the cursor to the beginning of the line (number 0) $ #Move cursor to end of line shift + g #Skip to the end of the file gg #Jump to file header #Display line number :set nu #Remove line number :set nonu #Search /xxx(Search content) #Search from the beginning and press n to find the next one ?xxx(Search content) #Retrieve from tail
Open a read-only file and save it after modification (you can save it without switching users)
#In normal mode :w !sudo tee %
View basic information of disk, file and directory
#View disk mounting mount #View disk partition information df #View directory and subdirectory size du -H -h #Check the space occupied by each file and folder in the current directory without recursion du -sh *
wc command
#See how many lines there are in the file wc -l filename #See how many word s there are in the file wc -w filename #What is the longest line in the file wc -L filename #Count bytes wc -c
Common compression and decompression commands
Compression command
tar czvf xxx.tar Compressed directory zip -r xxx.zip Compressed directory
Decompression command
tar zxvf xxx.tar #Extract to the specified folder tar zxvf xxx.tar -C /xxx/yyy/ unzip xxx.zip
Change the user and user group to which the file belongs
chown eagleye.eagleye xxx.log
cp, scp, mkdir
#copy
cp xxx.log
#Copy and force overwrite of files with the same name
cp -f xxx.log
#Copy folder
cp -r xxx(Source folder) yyy(Destination folder)
#Remote replication
scp -P ssh port username@10.10.10.101:/home/username/xxx /home/xxx
#Cascade create directory
mkdir -p /xxx/yyy/zzz
#When creating folders in batch, java and resources folders will be created under test and main
mkdir -p src/{test,main}/{java,resources}
Compare two files
diff -u 1.txt 2.txt
The number of bytes of log output, which can be used as a performance test
#If you do a performance test, you can output "." to the log every time you execute it In this way, the number of bytes in the log is the actual number of performance test runs, and the real-time rate can be seen tail -f xxx.log | pv -bt
View, remove special characters
#View special characters cat -v xxx.sh #Remove special characters sed -i 's/^M//g’ env.sh ， remove the special characters of the file, such as ^ M: you need to enter: ctrl+v+enter
Handle the problem of special characters in files caused by system reasons
#It can be converted to the file format under the system cat file.sh > file.sh_bak #First set file Copy the contents of the file in SH, then run it, then paste the contents, and finally ctrl + d # save and exit cat > file1.sh #In vim, set the file code and file format as follows :set fileencodings=utf-8 ，then w (Save it) and it can be converted into utf8 Format, :set fileformat=unix #Using dos2unix to format files under mac find . -name "*.sh" | xargs dos2unix
tee, output to the screen while redirecting
awk '{print $0}' xxx.log | tee test.log
Search correlation
grep
#Reverse matching to find content that does not contain xxx
grep -v xxx
#Exclude all blank lines
grep -v '^/pre>
#If the result {2 is returned, the second line is empty
grep -n "^$" 111.txt
#Query rows starting with abc
grep -n "^abc" 111.txt
#Also list the lines on which the word appears in the article
grep 'xxx' -n xxx.log
#Count the number of occurrences of the string
grep 'xxx' -c xxx.log
#When comparing, do not care about the difference between case and case
grep 'xxx' -i xxx.log
awk
#Take ':' as the separator. If there is a user in the fifth field, this line will be output
awk -F ':' '{if ($5 ~ /user/) print $0}' /etc/passwd
#Count the number of occurrences of a character (string) (Chinese is invalid) in a single file
awk -v RS='character' 'END {print --NR}' xxx.txt
find search command
#Find the suffix in the directory mysql files
find /home/eagleye -name '*.mysql' -print
#Start from the / usr directory to find the files that have been fetched from the memory in the last three days.
find /usr -atime 3 –print
#I will start from the / usr directory to find the files that have been modified in the last 5 days.
find /usr -ctime 5 –print
#I will start from the / doc directory to find the file of jacky whose file name starts with j.
find /doc -user jacky -name 'j*' –print
#Start from the / doc directory and look for files with file names beginning with ja or ma.
find /doc \( -name 'ja*' -o- -name 'ma*' \) –print
#You will start from the / doc directory to find the file with bak at the end of the file name and delete it- The exec option means to execute, rm means to delete the command, {} means the file name, "\;" Is the end of the specified command.
find /doc -name '*bak' -exec rm {} \;
Network related
See what processes use this port
lsof -i:por
Get local ip address
/sbin/ifconfig -a|grep inet|grep -v 127.0.0.1|grep -v inet6|awk '{print $2}'|tr -d "addr:"
iptables
#View iptables status service iptables status #To block an ip iptables -I INPUT -s ***.***.***.*** -j DROP #To unseal an IP, use the following command: iptables -D INPUT -s ***.***.***.*** -j DROP remarks: parameter-I Yes, it means Insert(Add),-D express Delete(Delete). Followed by rules, INPUT Indicates inbound,***.***.***.***It means to shut down IP，DROP Indicates abandoning the connection. #Open access to port 9090 /sbin/iptables -I INPUT -p tcp --dport 9090 -j ACCEPT #Firewall on, off and restart /etc/init.d/iptables status /etc/init.d/iptables start /etc/init.d/iptables stop /etc/init.d/iptables restart
nc command, tcp debugging tool
#When sending a TCP request to an endpoint, the content of data is sent to the opposite end nc 192.168.0.11 8000 < data.txt #nc can be used as a server to listen to a port number and store the contents of a request in received_ In data nc -l 8000 > received_data #The upper side only listens once. If it listens for multiple times, the - k parameter can be added nc -lk 8000
tcpdump
#dump the tcp packet of the local port 12301 tcpdump -i em1 tcp port 12301 -s 1500 -w abc.pcap
Track network routing path
#traceroute uses udp mode by default. If it is - I, it will be changed to icmp mode traceroute -I www.163.com #Track from ttl 3rd hop traceroute -M 3 www.163.com #Plus port tracking traceroute -p 8080 192.168.10.11ss #Displays all ports opened locally ss -l # Displays the specific open for each process socket ss -pl # Show all tcp socket ss -t -a #Show all UDP Socekt ss - u - a #Displays all established SMTP connections ss -o state established '( dport = :smtp or sport = :smtp )' #Displays all established HTTP connections ss -o state established '( dport = :http or sport = :http )' #Find all processes connected to the X server ss -x src /tmp/.X11-unix/* #List current socket statistics ss -s Explanation: netstat Is traversal/proc Each of the following PID catalogue ss Direct reading/proc/net The following statistics. therefore ss It consumes more resources and time than netstat Much less
netstat
#Output the number of connections per ip and the total number of connections in each state
netstat -n | awk '/^tcp/ {n=split($(NF-1),array,":");if(n<=2)++S[array[(1)]];else++S[array[(4)]];++s[$NF];++N} END {for(a in S){printf("%-20s %s\n", a, S[a]);++I}printf("%-20s %s\n","TOTAL_IP",I);for(a in s) printf("%-20s %s\n",a, s[a]);printf("%-20s %s\n","TOTAL_LINK",N);}'
#Count all connection status
# CLOSED: no connection is active or in progress
# LISTEN: the server is waiting for an incoming call
# SYN_RECV: a connection request has arrived, waiting for confirmation
# SYN_SENT: the application has started. Open a connection
# ESTABLISHED: normal data transmission status
# FIN_WAIT1: the application says it's finished
# FIN_WAIT2: the other side has agreed to release
# ITMED_WAIT: wait for all packets to die
# CLOSING: both sides try to close at the same time
# TIME_WAIT: the state of actively closing one end of the connection before waiting for feedback from the other end
# LAST_ACK: wait for all packets to die
netstat -n | awk '/^tcp/ {++state[$NF]} END {for(key in state) print key,"\t",state[key]}'
#Find more time_wait connection
netstat -n|grep TIME_WAIT|awk '{print $5}'|sort|uniq -c|sort -rn|head -n20
Monitoring linux performance commands
top
Press the uppercase F or O key, and then press a-z to sort the processes according to the corresponding columns, and then enter. The uppercase R key can reverse the current sort.
PID process id PPID Parent process id RUSER Real user name UID User of the process owner id USER User name of the process owner GROUP Group name of the process owner TTY The name of the terminal that started the process. Processes that are not started from the terminal are displayed as ? PR priority NI nice Value. Negative values indicate high priority and positive values indicate low priority P Last used CPU，Only in many CPU Meaningful in the environment %CPU From last update to now CPU Time occupancy percentage TIME Used by the process CPU Total time in seconds TIME+ Used by the process CPU Total time, unit 1/100 second %MEM Percentage of physical memory used by the process VIRT Total amount of virtual memory used by the process, in kb. VIRT=SWAP+RES SWAP The size, unit, of the virtual memory used by the process kb. RES The size of physical memory used by the process and not swapped out, unit: kb. RES=CODE+DATA CODE Physical memory occupied by executable code, unit kb DATA Parts other than executable code(Data segment+Stack)Size of physical memory occupied, unit kb SHR Shared memory size in kb nFLT Number of page errors nDRT The number of pages that have been modified since the last write. S Process status. D=A state of uninterrupted sleep,R=function,S=sleep,T=track/stop it,Z=Zombie process COMMAND Command name/command line WCHAN If the process is sleeping, the system function name in sleep is displayed Flags Task flag, reference sched.h
dmesg, view the system log
dmesg
iostat, disk IO condition monitoring
iostat -xz 1 # r/s, w/s, rkB/s, wkB/s: respectively represents the number of reads and writes per second and the amount of data read and written per second (kilobytes). Excessive reading and writing may cause performance problems. # await: average wait time of IO operation, in milliseconds. This is the time consumed when the application interacts with the disk, including IO waiting and actual operation time. If this value is too large, the hardware device may encounter a bottleneck or failure. # avgqu-sz: the average number of requests sent to the device. If this value is greater than 1, the hardware device may be saturated (some front-end hardware devices support parallel writing). #Utilization rate of equipment: util%. This value indicates the busy degree of the equipment. The empirical value is that if it exceeds 60, the IO performance may be affected (refer to the average waiting time of IO operation). If it reaches 100%, the hardware device has been saturated. #If the data of logical devices is displayed, the device utilization does not mean that the actual hardware devices at the back end have been saturated. It is worth noting that even if the IO performance is not ideal, it does not necessarily mean that the application performance will be poor. Strategies such as pre read and write cache can be used to improve the application performance.
free, memory usage
free -m eg: total used free shared buffers cached Mem: 1002 769 232 0 62 421 -/+ buffers/cache: 286 715 Swap: 1153 0 1153
Part I Mem line:
total Total memory: 1002M used Memory used: 769M free Free memory: 232M shared It has been abandoned,Always 0 buffers Buffer Cache memory: 62M cached Page Cache memory:421M
Relationship: total(1002M) = used(769M) + free(232M)
Part 2 (- / + buffers/cache):
(-buffers/cache) used Number of memory: 286 M (Refers to the first part of the Mem In line used – buffers – cached) (+buffers/cache) free Number of memory: 715M (Refers to the first part of the Mem In line free + buffers + cached)
It can be seen that - buffers/cache reflects the memory actually eaten by the program, while + buffers/cache reflects the total amount of memory that can be misappropriated
The third part refers to the switching partition
sar to view the network throughput status
# Here you can view the throughput of network devices with the sar command. When troubleshooting performance problems, you can judge whether the network equipment is saturated by the throughput of the network equipment. sar -n DEV 1 # # The sar command is used here to view the TCP connection status, including: # active/s: the number of locally initiated TCP connections per second, i.e. TCP connections created through connect call; # passive/s: the number of remote initiated TCP connections per second, that is, the TCP connections created through the accept call; # retrans/s: number of TCP retransmissions per second; # The number of TCP connections can be used to determine whether the performance problem is due to the establishment of too many connections, and further determine whether the connection is actively initiated or passively accepted. TCP retransmission may be caused by poor network environment or excessive server pressure, resulting in packet loss sar -n TCP,ETCP 1
vmstat, monitor CPU utilization, memory usage, virtual memory interaction, IO read and write at a given time
# 2 means to collect status information every 2 seconds, and 1 means to collect only once (ignore or collect all the time) vmstat 2 1 eg: r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 3499840 315836 3819660 0 0 0 1 2 0 0 0 100 0 0 0 0 3499584 315836 3819660 0 0 0 0 88 158 0 0 100 0 0 0 0 3499708 315836 3819660 0 0 0 2 86 162 0 0 100 0 0 0 0 3499708 315836 3819660 0 0 0 10 81 151 0 0 100 0 1 0 0 3499732 315836 3819660 0 0 0 2 83 154 0 0 100 0
-
r indicates the running queue (that is, how many processes are actually allocated to the CPU). At present, the CPU of the server I tested is relatively idle, and there are no programs running. When this value exceeds the number of CPUs, there will be a CPU bottleneck. This is also related to the load of the top. Generally, the load is higher when it exceeds 3, higher when it exceeds 5, and abnormal when it exceeds 10. The state of the server is very dangerous. The load of top is similar to the running queue per second. If the running queue is too large, it indicates that your CPU is very busy, which generally leads to high CPU utilization.
-
b represents a blocked process. I won't say much about it. Process blocking, you know.
-
If the size of swpd virtual memory used is greater than 0, it indicates that your machine is out of physical memory. If it is not the cause of program memory leakage, it is time to upgrade memory or migrate memory consuming tasks to other machines.
-
Free is the size of free physical memory. My machine memory is 8G in total, with 3415M remaining.
-
buff Linux/Unix system is used to store the cache of contents, permissions, etc. in the directory. My local machine takes about more than 300 M
-
cache cache is directly used to remember the files we open and buffer the files. My machine occupies about more than 300 m (here is the wisdom of Linux/Unix. Using part of the free physical memory as the cache of files and directories is to improve the performance of program execution. When the program uses memory, buffer/cached will be used quickly.)
-
The amount of virtual memory that si reads from the disk every second. If this value is greater than 0, it means that the physical memory is not enough or the memory is leaked. Find the memory consuming process and solve it. My machine has plenty of memory and everything is normal.
-
so the size of virtual memory written to disk per second. If this value is greater than 0, the same as above.
-
The number of blocks received by the bi block device per second. The block device here refers to all disks and other block devices on the system. The default block size is 1024byte. There is no IO operation on my machine, so it has always been 0. However, I have seen it on the machine that processes and copies a large amount of data (2-3T). It can reach 140000/s, and the disk write speed is almost 140M per second
-
bo block the number of blocks sent by the device per second. For example, when we read a file, bo must be greater than 0. bi and bo are generally close to 0, or IO is too frequent and needs to be adjusted.
-
in the number of CPU interrupts per second, including time interrupts
-
cs the number of context switches per second. For example, when we call a system function, we need to switch the context, thread and process context. The smaller the value, the better. If it is too large, we should consider reducing the number of threads or processes, such as in web servers such as apache and nginx, When we do performance test, we usually conduct thousands or even tens of thousands of concurrent tests. The process of selecting Web server can be lowered by the peak value of process or thread until cs reaches a relatively small value, and the number of processes and threads is a more appropriate value. System call is the same. Every time we call the system function, our code will enter the kernel space, resulting in context switching. This is very resource consuming. We should also try to avoid calling the system function frequently. Too many context switches means that most of your CPU is wasted on context switching, resulting in less time for the CPU to do serious things, and it is not advisable to make full use of the CPU.
-
The CPU time of us users. I used to work on a server that encrypts and decrypts frequently. I can see that us is close to 100 and the R running queue reaches 80 (the machine is doing stress testing and its performance is poor).
-
sy system CPU time. If it is too high, it indicates that the system call time is long, such as frequent IO operations.
-
ID idle CPU time. Generally speaking, id + us + sy = 100. Generally, I think id is the idle CPU utilization rate, us is the user CPU utilization rate, and sy is the system CPU utilization rate.
-
wt wait IO CPU time.