catalogue
- Network connectivity detection
- Check socket connection
- Network usage detection
- Network packet capture
- transfer files
- Connection idle time
- summary
Network connectivity detection
In case of network abnormality in the application, the first thing to confirm is whether the network connectivity is normal. The following group of commands can quickly detect the network connectivity, as follows:
Detect DNS
dig www.baidu.combash nslookup www.baidu.combash host www.baidu.com
Check whether the host is reachable
ping www.baidu.com
Check whether the port is reachable
#Check tcp port telnet www.baidu.com 80 #Check udp port nc -uvz ip port
Detect SSL
SSL authentication also often leads to program failure to connect, mainly in the process of SSL handshake.
openssl s_client -connect www.baidu.com:443 -prexit
One key detection
In most cases, you can use curl to check all processes. If there are problems, you can use the above commands to check them one by one.
curl -v http://www.baidu.com:80/
Time consumption distribution
curl can be used to detect the time spent in each stage of the http protocol interface.
$ curl -o /dev/null -s -w " time_namelookup:%{time_namelookup}s\n time_connect:%{time_connect}s\n time_starttransfer:%{time_starttransfer}s\n time_total:%{time_total}s\n speed_download:%{speed_download}\n http_code:%{http_code}" "http://www.baidu.com"
time_namelookup:0.016542s
time_connect:0.038686s
time_starttransfer:0.063550s
time_total:0.063593s
speed_download:37793.000
http_code:200
time_namelookup: the time from the beginning to the completion of DNS query
time_connect: the time from the start of TCP handshake to the completion of three handshakes
time_starttransfer: the time from the start to receiving the first byte of data sent by the server
time_total: the time from the beginning to the completion of data reception at the server
Check socket connection
Since network communication depends on sockets, it is necessary to check the socket connection and its distribution.
Check whether the port is listening
The server program must listen to at least one port and check whether the listening socket exists. It is also a method to judge whether the service process still exists.
netstat -nltp|grep 8080 lsof -nP -i -sTCP:LISTEN|grep 8080
View socket status distribution
$ ss -s
$ netstat -nat | awk '/tcp/{print $6}'|sort|uniq -c
9 CLOSE_WAIT
102 ESTABLISHED
55 LISTEN
70 TIME_WAIT
Time needs special attention_ Wait and close_ The number of wait states, if time_ If there are too many waits, consider optimizing kernel network parameters or using connection pool. If close_ If there is too much wait, you need to check where the connection leak occurs in the program code, resulting in the connection not being closed.
Who even me most
netstat -ant | awk '/tcp/{rl=split($5,r,":");printf "%16s\t%s\n",$4,r[rl-1]}' | sort | uniq -c | sort -nrk1 | head -n10
Who do I have the most
netstat -ant | awk '/tcp/{ll=split($4,l,":");printf "%11s\t%s\n",l[ll-1],$5}' | sort | uniq -c | sort -nrk1 | head -n10
Network usage detection
Check the network speed of each connection
iftop -B -nNP
Check the network speed of each process
nethogs
Check the network speed of the network card
sar -n DEV 1 ifstat
Check whether the network card loses packets
# ifconfig command, observe the items of overrun/error/drop ifconfig # Similarly, observe items like overflow, error, and drop ethtool -S eth0
TCP layer packet loss and retransmission
Sometimes, there is no packet loss in the network card layer, but there may be packet loss in the network intermediate link, which will lead to retransmission in the tcp layer. In addition, if the kernel parameters of the tcp layer are set unreasonably, it may also lead to packet loss. For example, the backlog setting is too small, and the server-side network io cannot handle it.
$ sar -n TCP,ETCP 1
$ sudo watch -d -n1 'netstat -s|grep -iE "listen|pruned|collapsed|reset|retransmit"'
2879 connection resets received
378542 segments retransmitted
3357875 resets sent
52 resets received for embryonic SYN_RECV sockets
5 times the listen queue of a socket overflowed
5 SYNs to LISTEN sockets dropped
TCPLostRetransmit: 235599
6337 fast retransmits
7877 retransmits in slow start
10385 connections reset due to unexpected data
1183 connections reset due to early user close
Network packet capture
Plain text capture
# ngrep is more suitable for capturing plain text protocols such as http sudo ngrep -W byline port 3306 # When the packet capturing command cannot be used, you can also use network tools such as nc and socat to do a port forwarding and print the forwarding traffic at the same time # In addition, when capturing https packets, you can also use socat to proxy https traffic into http traffic, and then capture packets socat -v TCP4-LISTEN:9999,bind=0.0.0.0,reuseaddr TCP4:remoteIp:9999
General packet capture tool
# tcpdump packet capture to wireshark analysis sudo tcpdump tcp -i eth1 -s 0 -c 10000 and port 9999 -w ./target.cap # Grab the rst packet, which is used in the case that connection reset exceptions often occur in the network sudo tcpdump -ni any -s0 tcp and 'tcp[13] & 4 != 0 ' -vvv # Catch the fin packet, which is used when the network is often disconnected sudo tcpdump -ni any -s0 tcp and 'tcp[13] & 1 != 0 ' -vvv
mysql packet capture
$ sudo tshark -i eth0 -n -f 'tcp port 3306' -Y 'mysql' -T fields -e frame.number -e frame.time_epoch -e frame.time_delta_displayed -e ip.src -e tcp.srcport -e tcp.dstport -e ip.dst -e tcp.stream -e tcp.len -e tcp.nxtseq -e tcp.time_delta -e tcp.analysis.ack_rtt -e mysql.query Running as user "root" and group "root". This could be dangerous. Capturing on 'ens33' 4 1605412440.114466205 0.000000000 10.224.72.135 3306 59016 10.221.38.217 0 88 89 0.001027726 6 1605412440.160709874 0.046243669 10.221.38.217 59016 3306 10.224.72.135 0 185 186 0.000020998 8 1605412440.160929986 0.000220112 10.224.72.135 3306 59016 10.221.38.217 0 48 137 0.000211802 9 1605412440.213810997 0.052881011 10.221.38.217 59016 3306 10.224.72.135 0 24 210 0.052881011 0.052881011 11 1605412440.214178087 0.000367090 10.224.72.135 3306 59016 10.221.38.217 0 22 159 0.000341184 12 1605412440.258391363 0.044213276 10.221.38.217 59016 3306 10.224.72.135 0 37 247 0.044213276 0.044213276 select @@version_comment limit 1 14 1605412440.258812895 0.000421532 10.224.72.135 3306 59016 10.221.38.217 0 83 242 0.000395748 15 1605412440.303693157 0.044880262 10.221.38.217 59016 3306 10.224.72.135 0 13 260 0.044880262 0.044880262 select 1 16 1605412440.303955060 0.000261903 10.224.72.135 3306 59016 10.221.38.217 0 49 291 0.000261903 0.000261903 17 1605412440.351387241 0.047432181 10.221.38.217 59016 3306 10.224.72.135 0 5 265 0.047432181 0.047432181
grpc packet capture
For grpc packet capture, you can use tcpdump to capture it first, and then view it in wireshark. You can also use this project I found from GitHub https://github.com/rmedvedev/grpcdump
sudo grpcdump -i eth0 -p 9999 -proto-path ~/protos -proto-files order/v1/log_service.proto
transfer files
Using scp
#Upload files to remote machine scp test.txt root@remoteIp:/home/ #Download files from remote machines scp root@remoteIp:/home/test.txt .
Using ncat
ncat is often referred to as nc, but because netcat is also called nc and its usage is slightly different (nc on ubuntu is netcat), ncat is directly used here to avoid confusion
# Receiving file end ncat -l 9999 > test.txt # Send file side ncat remoteIp 9999 < test.txt
Using python http server
python's http server is often used to share native files to others, which is very convenient.
python -m SimpleHTTPServer 8000 wget http://remoteIp:8000/test.txt
Using python ftp server
Using python, you can quickly build an ftp server, so that you can upload and download.
sudo pip3 install pyftpdlib python3 -m pyftpdlib -p 2121 -w #Upload to ftp curl ftp://remoteIp:2121/files/ -T file.txt #Download from ftp curl -O ftp://remoteIp:2121/files/file.txt
Connection idle time
Using the time command, plus nc or telnet, you can see the maximum idle time of the connection, which is about 15 seconds as follows
$ time nc -v www.zhihu.com 443 Ncat: Version 7.60 ( https://nmap.org/ncat ) Ncat: Connected to 58.49.157.164:443. Ncat: 0 bytes sent, 0 bytes received in 15.12 seconds. real 0m15.132s user 0m0.000s sys 0m0.031s $ time telnet www.zhihu.com 443 Trying 14.215.85.42... Connected to www.zhihu.com. Escape character is '^]'. Connection closed by foreign host. real 0m15.052s user 0m0.001s sys 0m0.002s
summary
It is very necessary to master the commonly used network commands. After all, the network is so complex that something must be able to spy on some internal operation information.