One of the most common problems in Travelling is that the site is slow or inaccessible, and what you see on the server is that the site is CC***.CC*** is one of the DDos*** types, and ** users use a large number of "chickens" to initiate requests to the site they are targeting, which is so frequent that the servers of the site they are targeting can't stand it.
CC*** Although it seems that there is little difference from normal access, if we carefully analyze the access log, we can still find some clues, such as the high frequency of access to an IP, or the fixed user_agent of several IPs, etc. Some even imitate the regular search engine, such as, setting yourself up"Spider and crawler" disguised as Baidu.When you encounter CC***, if you take some effort to analyze the access log, you can eventually find out the culprit who started CC***, and then we can do everything by blocking their IP.The requirements for this case are:
1) Access log/data/logs/access_log is analyzed every minute with the following log fragments:
61.227.224.229 - [19/Sep/2018:09:30:07 + 0800 ] "/misc.php?mod=ranklist&type=menber&view=post" 200 "GET HTTP/1.1" "http://www.wenyou.com/" "Mozilla/5.0(Windows NT 6.1;WOW64;rv:43.0)Gecko/20100101 Firefox/43.0" 61.227.224.229 - [19/Sep/2018:09:30:07 + 0800 ] "/misc.php?mod=ranklist&type=menber&view=onlinetime" 200 "GET HTTP/1.1" "http://www.wenyou.com/" "Mozilla/5.0(Windows NT 6.1;WOW64;rv:43.0)Gecko/20100101 Firefox/43.0"
2) Block IP with more than 100 visits
3) Enclosed IP s should be recorded in a log
4) Check the blocked IP every 30 minutes to unblock the IP with little or no access
5) Unencapsulated IP is recorded in another log
Point One: awk usage
In shell scripts, awk is highly likely to occur because of its strong ability to handle strings.Let's start with a small example:
# awk -F ":" '$3>500 {print $1,$3}' /etc/passwd
polkitd 999
chrony 998
mysql 1002Description: The core function of awk is to segment. You can specify a delimiter with the -F option and then process it against a field. In this case, use':'as the delimiter to find lines with a third paragraph larger than 500, and print out the first and third paragraphs.Logical judgment symbols such as >, <, >=, <=, ==,!= can be used in awk, which differs from shell s.In this case, do not put double quotation marks on the 500 after >, otherwise it will not compare numbers, but will treat 500 as strings, the resu lt will be different naturally.
The following:
# awk -F ":" '$3>"500" {print $1,$3}' /etc/passwd
shutdown 6
halt 7
mail 8
nobody 99
dbus 81
polkitd 999
postfix 89
sshd 74
chrony 998awk does more than that; it's actually a scripting language, like shell, that can write scripts.Its author designed it to format the output text, which can meet a variety of complex formatting requirements, but when we write shell scripts, we just use it as a command to process strings.Common usage:
1) Intercept the specified segment
# awk -F ':|#' {print $2}' 1.txtDescription: The separator can be a regular expression, or # in this case.
2) String after matching a character
# awk -F ':' '$1 ~ "abc"' 1.txt
Description: Filter out the first line containing abc, where ABC can be a regular expression, for example:
# awk -F ':' '$1 ~ "^ro+"' 1.txt
Description: ^ means the beginning, + means at least one character before + so the strings that ^ro+ can match are: ro, roo, rooo...
3) Use multiple statements at the same time
# awk -F ':' '$1 ~ "root" {print $1,$3};$3>100 {print $1,$2}' 1.txt4) Multiple Conditions
# awk '$3<100 && $7 ~ "bash" {print $0}' 1.txtDescription: If no delimiter is specified, a blank character is used as the delimiter, which can be represented by &&in awk and || or.$0 outputs the entire line.
5) Built-in variables
# awk -F ':' {print NF,NR}' 1.txtDescription: NF is the number of segments, NR is the number of rows
6) Mathematical operations
# awk -F ':' '$7=$3+$4 {print $0}' 1.txtBecause the value of $7 has been redefined, when printing $0, there will be no separator included, and of course, a separator can also be defined:
# awk -F ':' -v OFS='#' '$7=$3+$4 {print $0}' 1.txtDescription: Variables are defined with -v, and this OFS is also a built-in variable that represents the delimiter of the output.
It is also common to calculate the sum of a paragraph, for example:
# awk -F ':' '{(tot=tot+$3)}; END {print tot}' 1.txtPoint 2: sort ordering
Syntax: sort [-t delimiter] [-kn1,n2] [-nru] where n1<n2
-t separator: function has the same meaning as cut's-d
-n: Sort using pure numbers
-r: reverse sort
-u: to repeat
-kn1,n2: Sort from n1 interval to N2 interval, you can write-kn1 only, that is, sort the n1 field
Point three: uniq to repeat
There is only one common option for this command: -c, count the number of rows that are repeated, and write the number of rows first.
# uniq 1.txt 111 222 333 111
The premise of using uniq is to sort the files first, otherwise it will not work, so it is often used with sort as follows:
# sort 1.txt |uniq 111 222 333 # sort 1.txt |uniq -c 2 111 1 222 1 333
Point 4: Encapsulate/Unencapsulate IP with iptables tools
1) View iptables rules for filter tables
# iptables -nvL -t filter
Note: On centOS7, netfilter has five tables: filter, nat, mangle, raw, secuirty.We use the first two more, in this case the envelope IP is a filter table.The -t filter in this command can be omitted and is the filter table by default.
2) Add Rules
iptables -A INPUT -p tcp --dport 80 -s 1.1.1.1 -j DROP
Description: -A option is to add a rule, or more accurately, to append a rule, because the rules of iptables are divided before and after, and the higher the rule is, that is, the higher the priority.Therefore, rules appended with -A can only be placed last, but if you want to add a higher priority rule, you can use the -I option (Insert Rule).
You can also write only IP for ports, but you can also specify the destination IP address as follows:
# iptables -I INPUT -s 3.3.3.3 -d 4.4.4.4 -j DROP
3) Delete Rule
There are two ways to delete a rule: empty it all, and delete one.
# iptables -t filter -F
Deleting a rule can be cumbersome, because you need to remember the command that created it at that time and change -A to -D. Of course, there is a simpler way to do this. First, list the sequence number of the rule, as follows:
# iptables -nvL --line-number
num in the first column on the left is the sequence number of the rules. The rules for each chain start with 1. If you want to delete rule 2 of the INPUT chain, you only need to execute it:
# iptables -D INPUT 2
This will remove rule 2 under the INPUT chain.After deleting the rule, re-list the rule ordinal, and you will find that the ordinal still starts with 1.
When we execute iptables-nvL, the first column of the results listed represents the number of packets affected by the iptables rule, while the second column, bytes, represents the size of the data affected by the iptables rule.This number keeps changing. If you restart the iptables or firewalld service, the two columns will be cleared, and of course you can have an option to do so:
# iptables -Z
You can also specify a chain, such as:
# iptables -Z INPUT
The above points of knowledge are mainly for this case, there are many uses of iptables.
Point 5: Functions in shell scripts
If a piece of code appears more than once in a shell script, it should be encapsulated in a function so that subsequent calls are convenient and the code looks nice.Here is an example to illustrate the benefits of the function.
Requirements: Check if vim-enhanced, expect, wget packages are installed on the system or if not, the traditional shell code is written as follows:
if ! rpm -q vim-enhanced &>/dev/null then yum install -y vim-enhanced fi if ! rpm -q expect &>/dev/null then yum install -y expect fi if ! rpm -q wget &>/dev/null then yum install -y wget fi
Just ask, if you check 100 packages, would you like to write this code 100 times?The functions are much simpler, as follows:
##Define if_install function
if_install() {
if ! rpm -q $1 &>/dev/null
then
yum install -y $1
fi
}
##Check all rpm packages one by one with a for loop
for pkg in vim-enhanced expect wget
do
if_install $pkg
donePoint 6: sed usage
sed and grep, awk appear frequently in shell scripts, and the three of them are often called "regular three swordsmen", so the relationship between these three tools and regular expressions is unusual.For the use of the sed tool, common uses are:
1) Print the specified line
sed -n '10p' 1.txt
Note: 10 refers to the line number, p refers to print, plus -n only shows the 10th line, otherwise all lines of 1.txt will be displayed, where single quotation marks, or double quotation marks.You can also specify multiple lines:
sed -n '1,5p' 1.txt
Description: Print 1 to 5 lines, if they are not continuous, this can be done:
sed -n '1p;5p' 1.txt
sed also has a similar function as grep to filter specified rows as follows:
sed -n '/abc/p' 1.txt
Description: //is a fixed notation, which contains the string to be matched and can be regular, for example:
sed -n '/[a-z0-9]/p' 1.txt
In regular expressions, how many special symbols belong to the extended regular category, are they +,?,|, (), {}.To use them in grep, you need to add the -E option or the egrep command, and to use them in sed, you need to add the -r option.
2) Delete the specified line
sed '10d' 1.txt
Note: Line 10 is not printed when printing 1.txt. It does not really delete the contents of the file. Adding the -i option can change the contents of the file.
3) Find Replacement
Replace all aming s that appear in 1.txt with inux
sed 's/aming/linux/g' 1.txt
Description: s means replacement, g means global replacement, and without -g only replaces the first aming that appears in each row.
This case reference script
#!/bin/bash
##Block the IP with a large amount of access. If the IP that is blocked for 20 minutes has no requests or very few requests, it needs to be unblocked
##Author:
##Date:
##Version: v0.1
#Define a time before 1 minute to filter logs before 1 minute
t1=`data -d "-1 min" +%Y:%H:%M`
log=/data/logs/access_log
block_ip(){
#Filter the log for any number of seconds in the previous minute, 0-5 for 5 of 59 seconds, and + for any 0-9.
egrep "$t1:[0-5]+" $log > /tmp/tmp_last_min.log
#Record IPS with more than 100 visits in 1 minute into a temporary file
awk '{print $1}' /tmp/tmp_last_min.log |sort -n|uniq -c|sort -n|awk '$1>100 {print $2}' > /tmp/bad_ip.list
#Calculate the number of IPS
n=`wc -l /tmp/bad_ip.list |awk '{print $1}'`
#When the number of iptables is greater than 0, it will be blocked with iptables
if [ $n -ne 0 ]
then
for ip in `cat /tmp/bad_ip.list`
do
iptables -I INPUT -s $ip -j REJECT
done
#Log these enclosed IP s
echo "`date` Sealed off IP Yes:" >> /tmp/block_ip.log
cat /tmp/bad_ip.list >> /tmp/block_ip.log
fi
}
unblock_ip()
{
#First log IPS with a number less than 5 into a temporary file and mark them as whitelist IP
iptables -nvL INPUT |sed '1d' |awk '$1<5 {print $8}' > /tmp/good_ip.list
n=`wc -l /tmp/good_ip.list |awk '{print $1}'`
if [ $n -ne 0 ]
then
for ip in `cat /tmp/good_ip.list`
do
iptables -D INPUT -s $ip -j REJECT
done
echo "`date` Unsealed IP Yes:" >> /tmp/unblock_ip.log
cat /tmp/good_ip.list >> /tmp/unblock_ip.log
fi
#After unpacking the whitelist IP, clear the counter to the next counting cycle
iptables -Z
}
#Number of minutes for the current time
t=`date +%M`
#When the number of minutes is 00 or 30 (i.e., every 30 minutes), the function to unpack IP is executed, while the other time only the function to unpack IP is executed
if [ $t == "00" ] || [ $t == "30" ];then
unblock_ip
block_ip
else
block_ip
fi