Monitor CVE-2022 related information in Github through nail group to realize robot message push.

1, Database information

Create the mysql database cvemonitor and import the sql file into the database. The sql statement is as follows:

SET NAMES utf8mb4;
SET FOREIGN_KEY_CHECKS = 0;

-- ----------------------------

-- Table structure for github_monitor

-- ----------------------------

DROP TABLE IF EXISTS `github_monitor`;
CREATE TABLE `github_monitor`  (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `cve_name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `cve_url` varchar(100) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL,
  `cve_description` mediumtext CHARACTER SET utf8 COLLATE utf8_general_ci,
  PRIMARY KEY (`id`) USING BTREE,
  UNIQUE INDEX `cve_name`(`cve_name`) USING BTREE
) ENGINE = MyISAM AUTO_INCREMENT = 1 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

SET FOREIGN_KEY_CHECKS = 1;

2, Monitoring script

Note: the script needs execution permission chmod 777 cve_run.sh

import requests
import json
import dingtalkchatbot.chatbot as cb
import time
import pymysql

# Connect database
conn = pymysql.connect(host="Database host IP", user="Database account name", password="Database password", database="Database name",charset="utf8")
# Set the webhook value and secukey value of the nail group robot
webhook = 'Fill in the user-defined robot of nailing intelligent group assistant here webhook'
sckey = 'Fill in the key of nail smart group assistant custom robot here'

def dingding(text, msg,webhook,secretKey):
    ding = cb.DingtalkChatbot(webhook, secret=secretKey)
    ding.send_text(msg='{}\r\n{}'.format(text, msg), is_at_all=False)

def check(cve_name,url,cve_description):
    try:
        
        # Get a cursor object that can execute SQL statements
        cursor = conn.cursor()
        sql = '''insert into github_monitor(cve_name,cve_url,cve_description) values('{}','{}','{}')'''.format(cve_name,url,cve_description)
        cursor.execute(sql)
        print("Database inserted successfully")
        # Execute SQL statement
        body = "CVE No.:" + cve_name + "\r\n" + "Github Address:" + url + "\r\n" +"Description:"+ cve_description + "\r\n"
        dingding('There are new ones CVE find', body, webhook, sckey)
        print("Nail sending CVE success")
    except Exception as e:
        print('Should CVE{}Already exists'.format(cve_name))
    except Exception as e:
        pass


if __name__ == '__main__':
    while True:
        now_time=time.strftime("%Y-%m-%d %H:%M:%S", time.localtime())
        print("************************************************")
        print('*     The current detection time is:'+now_time+'      *')
        print("************************************************")
        github_headers = {
            'Authorization': 'token {}'.format('Fill in here github of token') 
        }
        url = 'https://api.github.com/search/repositories?q=CVE-2022&sort=updated'
        try:
            
            result = requests.get(url=url, headers=github_headers).text
            result = json.loads(result)
            for i in range(1, 20):
                url = result['items'][i]['html_url']
                cve_name = result['items'][i]['name']
                cve_description = result['items'][i]['description']
                check(cve_name, url,cve_description)
        except Exception as e:
            pass
        time.sleep(30*60) #Every 30 minutes, the number of seconds to delay execution

3, Configure the shell script (cve_run.sh) to execute the monitor

The script detects whether the server has a process to monitor the script. If there is no process, execute the command to start the script in the background

Note: the script needs execution permission chmod 777 cve_run.sh

#!/bin/bash
export PATH=usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/bin/java/bin:usr/lib/jvm/java-1.8.0-openjdk-1.8.0.302.b08-0.el7_9.x86_64/jre/bin/java/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/netcat-0.7.1/bin:/root/bin

ps aux |grep "python3 -u /opt/cvemonitor/cve-monitor.py" |grep -qv "grep"
if [ $? -ne 0 ];then
    rm -rf out.log
    nohup python3 -u /opt/cvemonitor/cve-monitor.py >/opt/cvemonitor/out.log 2>&1 &
    echo "start-up cve Monitoring script succeeded"
else
    echo "cve Monitoring script is running"
fi

4, Configure scheduled tasks

1. CVE scheduled start task_ run. SH script

vi /etc/crontab

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
# Time sharing day month week 
# Execute the script every minute

* * * * *  /opt/cvemonitor/cve_run.sh

2. Save scheduled tasks and view scheduled task execution logs

crontab /etc/crontab   #Save and load the scheduled task to make it effective
service crond restart   #Restart the scheduled service and take effect immediately
# View scheduled tasks
crontab -l
# Use the command to view the execution log of crontab: / var/log/cron will only record whether some scheduled scripts have been executed
tail -f /var/log/cron

5, Nail group assistant and Github

1. Creating nail intelligent group assistant robot

Add robot:

Select a custom robot:

Copy the key and Webhook into the python script:

2. Github Token value

Enter the configuration of personal account and select Developer settings:

Copy the Token value in the figure above to the python script.

6, Start monitoring script and precautions

1. Startup script

Use nohup to start the monitoring script CVE monitor in the background Py and save the output to out log

nohup python3 -u /opt/cvemonitor/cve-monitor.py >/opt/cvemonitor/out.log 2>&1 &

1. The last "&" indicates the background running program

2. "nohup" means that the program is not suspended

3. "Python3" means to execute Python code using python3

4. "- U" means that the cache is not enabled and the print information is output to the log file in real time (if - u is not added, the log file will not refresh the information of the print function in the code in real time)

5. "test.py" indicates the source code file of python

6. "test.log" indicates the output log file

7. ">" indicates that the print information is redirected to the log file

8. "2 > & 1" means that the standard error output is changed to standard output, and the error information can also be output to the log file (0 - > stdin, 1 - > stdout, 2 - > stderr)

2. Restart the scheduled task to ensure that the scheduled task takes effect

service crond restart

3. Problems encountered

After the scheduled task is configured and started, I can see that the scheduled task executes CVE normally by checking the log cat /var/log/cron_ run. SH script but CVE monitor Py script did not start normally.

Come to CVE_ run. The directory of the SH script. Use the command sh cve_run.sh executes the shell script, and then checks the process. It is found that the python script process executes normally; So why can't you start a python monitor program with a shell script through a scheduled task? It turned out to be the problem of environmental variables

Use the env |grep PATH command to view the environment variables of the current server, copy the contents of the paragraph PATH = and add it to the scheduled task script cve_run.sh in #/ The following line of bin/bash specifies the environment variables

#!/bin/bash
export PATH= 

6, Summary

Monitor CVE-2022 related information in Github through nail group to realize robot message push.

Main ideas:

1. Command background startup script
2. The scheduled task executes the shell script to detect whether the monitoring program is running. If the program fails, re execute the start command