SaltStack return and job management

Posted by jemrys on Sun, 07 Nov 2021 17:54:33 +0100

SaltStack return and job management

return

The return component understands that the SaltStack system stores or returns data returned by Minion execution to other programs. It supports a variety of storage methods, such as MySQL, MongoDB, Redis, Memcache, and so on. By returning, we can record every operation of SaltStack and provide data sources for future log audits. There are currently 30 return data stores and interfaces that are officially supported and can be easily configured and used. Of course, it also supports your own defined return, which is written by python. After selecting and configuring the return to use, simply specify it after the salt command

View all return list
[root@master ~]# salt 'minion1' sys.list_returners
minion1:
    - carbon
    - couchdb
    - etcd
    - highstate
    - local
    - local_cache
    - mattermost
    - multi_returner
    - pushover
    - rawfile_json
    - slack
    - slack_webhook
    - smtp
    - splunk
    - sqlite3
    - syslog
    - telegram

return process

Return triggers the task on the Master side, then Minion accepts the processing task and connects directly to the return storage server, then saves the data return to the storage server. It is important to note that since this process is a Minion-side operational storage server, to ensure that the Minion-side configuration and dependency packages are correct, this means that we will have to install the specified return-style dependency packages on each Minion. If Mysql is used as the return storage, then we will install the python-mysql module on each Minion.

Using mysql as return storage

## At all minion Upper Installation Mysql-python Modular ##
[root@master ~]# salt 'minion1' pkg.install python3-PyMySQL
minion1:
    ----------
    python3-PyMySQL:
        ----------
        new:
            0.10.1-2.module_el8.4.0+666+456f5f48
        old:
    python3-cffi:
        ----------
        new:
            1.11.5-5.el8
        old:
    python3-cryptography:
        ----------
        new:
            3.2.1-4.el8
        old:
    python3-ply:
        ----------
        new:
            3.9-9.el8
        old:
    python3-pycparser:
        ----------
        new:
            2.14-14.el8
        old:
[root@minion1 ~]# rpm -qa|grep -i pymysql
python3-PyMySQL-0.10.1-2.module_el8.4.0+666+456f5f48.noarch

## Deploy one mysql Server as storage server ##
[root@mysql ~]# yum -y install mariadb mariadb-server
[root@mysql ~]# systemctl start mariadb
[root@mysql ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 8
Server version: 10.3.28-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database `salt`
    -> default character set utf8
    -> default collate utf8_general_ci;
Query OK, 1 row affected (0.000 sec)

MariaDB [(none)]> use `salt`;
Database changed
MariaDB [salt]> drop table if exists `jids`;
Query OK, 0 rows affected, 1 warning (0.000 sec)

MariaDB [salt]> create table `jids` (
    -> `jid` varchar(255) not null,
    -> `load` mediumtext not null,
    -> unique key `jid` (`jid`)
    -> ) engine=innodb default charset=utf8;
Query OK, 0 rows affected (0.003 sec)

MariaDB [salt]> drop table if exists `salt_returns`;
Query OK, 0 rows affected, 1 warning (0.000 sec)

MariaDB [salt]> create table `salt_returns` (
    -> `fun` varchar(50) not null,
    -> `jid` varchar(255) not null,
    -> `return` mediumtext not null,
    -> `id` varchar(259) not null,
    -> `success` varchar(10) not null,
    -> `full_ret` mediumtext not null,
    -> `alter_time` timestamp default current_timestamp,
    -> key `id` (`id`),
    -> key `jid` (`jid`),
    -> key `fun` (`fun`)
    -> ) engine=innodb default charset=utf8;
Query OK, 0 rows affected (0.004 sec)

MariaDB [salt]> drop table if exists `salt_events`;
Query OK, 0 rows affected, 1 warning (0.000 sec)

MariaDB [salt]> create table `salt_events` (
    -> `id` bigint not null auto_increment,
    -> `tag` varchar(255) not null,
    -> `data` mediumtext not null,
    -> `alter_time` timestamp default current_timestamp,
    -> `master_id` varchar(255) not null,
    -> primary key (`id`),
    -> key `tag` (`tag`)
    -> ) engine=innodb default charset=utf8;
Query OK, 0 rows affected (0.003 sec)

MariaDB [salt]> show tables;
+----------------+
| Tables_in_salt |
+----------------+
| jids           |
| salt_events    |
| salt_returns   |
+----------------+
3 rows in set (0.000 sec)
MariaDB [salt]> desc jids;
+-------+--------------+------+-----+---------+-------+
| Field | Type         | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+-------+
| jid   | varchar(255) | NO   | PRI | NULL    |       |
| load  | mediumtext   | NO   |     | NULL    |       |
+-------+--------------+------+-----+---------+-------+
2 rows in set (0.001 sec)

Authorized Access
MariaDB [salt]> grant all on salt.* to 'salt'@'%' identified by 'salt';
Query OK, 0 rows affected (0.000 sec)

MariaDB [salt]> flush privileges;
Query OK, 0 rows affected (0.000 sec)

## stay minion install mariadb´╝îTest Connection mysql The server ##
[root@minion1 ~]# yum -y install mariadb
[root@minion1 ~]# which mysql
/usr/bin/mysql
[root@minion1 ~]# mysql -usalt -psalt -h192.168.145.195
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> quit
Bye

## To configure minion ##
[root@minion1 ~]# vim /etc/salt/minion
#return:
#  - mysql
#  - hipchat
#  - slack
 Add the following lines to the configuration file
mysql.host: '192.168.145.195'     mysqlIP
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
[root@minion1 ~]# systemctl restart salt-minion

## stay Master Store test on mysql in ##
[root@master ~]# salt 'minion1' test.ping    
minion:
    True
[root@master ~]# salt 'minion1' test.ping --return mysql
minion:
    True

## Query in database ##
MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
       fun: test.ping
       jid: 20211107120045632879
    return: true
        id: minion1
   success: 1
  full_ret: {"success": true, "return": true, "retcode": 0, "jid": "20211107120045632879", "fun": "test.ping", "fun_args": [], "id": "minion"}
alter_time: 2021-11-07 1:29:02
1 row in set (0.00 sec)

Summarize the return execution process: master sends instructions to minion, then minion executes them, minion reports to both sides after execution

job cache

Minion interacts directly with the storage server when returning, so you need to install a module with the specified storage style on each Minion, such as python-mysql. Can we store the returned results directly on the Master to the storage server?

The answer is yes, which is called job cache. This means that when Minion returns the results to Master, Master caches the results locally and then stores the cached results to a specified storage server, such as mysql

## open master End master_job_cache ##
[root@master ~]# vim /etc/salt/master
#job_cache: True
 Add the following lines to the configuration file
master_job_cache: mysql
mysql.host: '192.168.145.195'
mysql.user: 'salt'
mysql.pass: 'salt'
mysql.db: 'salt'
mysql.port: 3306
[root@master ~]# systemctl restart salt-master

## Empty table contents in database server ##
[root@mysql ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 21
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> use salt;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [salt]> delete from salt_returns;
Query OK, 4 rows affected (0.00 sec)

MariaDB [salt]> select * from salt_returns\G
Empty set (0.00 sec)

## stay master Test again on whether it can be stored in the database ##
[root@master ~]# salt '*' test.ping
master:
    True
minion:
    True
[root@master ~]# salt 'minion1' cmd.run 'df -h'
minion:
    Filesystem           Size  Used Avail Use% Mounted on
    devtmpfs             371M     0  371M   0% /dev
    tmpfs                391M  140K  391M   1% /dev/shm
    tmpfs                391M   16M  375M   4% /run
    tmpfs                391M     0  391M   0% /sys/fs/cgroup
    /dev/mapper/cs-root   66G  2.4G   63G   4% /
    /dev/sda1           1014M  195M  820M  20% /boot
    /dev/mapper/cs-home   32G  260M   32G   1% /home
    tmpfs                 79M     0   79M   0% /run/user/0
[root@master ~]# salt 'minion1' cmd.run 'date'
minion1:
    Sat Nov  6 21:00:03 CST 2021

## Query in database ##
MariaDB [salt]> select * from salt_returns\G
*************************** 1. row ***************************
       fun: test.ping
       jid: 20211107954368795245
    return: true
        id: master
   success: 1
  full_ret: {"cmd": "_return", "id": "master", "success": true, "return": true, "retcode": 0, "jid": "20211107954368795245", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-07T1:37:03.038821"}
alter_time: 2021-11-07 1:37:03
*************************** 2. row ***************************
       fun: test.ping
       jid: 20211106125442844031
    return: true
        id: minion
   success: 1
  full_ret: {"cmd": "_return", "id": "minion", "success": true, "return": true, "retcode": 0, "jid": "20211107954368795245", "fun": "test.ping", "fun_args": [], "_stamp": "2021-11-07T1:37:03.046980"}
alter_time: 2021-11-07 1:37:03
2 rows in set (0.00 sec)

job management

## Getting Tasks jid ##
root@master ~]# salt 'minion' cmd.run 'date' -v
Executing job with jid 20211107456953458735
-------------------------------------------

minion:
    Sat Nov  6 1:39:24 CST 2021

## adopt jid Get the return result of this task ##
[root@master ~]# salt-run jobs.lookup_jid 20211107456953458735
minion:
    Sat Nov  6 1:39:24 CST 2021