preface
This article is connected to the DSC active standby and standby stand-alone used by Dameng DSC shared cluster
Dameng DSC cluster https://blog.csdn.net/qing1122334455/article/details/117935136
1, Planning
There are DSC clusters ahead
A single machine is prepared here as the standby library
Example: pandas is a NumPy based tool created to solve data analysis tasks.
2, Relevant settings of system parameters
1. Turn off the firewall
The code is as follows:
systemctl status firewalld.service systemctl stop firewalld.service systemctl disable firewalld.service
2. Close selinux
vi /etc/selinux/config
SELINUX=disabled SELINUXTYPE=targeted
3. Create installation users and groups
The code is as follows:
groupadd -g 10001 dinstall useradd -u 20001 -g dinstall dmdba passwd dmdba Set password dmdba
4. Modify the operating system resource limit
vim /etc/security/limits.conf:
dmdba soft noproc 65536 dmdba hard noproc 65536 dmdba soft nofile 65536 dmdba hard nofile 65536 dmdba hard core unlimited dmdba soft core unlimited
Modify the parameter VI / etc / security / limits d/20-nproc. conf:
* soft nproc 65536
5. Modify kernel parameters
vi /etc/sysctl.conf
fs.aio-max-nr = 1048576 fs.file-max=6815744 net.ipv4.ip_local_port_range = 1024 65535 net.core.rmem_default = 262144 net.core.rmem_max= 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048586 vm.overcommit_memory=0
Make kernel parameters effective:
sysctl -p
7. Create data file storage directory
The code is as follows:
mkdir -p /dm/dmdba/dmdbms mkdir -p /dm/dmdata chown -R dmdba:dinstall /dm chmod -R 775 /dm
8. Modify user dmdba environment variable
su - dmdba
vi ~/.bash_profile
export LANG=zh_CN.UTF8 export DM_INSTALL_TMPDIR=/tmp export DM_HOME=/dm/dmdba/dmdbms export PATH=$DM_HOME/bin:$PATH:$HOME/bin
3, Install database software
1. Upload the database software to the operating system
Create software directory and upload
mkdir -p /dm/soft Upload software to this directory mount [root@dmdb01 ~]# mount /dm/soft/dm8_setup_rh7_64_ent_8.1.1.48_20191129.iso /mnt/
2. Install database software (standby database)
Enter user dmdba
su - dmdba
cd /mnt
Execute installation command
[dmdba@dmrw2 mnt]$ ./DMInstall.bin -i #Add the parameter - i after the installation in the command line interface
give the result as follows
Please select the installation language(C/c:chinese E/e:english) [C/c]:c Unzip the installer......... Welcome to Damon database setup Enter Key File path? (Y/y:yes N/n:no) [Y/y]:n Set time zone? (Y/y:yes N/n:no) [Y/y]:y Set time zone: [ 1]: GTM-12=West of the sun line [ 2]: GTM-11=Samoa Islands [ 3]: GTM-10=Hawaii [ 4]: GTM-09=Alaska [ 5]: GTM-08=Pacific time (US and Canada) [ 6]: GTM-07=Arizona [ 7]: GTM-06=Central time (US and Canada) [ 8]: GTM-05=Eastern time (USA and Canada) [ 9]: GTM-04=Atlantic time (US and Canada) Please select and set the time zone [21]: Installation type: 1 Typical installation 2 The server 3 client 4 custom Please select the digital serial number of the installation type [1 Typical installation]:1 space required: 1012M Please select the installation directory [/home/dmdba/dmdbms]:/dm/dmdba/dmdbms #Enter the planned software installation directory here Available space: 67G Confirm the installation path(/dm/dmdba/dmdbms)? (Y/y:yes N/n:no) [Y/y]:y Summary before installation Installation position: /dm/dmdba/dmdbms space required: 1012M Available space: 67G Version information: Effective date: End of installation installation type: Typical installation Confirm installation? (Y/y:yes N/n:no):y 2021-06-17 14:03:05 [INFO] Install Damon database... .......... [INFO] Installation of Damon database is complete. Please take root System users execute commands: /dm/dmdba/dmdbms/script/root/root_installer.sh
Switch to root
[root@dmdb01 dmdbms]# /dm/dmdba/dmdbms/script/root/root_installer.sh ##Create and start the DmAPService service
3. Initialize the master database instance
Software installation directory: / dm/dmdba/dmdbms, instance initialization Directory: / dm/dmdata /. The initialization script is as follows:
cd /dm/dmdba/dmdbms/bin
./dminit path=/dm/dmdata/ instance_name=dscstd page_size=32
4.DSC main library offline backup
Close the window that starts the instance in the background
When the dmdscdmddsc cluster generates a attributed cluster, a attributed cluster generates an archived log, an offline backup is performed for subsequent connectivity verification.
After closing the database, use the dmrman tool to back up the main database offline. The dmdba user goes to the bin of the installation directory and executes the following commands:
The code is as follows (example):
./dmrman use_ap=2 dcr_ini=/dm/dmdba/config/dmdcr.ini
Execute backup full library.
backup database '/dm/dmdba/config/dsc1/dm.ini' backupset '/dm/dmbak/bakfull';
5. Backup and restore (executed by the standby database)
Copy the primary database to the appropriate directory of the secondary database, and execute the following command
scp -r /dm/dmbak/bakfull 192.168.16.151:/dm/dmbak/
After closing the database, use the dmrman tool to restore the standby database. The dmdba user goes to the bin of the installation directory and executes the following commands:
./dmrman
Execute restore.
restore database '/dm/dmdata/DAMENG/dm.ini' from backupset '/dm/dmbak/bakfull';
Perform recover.
recover database '/dm/dmdata/DAMENG/dm.ini' from backupset '/dm/dmbak/bakfull';
Execute recover update db_magic.
recover database '/dm/dmdata/DAMENG/dm.ini' update db_magic;
The implementation results are as follows
recover database '/dm/dmdba/dmdbms/dmrw/dm.ini' update db_magic; Database mode = 0, oguid = 0 EP[0]'s cur_lsn[37904] EP[0]'s apply_lsn[37904] >= end_lsn[37904] recover successfully! time used: 984.325(ms)
IV Configure DSC parameters (dmdba user)
1. Modify the configuration file DM of the two instances ini
Modify the instance's DM INI file parameter, execute the following command:
vi /dm/dmdba/dmdbms/dmrw/dm.ini
The main library modifies the following parameter values:
ALTER_MODE_STATUS = 0 #Manual modification of instance mode / status is not allowed ENABLE_OFFLINE_TS = 2 #OFFLINE tablespace of standby database is not allowed ARCH_INI = 1 #Enable Archive Mode MAL_INI = 1
2. Modify the archive configuration file dmarch ini
Create a new file dmarch. In the instance directory Ini, execute the following command:
vi /dm/dmdba/dmdbms/dmrw/dmarch.ini
Add the following content to the main library:
[ARCHIVE_REALTIME] ARCH_TYPE = REALTIME #Real time archive type ARCH_DEST = dmrw2 #Real time archiving target instance name (fill in the standby instance name on the main database side) [ARCHIVE_LOCAL1] ARCH_TYPE = LOCAL #Local archive type ARCH_DEST = /dm/dmarch #Local archive file storage path ARCH_FILE_SIZE = 128 #Unit: MB, maximum value of local single archive file ARCH_SPACE_LIMIT = 10240 #Unit: MB, 0 means unlimited, range: 1024~4294967294 MB
Add the following contents to the standby database:
[ARCHIVE_REALTIME] ARCH_TYPE = REALTIME #Real time archive type ARCH_DEST = dmrw1 #Real time archiving target instance name (fill in the main database instance name on the standby database side) [ARCHIVE_LOCAL1] ARCH_TYPE = LOCAL #Local archive type ARCH_DEST = /dm/dmarch #Local archive file storage path ARCH_FILE_SIZE = 128 #Unit: MB, maximum value of local single archive file ARCH_SPACE_LIMIT = 10240 #Unit: MB, 0 means unlimited, range: 1024~4294967294 MB
3. Modify MAL system configuration file dmmal ini
Create a new file dmmal. In the instance directory Ini, execute the following command:
vi /dm/dmdba/dmdbms/dmrw/dmmal.ini
The contents of the primary and standby database files should be the same.
MAL_CHECK_INTERVAL = 5 #MAL link detection interval MAL_CONN_FAIL_INTERVAL = 5 #Determine the disconnection time of MAL link [MAL_INST1] MAL_INST_NAME = dmrw1 #And DM Instance in ini_ Name consistent MAL_HOST = 10.10.10.135 #MAL system monitors TCP internal network IP MAL_PORT = 61141 #MAL system listens to the port of TCP connection MAL_INST_HOST = 192.168.16.135 #External service IP address of the instance MAL_INST_PORT = 5236 #And DM Port in ini_ Num consistent MAL_DW_PORT = 52141 #The daemon corresponding to the instance listens to the TCP port MAL_INST_DW_PORT = 33141 [MAL_INST2] MAL_INST_NAME = dmrw2 #And DM Instance in ini_ Name consistent MAL_HOST = 10.10.10.136 # MAL system monitors TCP internal network IP MAL_PORT = 61141 #MAL system listens to the port of TCP connection MAL_INST_HOST = 192.168.16.136 #External service IP address of the instance MAL_INST_PORT = 5236 #And DM Port in ini_ Num consistent MAL_DW_PORT = 52141 #The daemon corresponding to the instance listens to the TCP port MAL_INST_DW_PORT = 33141
4. Modify the daemon configuration file dmwatcher ini
Create a new file dmwatcher.com in the instance directory Ini, execute the following command:
vi /dm/dmdba/dmdbms/dmrw/dmwatcher.ini
The contents of the primary and standby database files should be the same.
[GRP_RW] DW_TYPE = GLOBAL #Global guard type DW_MODE = AUTO #Automatic switching mode DW_ERROR_TIME = 10 #Remote daemon failure determination time INST_RECOVER_TIME = 60 #The interval between the main library daemon starting recovery INST_ERROR_TIME = 10 #Local instance fault determination time INST_OGUID = 453331 #Unique OGUID value of daemon INST_INI = /dm/dmdba/dmdbms/dmrw/dm.ini #dm.ini configuration file path INST_AUTO_RESTART = 1 #Turn on the auto start function of the instance INST_STARTUP_CMD = /dm/dmdba/dmdbms/bin/dmserver #Command line startup
5. Start the database instance in Mount mode
Use the dmdba user to execute in the bin of the database installation directory (both active and standby databases).
./dmserver /dm/dmdba/dmdbms/dmrw/dm.ini mount
Use the disql tool to connect to the database on the new terminal and execute the following command:
./disql SYSDBA/SYSDBA
Modify oguid for both active and standby databases, and execute the following commands:
sp_set_oguid(453331);
Modify the database mode of the master database to primary, and execute the following commands:
alter database primary;
Modify the standby database mode to standby and execute the following commands:
alter database standby;
6. Start the daemon
Under the dmdba user, execute in the bin of the database installation directory (both active and standby databases).
./dmwatcher /dm/dmdba/dmdbms/dmrw/dmwatcher.ini
After the daemon starts, the instance of Mount will be opened.
5, Start confirmation monitor
When the daemon is configured to automatically switch, the confirmation monitor must be configured. On servers other than the active and standby servers (DM database software shall be installed and open to the active and standby heartbeat network port).
Create a new confirmation monitor configuration file dmmonitor Ini, execute the following command:
vi /dm/dmdba/dmdbms/dmmonitor.ini
Add the following:
MON_DW_CONFIRM = 1 #Confirm monitor mode MON_LOG_PATH = /home/dmdba/dmdbms/log #Storage path of monitor log file MON_LOG_INTERVAL = 60 #Regularly record the system information to the log file every 60 s MON_LOG_FILE_SIZE = 32 #Maximum 32 MB per log file MON_LOG_SPACE_LIMIT = 0 #Unlimited total log file space [GRP_RW] MON_INST_OGUID = 453331 #Group GRP_ Unique OGUID value of RW #The following configuration is monitor to group GRP_ The connection information of RW daemon is configured in the form of "IP:PORT" #IP corresponds to dmmal Mal in ini_ Host and port correspond to dmmal Mal in ini_ DW_ PORT MON_DW_IP = 10.10.10.135:52141 MON_DW_IP = 10.10.10.136:52141
Start the monitor and execute the following command:
./dmmonitor /dm/dmdba/dmdbms/dmmonitor.ini
After startup, enter the show command to view the cluster status.
show 2021-06-17 19:26:10 #================================================================================# GROUP OGUID MON_CONFIRM MODE MPP_FLAG GRP_RW 453331 TRUE AUTO FALSE <<DATABASE GLOBAL INFO:>> IP MAL_DW_PORT WTIME WTYPE WCTLSTAT WSTATUS INAME INST_OK N_EP N_OK ISTATUS IMODE DSC_STATUS RTYPE RSTAT 10.10.10.135 52141 2021-06-17 19:26:10 GLOBAL VALID OPEN DMRW1 OK 1 1 OPEN PRIMARY DSC_OPEN REALTIME VALID EP INFO: INST_PORT INST_OK INAME ISTATUS IMODE DSC_SEQNO DSC_CTL_NODE RTYPE RSTAT FSEQ FLSN CSEQ CLSN DW_STAT_FLAG 5236 OK DMRW1 OPEN PRIMARY 0 0 REALTIME VALID 3824 42630 3824 42630 NONE <<DATABASE GLOBAL INFO:>> IP MAL_DW_PORT WTIME WTYPE WCTLSTAT WSTATUS INAME INST_OK N_EP N_OK ISTATUS IMODE DSC_STATUS RTYPE RSTAT 10.10.10.136 52141 2021-06-17 19:26:10 GLOBAL VALID OPEN DMRW2 OK 1 1 OPEN STANDBY DSC_OPEN REALTIME VALID EP INFO: INST_PORT INST_OK INAME ISTATUS IMODE DSC_SEQNO DSC_CTL_NODE RTYPE RSTAT FSEQ FLSN CSEQ CLSN DW_STAT_FLAG 5236 OK DMRW2 OPEN STANDBY 0 0 REALTIME VALID 3774 42630 3774 42630 NONE DATABASE(DMRW2) APPLY INFO FROM (DMRW1): DSC_SEQNO[0], (ASEQ, SSEQ, KSEQ)[3824, 3824, 3824], (ALSN, SLSN, KLSN)[42630, 42630, 42630], N_TSK[0], TSK_MEM_USE[0] #================================================================================#
6, Registration service
The above startup is in the foreground mode, and only the configuration is verified during the user's construction. After the configuration is OK, you need to register the instance, daemon and confirmation monitor as system services.
Using the root user, go to script/root in the database installation directory and execute.
Register the daemon service (both active and standby databases are executed).
./dm_service_installer.sh -t dmwatcher -p dmrw -watcher_ini /dm/dmdba/dmdbms/dmrw/dmwatcher.ini
Register the database instance service (both active and standby databases are executed).
./dm_service_installer.sh -t dmserver -p dmrw -dm_ini /dm/dmdba/dmdbms/dmrw/dm.ini
Register the monitor service (only on the monitor server), and execute the following command:
./dm_service_installer.sh -t dmmonitor -p confirm -monitor_ini /dm/dmdba/dmdbms/dmmonitor.ini
1. Start in service mode
Start the active and standby database instances and execute the following commands:
systemctl start DmServicedmrw
The active and standby start the daemon and execute the following commands:
systemctl start DmWatcherServicedmrw
The monitoring server (outside the active and standby) starts the monitor and executes the following commands:
systemctl start DmMonitorServiceconfirm
7, disql client authentication
Log in to the main database using the disql client, create a test table, insert data, and execute the following commands:
./disql SYSDBA/SYSDBA@192.168.16.135:5236
Execute the following command from the SQL prompt:
create table test(id int); insert into test values (1); commit;
Log in to the standby database using the disql client, query the test table for verification, and execute the following commands:
./disql SYSDBA/SYSDBA@192.168.56.12:5236
Execute the following command from the SQL prompt:
select * from test;
The server[LOCALHOST:5236]:The standby database is open Login usage time: 6.908(millisecond) disql V8 SQL> select * from test; Line number ID ---------- ----------- 1 1 Elapsed time: 9.111(millisecond). Execution number:2.
1. Import and storage
The code is as follows (example):
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.filterwarnings('ignore')
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
2. Read in data
The code is as follows (example):
data = pd.read_csv(
'https://labfile.oss.aliyuncs.com/courses/1283/adult.data.csv')
print(data.head())
The url used here is the data requested by the network.
summary
Tip: here is a summary of the article:
For example, the above is what we want to talk about today. This paper only briefly introduces the use of pandas, which provides a large number of functions and methods that enable us to process data quickly and conveniently.