Background description

System a is a dual node RAC. System a is short of disk space, so apply to Party A and add 1T disk to system a;
System B is a single instance. System B has the remaining 500G disk and 800G disk DATA disk space. The archive storage location is in the DARA disk. The archive space size is set to 800G. Apply to Party A. system B adds 1T disk and FRA disk, changes the archive storage location to FRA, expands the original 500G disk to the DATA disk, and expands the new 1T disk to the FRA disk

Disk expansion

Capacity expansion ideas

Scan new disks - create ASM according to udev rules - restart udev - expand disks - view balance progress - expansion complete

Capacity expansion steps

The new disk number of system A is AAAA, node 1: dm-42, node 2: dm-47
The newly added disk numbers in system B are BBB and dm-14
The original disk numbers of system B are BB1 and dm-5

Scan new disk

multipath -l -- check whether the new disk can be displayed, and check whether the wwn number of the new disk is consistent with that given by Party A
If no new disks are found, the SCSI bus needs to be rescanned to add new devices

echo "- - -" > /sys/class/scsi_host/host0/scan
echo "- - -" > /sys/class/scsi_host/host1/scan
echo "- - -" > /sys/class/scsi_host/host2/scan

Note: / sys / class / SCSI_ Several host files under the host / file need to be scanned several times
A system node 1:

/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-42

Node 2 of system A:

/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-47

B system:

/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-14

B system:

/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/dm-5

Note: obtain the wwn number of the new disk according to udev rules
multipath -l -- check whether the new disk can be displayed again and compare the wwn number
Note: scanning the new disk needs to be performed on both nodes of system A to ensure that both nodes can see the disk. The disk number scanned by dm-42 of node 1 and dm47 of node 2 of system A should be the same, which needs to be performed by root user.

Create ASM

According to udev rules, add the following content to the 99-asmdisk-udev.rules file:

A System node 1:
vi /etv/udev/rules.d/99-asmdisk.rules
KERNEL =="dm-*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="aaaaaaaaaa", NAME="asm_data23", OWNER="grid", GROUP="asmadmin", MODE="0660"

A System 2 node:
vi /etv/udev/rules.d/99-asmdisk.rules
KERNEL =="dm-*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="aaaaaaaaaa", NAME="asm_data23", OWNER="grid", GROUP="asmadmin", MODE="0660"

#B system:
vi /etv/udev/rules.d/99-asmdisk.rules
KERNEL =="dm-*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="bbbbbbbbbb", NAME="asm_fra01", OWNER="grid", GROUP="asmadmin", MODE="0660"       -—by B System increase FRA
B System: vi /etv/udev/rules.d/99-asmdisk.rules
KERNEL =="dm-*", SUBSYSTEM=="block", PROGRAM=="/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name", RESULT=="bbbbbbbbb1", NAME="asm_data07", OWNER="grid", GROUP="asmadmin", MODE="0660" 

Note: Please note that the wwn number of the disk and the name of the disk to be created must be filled in correctly by the root user.

Restart UDEV

Restart udev for the previous changes to take effect.
/sbin/start_udev
Note: each node needs to restart udev to check whether there is an error. If there is an error, it should be solved in time.
ls -al /dev/asm_*
Check whether the newly added ASM disk is recognized.
Note: it needs to be executed by the root user.

Disk expansion

Under grdi user, first check the current status of the disk, determine whether the new disk can be seen in the disk group, and check the current disk capacity

sqlplus / as sysasm
set lines 500;
col path for a40;
select name,path,total_mb,free_mb,state,header_status from v$asm_disk;
alter diskgroup DATA add disk '/dev/asm_data23' rebalance 4;

Note: the disk name in alter is the same as that created before. 4 means parallel. When the database is busy, the parallel should not be too large. After capacity expansion, the resources of each disk will be balanced, which will take some time

View balance progress

select * from v$asm_operation;

Note: always pay attention to the balance progress. After the balance, no information will be found in this view.

Add FRA

Add ideas

Scan new disks – create ASM according to udev rules – restart udev – create disks – modify archiving policy

Add step

In the previous operation, the disks of FRA have been successfully scanned, that is, the steps of "scan new disks - create ASM according to udev rules - restart udev" are the same as before. For convenience, the disks required by FRA have been successfully expanded in the previous disk expansion. Therefore, start from creating disks here.

create disk

Under grid user, check the status of each disk.
In the asmcmd window, execute lsdg and lsct to view the disk group information.

sqlplus / as sysasm
set lines 500;
col path for a40;
select name,path,total_mb,free_mb,state,header_status from v$asm_disk;
create disk group FRA external redundancy disk '/dev/asm_fra01' attribute 'compatible.asm'='11.2','compatible.rdbms'='11.2';

Note: several important asm diskgroup parameters:
1, ACCESS_CONTROL.ENABLED this attribute is used to control ASM FILE Access Control on a disk group. This parameter has two values: true and false. The default is false. When this parameter is set to true. Then you must control the access rights to access ASM FILES on the disk group. If false, any user can access files on disk group.

--Modify command:
ALTER DISKGROUP XXX SET ATTRIBUTE 'access_control.enabled' = 'true';  

2. COMPATIBLE.RDBMS this attribute is used to specify the minimum COMPATIBLE database instance of disk group. Before modifying this parameter, you must first ensure that the COMPATIBLE parameter of all database instances is greater than or equal to the value of this parameter.
In Oracle 11g database, the COMPATIBLE.RDBMS attribute value of Oracle ASM disk group is 10.1 by default

--View command: 
 SQL> set line 500;
SQL> col path for a30  ;
SQL> col value for a30  ;
SQL> select name,value from v$asm_attribute where group_number=1 and name like 'compatible%';  
--Modify command:  
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.rdbms' = '11.2.0.0.0';  
    Diskgroup altered.  

3, ACCESS_CONTROL.UMASK this attribute is used to specify the permission to access ASM files. It can only be modified with the alert command. The parameter value contains three numbers: {0 | 2 | 6} {0 | 2 | 6} {0 | 2 | 6}. The default is 066. 0: indicates that you have read-write permission. 2: indicates that you have read-only permission. 6: indicates that you cannot read and write
Modify command: set disk group access_ Before the control.umask property, you must ensure access_ The control.enabled parameter is set to TRUE. This parameter cannot be set until file access control is enabled.

SQL> alter diskgroup fra set attribute 'access_control.enabled' = 'true';  Diskgroup altered.  SQL> alter diskgroup fra set attribute 'access_control.umask' = 'XXX'; 
(If the setting is wrong alert (error will be reported)

4. COMPATIBLE.ASM if you use the CREATE DISKGROUP command, the mkdg command of ASMCMD, or the disk group created by OEM, the default value of the parameter is 10.1. If you use ASMCA to create, the default value is 11.2. This parameter setting will affect the data structure information of ASM metadata on the disk. The COMPATIBLE.ASM attribute determines the version of the smallest Oracle ASM instance software that can use the corresponding disk group.

--View parameters:
  SQL> select name,value from v$asm_attribute where group_number=1 and name like 'compatible%';  

5, AU_SIZE AU_ The value of size can be in the following units: 1, 2, 4, 8, 16, 32, and 64 MB. This parameter can only be specified when creating a disk group. Once specified, it cannot be modified. In oracle 10g, for large databases, it is also recommended to increase AU size because Au is increased_ Size can increase the limit on the size of files. In Oracle 11g, although the default value is 1M, Oracle recommends setting this value to 4M.

--see AU_SIZE size. 
 SQL> select name, allocation_unit_size/1024/1024||'M' from v$asm_diskgroup;  

6, DISK_REPAIR_TIME before Oracle Database 11g, if the ASM disk is damaged, it is a disaster, even if your ASM disk group is protected by Normal Redundancy or High Redundancy. Because in Oracle 10g, the damaged ASM disk will be offline immediately, and the damaged disk will be dropped by Oracle immediately.
To avoid this problem, Oracle introduced a parameter disk in 11g_ repair_ Time, the default value is 3.6 hours and can be modified by the user. The unit of this parameter can be minute (m or M) or hour (h or H). If we do not specify a unit when specifying the attribute, the default unit is hour. This property can only be modified by the ALTER command. When a damaged disk is offline, Oracle will not drop it immediately, but will wait for the parameter disk_ repair_ The time represented by time.
During this period, Oracle will record the modifications made to the EXT on the damaged disk. Once the offline damaged disk is on the disk_ repair_ If you go online again within the time indicated by time, Oracle will resynchronize the previously recorded modifications to the extent on this disk to efficiently synchronize the data (because only the data on the extent modified after offline is synchronized here), avoiding the extremely time-consuming rebalance process of reconstructing all data. This process is the one in 11g: for fast mirror sync, the attribute value of COMPATIBLE.ASM of the disk group must be set greater than 11.1.
When the disk fails and is offline, we can check the ASM_ Repair of disk view_ TIMER. This column displays the remaining time before the disk is dropped, in seconds. Once disk is exceeded_ repair_ Time, the disk will be dropped. Then we need to rebalance instead of fast image synchronization.

--see drop Time remaining before:  
SQL> col name for a15  SQL>select disk_number,name,path,MODE_STATUS,REPAIR_TIMER from v$asm_disk where group_number=X;  

7, SECTOR_SIZE SECTOR_SIZE can only be specified when creating a diskgroup. This property can be set to 512, 4096, or 4K. The specific default values are determined by different platforms. It can only be set when creating a disk group. Generally, it cannot be modified after the creation. The traditional disk has 512 bytes per sector. With the continuous increase of disk capacity, it is no longer so reasonable to continue to use 512 bytes. Therefore, 512 bytes per sector are changed to 4096 bytes per sector, which is now commonly known as "4K sector".

--see SECTOR_SIZE:  
 SQL> SELECT name, value FROM V$ASM_ATTRIBUTE WHERE name = 'sector_size' AND group_number = 1;  
How to modify: not all disks support SECTOR_SIZE If you want to set this property, you must first ensure the compatibility of the physical disk. If you want to modify sector size Value, then you must ensure that disk group of COMPATIBLE.ASM and COMPATIBLE.RDBMS The attribute value is greater than or equal to 11.2.   

Note: Oracle Automatic Storage Management Cluster File System (Oracle ACFS) does not support 4 KB sector drives
Note: after the disk group is created, you should check again to see whether the creation is successful.

Modify archiving policy

To modify the archive policy, you need to modify the archive storage path and decide how to modify the policy according to the actual situation. In this event, check the archive policy and spfile file of system B and find that the archive storage path pointed to by the archive policy is in + DATA. Therefore, you only need to modify the archive policy to point to the new FRA.

sqlplus / as sysdba
create pfile='/home/oracle/spfilebak.ora' from spfile;
select * from v$parameter where name = 'db_recovery_file_dest';
select * from v$parameter where name = 'db_recovery_file_dest_size';
alter system set db_recovery_file_dest='+FRA' scope=both;
alter system set db_recovery_file_dest_size='1000G' scope=both;

Note: the created pfile file can view the relevant policies of the current database and decide how to modify this operation next, v p a r a m e t e r this individual regard chart ， yes check see this individual ginseng number yes move state ginseng number still yes quiet state ginseng number ， quiet state ginseng number need want heavy Start number according to library ， move state ginseng number no need want ， v The parameter view is used to check whether the parameter is a dynamic parameter or a static parameter. The static parameter needs to restart the database, but the dynamic parameter does not. v The parameter view is used to check whether the parameter is a dynamic parameter or a static parameter. The static parameter needs to restart the database, but the dynamic parameter does not. The issys in vparameter_ In the modifiable column, IMMEDIATE represents a dynamic parameter, false represents a static parameter, DEFERRED represents a delay parameter, and the next session connection is valid. Through viewing, they are all dynamic parameters, so they can be modified online without restarting the database. scope=both represents the modification in memory and takes effect immediately for the current instance. Since the previous archive size was set to 800G, after this expansion, the FRA disk has 1T, so the archive storage size will be modified to 1T.
After modification, switch the archive to test whether the newly generated archive can be normally stored in the new path.
alter system switch logfile;
Check in FRA for new archive log generation.
After a new archive log is generated, delete the old archive and clean up the expired archive!
Note: hard disk expansion is a high-risk operation. Be careful! Be careful! Be careful!