Online Disk Migration through ASM Rebalancing
Rebalancing feature of ASM is very helpful whenever you want to change the underlying disks aka storage without taking the database offline
ASM Rebalancing:
It automatically rebalances—redistributes file data evenly across all the disks of the disk group—after you add disks to or drop disks from a disk group. ASM automatically rebalances whenever disks are added or dropped. For a normal drop operation (without the FORCE option), a disk is not released from a disk group until data is moved off of the disk through rebalancing.
Likewise, a newly added disk cannot support its share of the I/O workload until rebalancing completes. It is more efficient to add or drop multiple disks at the same time so that they are rebalanced as a single operation. This avoids unnecessary movement of data.
The Setup:
There are 3 diskgroups DATA, FRA & OCR_VD with one disk each.
SQL> select a.name "Group" ,b.name "Disk",b.path "Path", b.MODE_STATUS "Status" from v$asm_diskgroup a,v$asm_disk b where a.GROUP_NUMBER=b.GROUP_NUMBER; Group Disk Path Status ---------- ------------- ------------------ ------- DATA DATA_0000 /dev/asm-disk2 ONLINE FRA FRA_0000 /dev/asm-disk3 ONLINE OCR_VD OCR_VD_0000 /dev/asm-disk1 ONLINE
We will add new disks to the RAC instances, in DATA diskgroup and wait for the rebalancing to complete. Once the rebalancing is done the old disks can be removed from the diskgroups and another rebalancing will happen. Once completed, old disks can be removed
Available ASM disks at OS level.
Disks are already partitioned at OS level and are ready for addition.
[grid@rac1 ~]$ ls -lrt /dev/asm-disk* brw-rw---- 1 grid oinstall 8, 23 Aug 28 22:30 /dev/asm-disk3 brw-rw---- 1 grid oinstall 8, 22 Aug 28 22:46 /dev/asm-disk2 brw-rw---- 1 grid oinstall 8, 21 Aug 28 22:46 /dev/asm-disk1 brw-rw---- 1 grid oinstall 8, 26 Aug 29 2013 /dev/asm-disk6 brw-rw---- 1 grid oinstall 8, 24 Aug 29 2013 /dev/asm-disk4 brw-rw---- 1 grid oinstall 8, 27 Aug 29 2013 /dev/asm-disk7 brw-rw---- 1 grid oinstall 8, 29 Aug 29 2013 /dev/asm-disk9 brw-rw---- 1 grid oinstall 8, 30 Aug 29 2013 /dev/asm-disk10 brw-rw---- 1 grid oinstall 8, 25 Aug 29 2013 /dev/asm-disk5 brw-rw---- 1 grid oinstall 8, 28 Aug 29 2013 /dev/asm-disk8 brw-rw---- 1 grid oinstall 8, 31 Aug 29 2013 /dev/asm-disk11 SQL> sho parameter asm_diskstring NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ asm_diskstring string /dev/asm-disk*
These new disks are visible in the v$asm_disk view, but check for header status
SQL> select path, header_status, mode_status from v$asm_disk; PATH HEADER_STATU MODE_ST --------------- ------------ ------- /dev/asm-disk11 CANDIDATE ONLINE /dev/asm-disk8 CANDIDATE ONLINE /dev/asm-disk5 CANDIDATE ONLINE /dev/asm-disk10 CANDIDATE ONLINE /dev/asm-disk9 CANDIDATE ONLINE /dev/asm-disk7 CANDIDATE ONLINE /dev/asm-disk6 CANDIDATE ONLINE /dev/asm-disk4 CANDIDATE ONLINE /dev/asm-disk2 MEMBER ONLINE /dev/asm-disk1 MEMBER ONLINE /dev/asm-disk3 MEMBER ONLINE 11 rows selected.
Before adding the disks check for datafile location and observe that the database is online,
SQL> select file_name from dba_data_files; FILE_NAME --------------------------------------------- +DATA/racdb/datafile/undotbs2.268.823906247 +DATA/racdb/datafile/system.260.823906197 +DATA/racdb/datafile/sysaux.265.823906229 +DATA/racdb/datafile/undotbs1.264.823906223 +DATA/racdb/datafile/users.267.823906245 SQL> select name from v$controlfile; NAME ------------------------------------------- +DATA/racdb/datafile/control01.ctl +DATA/racdb/datafile/control02.ctl SQL> select member from v$logfile; MEMBER --------------------------------------------- +DATA/racdb/onlinelog/group_1.259.823905645 +DATA/racdb/onlinelog/group_2.263.823905355 +DATA/racdb/onlinelog/group_3.262.823905373 +DATA/racdb/onlinelog/group_4.261.823905379
Addition of new disk in DATA diskgroup
From ASM instance,
SQL> alter diskgroup data add disk '/dev/asm-disk4','/dev/asm-disk5' rebalance power 5; Diskgroup altered.
This will immediately start a rebalancing operation that we can view through the v$asm_operation view.
SQL> select group_number, operation, est_minutes from v$asm_operation; GROUP_NUMBER OPERA EST_MINUTES ------------ ----- ----------- 1 REBAL 4
Once completed the operation will disappear
SQL> select group_number, operation, est_minutes from v$asm_operation; no rows selected
Check for the addition of the disks:
SQL> select a.name "Group" ,b.name "Disk",b.path "Path", b.MODE_STATUS "Status" from v$asm_diskgroup a,v$asm_disk b where a.GROUP_NUMBER=b.GROUP_NUMBER and a.name='DATA'; Group Disk Path Status ----- ---------- -------------------- ------- DATA DATA_0000 /dev/asm-disk2 ONLINE DATA DATA_0002 /dev/asm-disk5 ONLINE DATA DATA_0001 /dev/asm-disk4 ONLINE
Drop the old disks now
It will automatically rebalancing where ASM relocates all the data from the disks being dropped. When the data is moved away the disk header is cleared and the disk will no longer part of the diskgroup.
SQL> alter diskgroup data drop disk DATA_0001,DATA_0002 rebalance power 4; Diskgroup altered. SQL> select group_number, operation, est_minutes from v$asm_operation; GROUP_NUMBER OPERA EST_MINUTES ------------ ----- ----------- 1 REBAL 3 SQL> select group_number, operation, est_minutes from v$asm_operation; no rows selected
If you will query v$asm_disk will show that asm1 and asm2 disks as FORMER, means they were cleanly dropped and it is now safe to remove them from the storage.
SQL> select path, header_status, mode_status from v$asm_disk; PATH HEADER_STATU MODE_ST -------------------- ------------ ------- /dev/asm-disk5 FORMER ONLINE /dev/asm-disk4 FORMER ONLINE /dev/asm-disk11 CANDIDATE ONLINE /dev/asm-disk8 CANDIDATE ONLINE /dev/asm-disk10 CANDIDATE ONLINE /dev/asm-disk9 CANDIDATE ONLINE /dev/asm-disk6 CANDIDATE ONLINE /dev/asm-disk7 CANDIDATE ONLINE /dev/asm-disk2 MEMBER ONLINE /dev/asm-disk1 MEMBER ONLINE /dev/asm-disk3 MEMBER ONLINE
ASM disk names, group membership and status after dropping
SQL> select a.name "Group" ,b.name "Disk",b.path "Path", b.MODE_STATUS "Status" from v$asm_diskgroup a,v$asm_disk b where a.GROUP_NUMBER=b.GROUP_NUMBER and a.name='DATA'; Group Disk Path Status ------------------------------ ---------------- -------------------- ------- DATA DATA_0000 /dev/asm-disk2 ONLINE
Note: On RAC environment, “ASM can perform one disk group rebalance at a time on a given instance. Therefore, if you have initiated multiple rebalances on different disk groups, then Oracle processes this operation serially. However, you can initiate rebalances on different disk groups on different nodes in parallel”