udev | StartupForce

Configuring ASM disks using UDEV

August 18, 2013 Bharat Leave a comment

Why are we using UDEV if ASMLib is available for working with ASM disks?

“For RHEL6, Oracle will only provide ASMLib software and updates when configured with a kernel distributed by Oracle. Oracle will not provide ASMLib packages for kernels distributed by Red Hat as part of RHEL6. ASMLib updates will be delivered via Unbreakable Linux Network(ULN) which is available to customers with Oracle Linux support. ULN works with both Oracle Linux or Red Hat Linux installations, but ASMlib usage will require replacing any Red Hat kernel with a kernel provided by Oracle.”

–> UDEV rules files will be available in /etc/udev/rules.d directory

UDEV Filename Rules:

• Filenames begin with two decimal digits, to sequence the rules into ascending numerical order.
• After a dash, filenames contain descriptive words.
• The filename must end with “.rules” to be recognized.
• Rule clauses using a double equal sign (==) are predicate matching patterns, to determine if the rest of the line should be processed.
• Rule clauses are separated by a comma.
• Rule clauses with only a single equal sign (=) are actions to be performed.
• Content after a sharp sign / hash mark / octothorpe (#) is ignored as a comment.
• Blank lines are OK.

Caution About Changing Files Owned By The Distribution

Do not change the default 50-udev-default.rules file. This file will be over-written should the UDEV package be updated.

The Setup:

1. I have created a virtual storage of 10GB in virtual box and attached it as shareable device on both of my rac nodes (rac1 & rac2). OS is OEL 5.5

VBoxManage createhd –filename asm.vdi –size 10240 –format VDI –variant Fixed
VBoxManage storageattach rac1 –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm.vdi –mtype shareable
VBoxManage storageattach rac2 –storagectl “SATA” –port 1 –device 0 –type hdd –medium asm.vdi –mtype shareable
VBoxManage modifyhd asm.vdi –type shareable

2. Created 5 partitions of 1GB each on the attached storage using fdisk

[root@rac1 ~]# fdisk -l

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 1305 10482381 5 Extended
/dev/sdb5 1 125 1003999+ 83 Linux
/dev/sdb6 126 250 1004031 83 Linux
/dev/sdb7 251 375 1004031 83 Linux
/dev/sdb8 376 500 1004031 83 Linux
/dev/sdb9 501 625 1004031 83 Linux

Setting up UDEV rules

1. Add the following to the “/etc/scsi_id.config” file to configure SCSI devices as trusted. Create the file if it doesn’t already exist.

options=-g

2. Identify the unique SCSI id from scsi_id command

If linux 5,

[root@rac1 install]# /sbin/scsi_id -g -u -s /block/sdb
SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_

If linux 6,

[root@rac1 install]#/sbin/scsi_id -g -u -d /dev/sdb
SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_

3. Now, setup rules file : vi /etc/udev/rules.d/99-oracle-asmdevices.rules

Add a rule for the above device. A rule is essentially an if-then-else logic. In the rule, we specify is satisfied for an event, actions will be taken to setup the device.

If the event is a scsi device (KERNEL, BUS attributes), then call the /sbin/scsi_id -g -u -s program, passing the block device as first argument (PROGRAM attribute and %p in the rule definition). If the RESULT of the program call matches with a value of 3600143801259abde00006000004c0000 (RESULT attribute in the rule), then create a device entry as “asmcrs01″, with owner as grid, group owner as oinstall, and permissions as 0660.

Syntax:
KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”, RESULT==”<SCSI ID>″, NAME=”<Specify ASM disk name>″,OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″

3.1. Rule Prior Up-To Version 11.1.0.7

# /bin/cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”, RESULT==” SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=” asm-disk1″, ACTION==”add|change”,OWNER=”oracle”, GROUP=”oinstall”, MODE=”0660″

3.2 Rule For Version 11.2 And Later

For RHEL5/OL5/OEL5,

# /bin/cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL==”sd*”, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s %p”, RESULT==” SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=” asm-disk1″, ACTION==”add|change”, OWNER=”grid”,GROUP=”asmadmin”, MODE=”0660″

For RHEL6/OL6:

KERNEL==”sd?1″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -d /dev/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB7437a3b7-95b199cd_”, NAME=”asm-disk1″, OWNER=”oracle”, GROUP=”dba”, MODE=”0660″

KERNEL==”sd*”, BUS==”scsi”,PROGRAM==”/sbin/scsi_id –whitelisted –device=/dev/$name”,, RESULT==” SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=” asm-disk1″, ACTION==”add|change”, OWNER=”grid”,GROUP=”asmadmin”, MODE=”0660″

Rules file in my setup

[root@rac1 rules.d]# cat 99-oracle-asmdevices.rules

KERNEL==”sdb5″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s /block/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=”asm-disk1″, OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″
KERNEL==”sdb6″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s /block/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=”asm-disk2″, OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″
KERNEL==”sdb7″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s /block/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=”asm-disk3″, OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″
KERNEL==”sdb8″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s /block/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=”asm-disk4″, OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″
KERNEL==”sdb9″, BUS==”scsi”, PROGRAM==”/sbin/scsi_id -g -u -s /block/$parent”, RESULT==”SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_”, NAME=”asm-disk5″, OWNER=”grid”, GROUP=”oinstall”, MODE=”0660″

4. Load Updated Block Device Partitions (/sbin/partprobe)

# /sbin/partprobe /dev/sdb5
# /sbin/partprobe /dev/sdb6
# /sbin/partprobe /dev/sdb7
# /sbin/partprobe /dev/sdb8
# /sbin/partprobe /dev/sdb9

5. Test the rules using udevtest

# For OEL5/RHEL5

# udevtest /block/sdb/sdb5
# udevtest /block/sdb/sdb6
# udevtest /block/sdb/sdb7
# udevtest /block/sdb/sdb8
# udevtest /block/sdb/sdb9

# For OEL6/RHEL6

# udevadm test /block/sdb/sdb5
# udevadm test /block/sdb/sdb6
# udevadm test /block/sdb/sdb7
# udevadm test /block/sdb/sdb8
# udevadm test /block/sdb/sdb9

This would show that udev might create three symlinks, a symlink named/dev/asmcrs01, one symlink in /dev/disk/by-id/, and third symlink in /dev/disk/by-path/. We will use /dev/asmcrs01 symlink for ASM setup

Output will be like this,

[root@rac1 ~]# udevtest /block/sdb/sdb5

main: looking at device ‘/block/sdb/sdb5’ from subsystem ‘block’
udev_rules_get_name: add symlink ‘disk/by-id/scsi-SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4-part5’
udev_rules_get_name: add symlink ‘disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part5’
run_program: ‘/lib/udev/vol_id –export /dev/.tmp-8-21’
run_program: ‘/lib/udev/vol_id’ returned with status 4
run_program: ‘/sbin/scsi_id -g -u -s /block/sdb’
run_program: ‘/sbin/scsi_id’ (stdout) ‘SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4_’
run_program: ‘/sbin/scsi_id’ returned with status 0
udev_rules_get_name: rule applied, ‘sdb5’ becomes ‘asm-disk1’
udev_device_event: device ‘/block/sdb/sdb5’ already in database, validate currently present symlinks
udev_node_add: creating device node ‘/dev/asm-disk1’, major = ‘8’, minor = ’21’, mode = ‘0660’, uid = ‘502’, gid = ‘501’
udev_node_add: creating symlink ‘/dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4-part5’ to ‘../../asm-disk1’
udev_node_add: creating symlink ‘/dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-part5’ to ‘../../asm-disk1’
main: run: ‘socket:/org/kernel/udev/monitor’
main: run: ‘/lib/udev/udev_run_devd’
main: run: ‘socket:/org/freedesktop/hal/udev_event’
main: run: ‘/sbin/pam_console_apply /dev/asm-disk1 /dev/disk/by-id/scsi-SATA_VBOX_HARDDISK_VB52c7cf45-b84d83a4-part5 /dev/disk/by-path/pci-0000:00:0d.0-scsi-1:0:0:0-par

6. Reload rules and start udev

This should create the symlink in /dev/asm-disk01

# In OEL5/RHEL5

# /sbin/udevcontrol reload_rules

# In OEL6/RHEL6

# udevadm control –reload-rules

# In OL5 and OL6

# /sbin/start_udev.

7. Check Ownership and Permissions

Check the disks are now available with the “asm-disk*” alias and the correct ownership and permissions.

# cd /dev
# ls -al asm-disk*

brw-rw—- 1 oracle dba 8, 17 Apr 8 22:47 asm-disk1
brw-rw—- 1 oracle dba 8, 33 Apr 8 22:47 asm-disk2
brw-rw—- 1 oracle dba 8, 49 Apr 8 22:47 asm-disk3
brw-rw—- 1 oracle dba 8, 65 Apr 8 22:47 asm-disk4

8. Now, setup asm_diskstring parameter to ‘/dev/asmcrs*’ so that ASM will identify these devices.

9. Now copy the file /etc/udev/rules.d/99-asmdevices.rules to all nodes of RAC cluster and restart udev