Create Oracle ASM disks using multipath devices and udev rules on Oracle Linux / Red Hat Enterprise Linux

 Since Oracle Automatic Storage Management Filter Driver (ASMFD) was released with Oracle Database 12c, DBA has very few chance to work with udev rules when creating ASM disks. 

Fortunately, Oracle Database 19.6 introduced a new feature, Zero-Downtime Grid Infrastructure patching which brings the requirement of udev rules back to DBA. When Oracle ASMFD is used for ASM disks, Oracle patching has to update the operating system driver for ASMFD, which cannot be done in zero-downtime mode. Therefore, in order to patch Grid Infrastructure in zero-downtime mode instead of rolling mode, udev rules becomes a perfect option.

Udev rules can help us maintain the persistent device's ownership and permission across system reboot, so that ASM instance can access the disk all the time. I am going to demonstrate how to do it for multipath devices.

1. Identify the devices

Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging. By default, the name of a multipath device is set to its WWID. Alternately, you can set the user_friendly_names option in the multipath configuration file, and Device Mapper Multipathing, as a Linux native multpath tool, creates a block device under /dev/mapper for each LUN attached to the system with a node-unique name of the form mpathN.

An example of a system with multipath devices attached

[root@rac01]# ls -l /dev/mapper/

total 0

crw------- 1 root root 10, 236 Nov 17 12:17 control

lrwxrwxrwx 1 root root       8 Nov 26 18:06 mpatha -> ../dm-11

lrwxrwxrwx 1 root root       8 Nov 26 18:06 mpathb -> ../dm-13

lrwxrwxrwx 1 root root       8 Nov 26 18:06 mpathc -> ../dm-15

[root@rac01]# ls -l /dev/dm*

brw-rw---- 1 root disk  249, 11 Nov 26 18:06 /dev/dm-11

brw-rw---- 1 root disk  249, 13 Nov 26 18:06 /dev/dm-13

brw-rw---- 1 root disk  249, 15 Nov 26 18:06 /dev/dm-15

[root@rac01]# multipath -ll

mpathb (360060e8012a1b4005040a1b400000133) dm-13 HITACHI ,OPEN-V

size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 8:0:0:7   sdo                8:224      active ready running

  |- 10:0:0:7  sdn                8:208      active ready running

  |- 8:0:1:7   sdam               66:96      active ready running

  `- 10:0:1:7  sdag               66:0       active ready running

mpatha (360060e8012a1b4005040a1b400000132) dm-11 HITACHI ,OPEN-V

size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 8:0:0:6   sdm                8:192      active ready running

  |- 10:0:0:6  sdl                8:176      active ready running

  |- 8:0:1:6   sdal               66:80      active ready running

  `- 10:0:1:6  sdae               65:224     active ready running

mpathc (360060e8012a1b4005040a1b400000135) dm-15 HITACHI ,OPEN-V

size=400G features='1 queue_if_no_path' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=1 status=active

  |- 8:0:0:9   sds                65:32      active ready running

  |- 10:0:0:9  sdr                65:16      active ready running

  |- 8:0:1:9   sdaq               66:160     active ready running

  `- 10:0:1:9  sdak               66:64      active ready running

From the example, we can find that, one same multipath disk can appear in three different forms,

   1. /dev/mapper/mpath*   Device alias created early in the boot process as symbol link file and persistent while adding/removing devices and rebooting system. It is recommended to be used for ASM disk identifier.

   2. /dev/dm-*            Actual multipath device name (ID), could be changed after rebooting system, internal use only by multipath tools. Never use it directly for ASM disk

   3. /dev/sd*             Device name for each non-multipath device or each channel (path) of multpath device

In the example, /dev/mapper/mpathb is alias of actual multipath device /dev/dm-13, and /dev/dm-13 device has four paths and four devices (/dev/sdo, /dev/sdn, /dev/sdam, /dev/sdag) are created to represent the four paths respectively. Any one of these four /dev/sd* devices can be accessed, but the access is to the same disk. However, when you use one of /dev/sd* device name to access the disk and the path is disconnected because of any reason, you will loose the device access. Instead, if /dev/mapper/mpathb is used to access the disk, the disk is available until all four paths have problem at same time.

Run command 'udevadm info' to get device identification information

[root@rac01]# udevadm info --query=all --name=/dev/mapper/mpathb

P: /devices/virtual/block/dm-13

N: dm-13

L: 10

S: disk/by-id/dm-name-mpathb

S: disk/by-id/dm-uuid-mpath-360060e8012a1b4005040a1b400000133

S: mapper/mpathb

E: DEVLINKS=/dev/disk/by-id/dm-name-mpathb /dev/disk/by-id/dm-uuid-mpath-360060e8012a1b4005040a1b400000133 /dev/mapper/mpathb

E: DEVNAME=/dev/dm-13

E: DEVPATH=/devices/virtual/block/dm-13

E: DEVTYPE=disk

E: DM_NAME=mpathb

E: DM_SUSPENDED=0

E: DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1

E: DM_UDEV_PRIMARY_SOURCE_FLAG=1

E: DM_UDEV_RULES_VSN=2

E: DM_UUID=mpath-360060e8012a1b4005040a1b400000133

E: ID_FS_TYPE=oracleasm

E: ID_FS_USAGE=filesystem

E: MAJOR=249

E: MINOR=13

E: MPATH_SBIN_PATH=/sbin

E: SUBSYSTEM=block

E: TAGS=:systemd:

E: USEC_INITIALIZED=6853149908

In the command output, either DM_NAME or DM_UUID can be used to uniquely identify the multipath device /dev/mapper/mpathb.

2. Create udev rules

Although the ownership and permission of a device can be configured manually with commands 'chown', 'chgrp' and 'chmod', no DBA use it because the ownership and permission of the device will be reset and root always become the owner when the system is rebooted. In order to maintain persistent ownership and permission of a device, udev rule has to be created.

Create a text file '52-oracle.rules' under directory /etc/udev/rules.d/ to include following line

ACTION=="add|change", ENV{DM_NAME}=="mpathb", OWNER="grid", GROUP="asmadmin", MODE="0660"

It is using the DM_NAME value (mpathb) of device /dev/mapper/mpathb, the values is retrieved by running command 'udevadm info' upon the device. Alternatively, following rule can be used to set ownership and permission on same device,

ACTION=="add|change", ENV{DM_UUID}=="mpath-360060e8012a1b4005040a1b400000133", OWNER="grid", GROUP="asmadmin", MODE="0660"

Either of them works because both DM_NAME and DM_UUID are unique on the server and can be used to uniquely identify the same device, but do not use both of them at same time though it is not harmful.

After the new udev rule is loaded, the owner of /dev/mapper/mpathb becomes grid, group becomes asmadmin, and only owner grid and group asmadmin have read/write permission on this deivce. 

Now, what if I need second device /dev/mapper/mpathc as ASM disk? Yes, you can add one more line into the same rules file as following

ACTION=="add|change", ENV{DM_NAME}=="mpathb", OWNER="grid", GROUP="asmadmin", MODE="0660"

ACTION=="add|change", ENV{DM_NAME}=="mpathc", OWNER="grid", GROUP="asmadmin", MODE="0660"

But, what if I need more devices as ASM disks? Do I have to modify udev rules every time when I add devices as ASM disks? Following rule can help

ACTION=="add|change", ENV{DM_NAME}=="mpath*", OWNER="grid", GROUP="asmadmin", MODE="0660"

Any devices which names start with mpath will be configured as ASM disks. Wait, what if not all /dev/mapper/mpath* devices are used as ASM disks? In the previous example system, there are three multipath devices mpatha, mpathb and mpathc, and only two of them are used as ASM disks and maybe new devices, in the future, will be added as ASM disks. The solution for this scenario is to customize (rename) the devices' name with more meanful strings. 

I renamed the devices' name on the example system as following

[root@rac01]# ls -l /dev/mapper/

total 0

crw------- 1 root root 10, 236 Nov 17 12:17 control

lrwxrwxrwx 1 root root       8 Nov 26 18:26 asmDATA -> ../dm-13

lrwxrwxrwx 1 root root       8 Nov 26 18:28 asmFRA -> ../dm-15

lrwxrwxrwx 1 root root       8 Nov 26 18:06 mpatha -> ../dm-11

If you struggle with changing device names, the article Rename / Change the Multipath Device Names on Linux may help.

Since all ASM disk devices are named starting with asm, the udev rules can be changed as following

ACTION=="add|change", ENV{DM_NAME}=="asm*", OWNER="grid", GROUP="asmadmin", MODE="0660"

As soon as you add new deivces and name it with prefix 'asm', udev rules will set proper permissions automatically without modifying the udev rules.

3. Reload udev rules

As best practice, it is recommended to reboot the system after new udev rules are created, it can validate and guarantee the rules work across rebooting. However, sometimes, you may want to validate the rules without rebooting the system because you cannot get downtime.

The udev rules can be validated/tested with 'udevadm control --reload-rules' and 'udevadm trigger --type=devices' as following

[root@rac01]# ls -l /dev/mapper/

total 0

crw------- 1 root root 10, 236 Nov 17 12:17 control

lrwxrwxrwx 1 root root       8 Nov 26 18:26 asmDATA -> ../dm-13

lrwxrwxrwx 1 root root       8 Nov 26 18:28 asmFRA -> ../dm-15

lrwxrwxrwx 1 root root       8 Nov 26 18:06 mpatha -> ../dm-11

[root@rac01]# ls -l /dev/dm*

brw-rw---- 1 root disk  249, 11 Nov 26 18:06 /dev/dm-11

brw-rw---- 1 root disk  249, 13 Nov 26 18:26 /dev/dm-13

brw-rw---- 1 root disk  249, 15 Nov 26 18:28 /dev/dm-15

[root@rac01]# udevadm control --reload

[root@rac01]# udevadm trigger --type=devices --action=change

[root@rac01]# 

[root@rac01]# ls -l /dev/dm*

brw-rw---- 1 root disk      249, 11 Nov 26 18:54 /dev/dm-11

brw-rw---- 1 grid asmadmin  249, 13 Nov 26 18:54 /dev/dm-13

brw-rw---- 1 grid asmadmin  249, 15 Nov 26 18:54 /dev/dm-15

The devices' ownership are changed as expected.