On Thu, Jun 19, 2008 at 12:20:03AM +0800, sunhux G wrote: > We have SAN disk partitions presented to 2 Redhat Linux servers, running > Oracle RAC (cluster) ASM. Hey there, First off, you're using ASMLib, so you don't need to worry about device naming - ASMLib makes sure you are accessing the same device. But read on for furthur discussion. > As suggested by Linux "man multipath", I've made the > /var/lib/multipath/bindings the same > on both servers "to ensure That the multipath devices have the same names on > all nodes > accessing them" : > # Format: > # alias wwid > mpath0 360a98000567244396334493370345055 This works as you expect. The names in /dev/(mapper|mpath)/mpathX should be identical on each server. > However /proc/partition on both servers showed differences on the > dm-x devices (but not on the /dev/sdxx devices) : The dm-X devices are not expected to be the same. You see, the system creates the dm-X devices in the order they are seen - this can be different. After they are created, the udev subsystem uses the information in /var/lib/multipath/bindings to map the mpathX name to the appropriate dm-X device. This ensures that the mpathX names are identical on each system. > Q1 : > Do mpath0, mpath1, ... shown in "multipath -ll" listing refer to > /dev/mapper/mpathx or to /dev/mpath/mpathx ? If there are names /dev/mapper/mpathx and /dev/mpath/mpathx, they should be refer to the same thing. > Q2 : > How do we make it such that the dm-x devices are accessing the same > SAN LUNs across the servers? I believe they are not the same based > on the observations of the disk spaces associated with each of the dm-x > shown in /proc/partition You don't. You don't care. If you want a /dev name, you use the mpathX name that you know maps correctly. Since you are using ASMLib, you don't worry about that either. The LANDx names are read by ASMLib to ensure ASM sees the correct devices no matter what /dev name they have. > [root@landnet-rac1-temp mpath]# /etc/init.d/oracleasm querydisk land4 > Disk "LAND4" is a valid ASM disk on device [253, 0]On Server 1 : That maps to dm-0. > [root@landnet-rac2-temp dev]# /etc/init.d/oracleasm scandisks > Scanning system for ASM disks: OK ] > [root@landnet-rac2-temp dev]# /etc/init.d/oracleasm querydisk land4 > Disk "LAND4" is a valid ASM disk on device [8, 32] <<== note the number And that is /dev/sdc. Your problem is SCANORDER. ASMLib's disk scan is seeing the scsi device (one of the paths) before the multipath device. This is actually safe, it just doesn't take advantage of the multipath. You want to use the multipath of course, so you want to configure it correctly. See the instructions at http://www.oracle.com/technology/tech/linux/asmlib/multipath.html to configure ASMLib to see dm devices before scsi devices. Then your scandisks will always see the multipath devices. > We logged a call to Oracle who responded : > If we are using device mapper and ASMLib then, we need to use disks from > /dev/dm-* disks > instead of disks from /dev/mapper/mpath* That's not the issue, and I'm sorry Oracle support gave you the wrong answer. You can createdisk against any name that's correct (/dev/dm-X, /dev/mapper/mpathX, /dev/mpath/mpathX). scandisks doesn't even need a name - it just needs the correct order in your case. Joel -- "War doesn't determine who's right; war determines who's left." Joel Becker Principal Software Developer Oracle E-mail: joel.becker@xxxxxxxxxx Phone: (650) 506-8127 -- Linux-cluster mailing list Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
