On 6/24/23 04:45, Jianlin Lv wrote:
On Fri, Jun 9, 2023 at 12:23 AM Bart Van Assche <bvanassche@xxxxxxx> wrote:
On 6/7/23 19:51, Jianlin Lv wrote:
On Thu, Jun 8, 2023 at 1:07 AM Bart Van Assche <bvanassche@xxxxxxx> wrote:
On 6/7/23 08:55, Jianlin Lv wrote:
I see two possible solutions:
- Change the volume provisioner such that it uses disk references that do
not depend on the probing order, e.g. /dev/disk/by-id/...
Yes, The "/dev/disk/by-id/" can uniquely identify SCSI devices. However,
I don't think it is suitable for the volume provisioner workflow.
For nodes of the same SKU , a unified YAML file will be defined to instruct
the volume provisioner on how to manage the local disks.
If use WWID, it would mean that a unique YAML file needs to be defined
for each node. This approach becomes impractical when dealing with a large
number of work nodes.
Please consider using the paths available in /dev/disk/by-path.
Sorry for the late reply.
I carefully checked the server in the production environment and found
some corner cases where there are differences in the dev/disk/by-path/ of
nodes with the same SKU. These differences are caused by inconsistent
target_numbers.
For example:
diff -y aa-by-path bb-by-path
pci-0000:86:00.0-scsi-0:3:86:0 -> ../../sda |
pci-0000:86:00.0-scsi-0:3:88:0 -> ../../sda
pci-0000:86:00.0-scsi-0:3:87:0 -> ../../sdb |
pci-0000:86:00.0-scsi-0:3:89:0 -> ../../sdb
pci-0000:86:00.0-scsi-0:3:88:0 -> ../../sdc |
pci-0000:86:00.0-scsi-0:3:90:0 -> ../../sdc
pci-0000:86:00.0-scsi-0:3:89:0 -> ../../sdd |
pci-0000:86:00.0-scsi-0:3:91:0 -> ../../sdd
pci-0000:86:00.0-scsi-0:3:90:0 -> ../../sde |
pci-0000:86:00.0-scsi-0:3:92:0 -> ../../sde
pci-0000:86:00.0-scsi-0:3:92:0 -> ../../sdf |
pci-0000:86:00.0-scsi-0:3:93:0 -> ../../sdf
pci-0000:86:00.0-scsi-0:3:93:0 -> ../../sdg |
pci-0000:86:00.0-scsi-0:3:94:0 -> ../../sdg
pci-0000:86:00.0-scsi-0:3:94:0 -> ../../sdh |
pci-0000:86:00.0-scsi-0:3:95:0 -> ../../sdh
I'm still not sure what causes the target_numbers to be different.
However, the existence of such corner cases makes /dev/disk/by-path
unusable for the volume provisioner, similar to /dev/disk/by-id/.
So, If it's not possible to configure disk serialization detection, then
it seems that implementing predictable disk names is the only option.
I think the following code from the systemd source file
src/udev/udev-builtin-path_id.c generates the last part of the path ID:
[ ... ]
if (sscanf(name, "%d:%d:%d:%d", &host, &bus, &target, &lun)!=4)
return NULL;
[ ... ]
path_prepend(path, "scsi-%i:%i:%i:%i", host, bus, target, lun);
[ ... ]
Is megasas_add_remove_devices() the code that adds SCSI devices on your
setup? In that code I see that scsi_device_lookup() is called in a loop
until channel and target numbers are found that are not in use. I think
this is what makes the target numbers unpredictable on your setup. There
must be a better way for the megaraid driver to assign a target number.
Bart.
