On 6/8/21 4:26 PM, Martin Wilck wrote:
On Mo, 2021-06-07 at 16:30 -0500, David Teigland wrote:
On Mon, Jun 07, 2021 at 10:27:20AM +0000, Martin Wilck wrote:
Most importantly, this was about LVM2 scanning of physical volumes.
The
number of udev workers has very little influence on PV scanning,
because the udev rules only activate systemd service. The actual
scanning takes place in lvm2-pvscan@.service. And unlike udev,
there's
no limit for the number of instances of a given systemd service
template that can run at any given time.
Excessive device scanning has been the historical problem in this area,
but Heming mentioned dev_cache_scan() specifically as a problem. That
was
surprising to me since it doesn't scan/read devices, it just creates a
list of device names on the system (either readdir in /dev or udev
listing.) If there are still problems with excessive
scannning/reading,
we'll need some more diagnosis of what's happening, there could be some
cases we've missed.
Heming didn't include his measurement results in the initial post.
Here's a small summary. Heming will be able to provide more details.
You'll see that the effects are quite drastic, factors 3-4 between
every step below, factor >60 between best and worst. I'd say these
results are typical for what we observe also on real-world systems.
kvm-qemu, 6 vcpu, 20G memory, 1258 scsi disks, 1015 vg/lv
Shown is "systemd-analyze blame" output.
1) lvm2 2.03.05 (SUSE SLE15-SP2),
obtain_device_list_from_udev=1 & event_activation=1
9min 51.782s lvm2-pvscan@253:2.service
9min 51.626s lvm2-pvscan@65:96.service
(many other lvm2-pvscan@ services follow)
2) lvm2 latest master
obtain_device_list_from_udev=1 & event_activation=1
2min 6.736s lvm2-pvscan@70:384.service
2min 6.628s lvm2-pvscan@70:400.service
3) lvm2 latest master
obtain_device_list_from_udev=0 & event_activation=1
40.589s lvm2-pvscan@131:976.service
40.589s lvm2-pvscan@131:928.service
4) lvm2 latest master
obtain_device_list_from_udev=0 & event_activation=0,
21.034s dracut-initqueue.service
8.674s lvm2-activation-early.service
IIUC, 2) is the effect of _pvscan_aa_quick(). 3) is surprising;
apparently libudev's device detection causes a factor 3 slowdown.
While 40s is not bad, you can see that event based activation still
performs far worse than "serial" device detection lvm2-activation-
early.service.
Personally, I'm sort of wary about obtain_device_list_from_udev=0
because I'm uncertain whether it might break multipath/MD detection.
Perhaps you can clarify that.
Regards
Martin
my latest test results. there combines 3 cfg items:
devices/obtain_device_list_from_udev
global/event_activation
activation/udev_sync
<0> is under lvm2-2.03.05+
<1> ~ <8> is under lvm2-2.03.12+
all results are from "systemd-analyze blame", and I only
post top n services.
0>
with suse 15sp2 lvm2 version: lvm2-2.03.05+
"systemd-analyze blame" show the top serives:
devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=1
9min 51.782s lvm2-pvscan@253:2.service <===
9min 51.626s lvm2-pvscan@65:96.service
9min 51.625s lvm2-pvscan@65:208.service
9min 51.624s lvm2-pvscan@65:16.service
9min 51.622s lvm2-pvscan@8:176.service
9min 51.614s lvm2-pvscan@65:144.service
1>
devices/obtain_device_list_from_udev=1
global/event_activation=0
activation/udev_sync=0
18.307s dracut-initqueue.service
6.168s btrfsmaintenance-refresh.service
4.327s systemd-udev-settle.service
3.633s wicked.service
2.976s lvm2-activation-early.service <===
1.560s lvm2-pvscan@135:832.service
1.559s lvm2-pvscan@135:816.service
1.558s lvm2-pvscan@135:784.service
1.558s lvm2-pvscan@134:976.service
1.557s lvm2-pvscan@134:832.service
1.556s dev-system-swap.swap
1.554s lvm2-pvscan@134:992.service
1.553s lvm2-pvscan@134:1008.service
2>
devices/obtain_device_list_from_udev=0
global/event_activation=0
activation/udev_sync=0
17.164s dracut-initqueue.service
10.420s wicked.service
7.109s btrfsmaintenance-refresh.service
4.471s systemd-udev-settle.service
3.415s lvm2-activation-early.service <===
1.679s lvm2-pvscan@135:816.service
1.678s lvm2-pvscan@135:832.service
1.677s lvm2-pvscan@134:992.service
1.675s lvm2-pvscan@135:784.service
1.674s lvm2-pvscan@134:928.service
1.673s lvm2-pvscan@134:896.service
1.673s dev-system-swap.swap
1.672s lvm2-pvscan@134:1008.service
3>
devices/obtain_device_list_from_udev=1
global/event_activation=0
activation/udev_sync=1
17.552s dracut-initqueue.service
7.401s lvm2-activation-early.service <====
6.519s btrfsmaintenance-refresh.service
5.375s systemd-udev-settle.service
3.588s wicked.service
1.723s wickedd-nanny.service
1.686s wickedd.service
1.655s lvm2-pvscan@129:992.service
1.654s lvm2-pvscan@129:960.service
1.653s lvm2-pvscan@129:896.service
1.652s lvm2-pvscan@130:784.service
1.651s lvm2-pvscan@130:768.service
4>
devices/obtain_device_list_from_udev=0
global/event_activation=0
activation/udev_sync=1
17.975s dracut-initqueue.service
10.162s wicked.service
8.238s lvm2-activation-early.service <===
6.955s btrfsmaintenance-refresh.service
4.444s systemd-udev-settle.service
1.800s rsyslog.service
1.768s wickedd.service
1.751s kbdsettings.service
1.751s kdump-early.service
1.602s lvm2-pvscan@135:832.service
1.601s lvm2-pvscan@135:816.service
1.601s lvm2-pvscan@135:784.service
1.600s lvm2-pvscan@134:1008.service
1.599s dev-system-swap.swap
1.598s lvm2-pvscan@134:832.service
5>
devices/obtain_device_list_from_udev=0
global/event_activation=1
activation/udev_sync=1
34.908s dracut-initqueue.service
25.440s systemd-udev-settle.service
23.335s lvm2-pvscan@66:832.service <===
23.335s lvm2-pvscan@65:976.service
23.335s lvm2-pvscan@66:784.service
23.335s lvm2-pvscan@65:816.service
23.335s lvm2-pvscan@8:976.service
23.327s lvm2-pvscan@66:864.service
23.323s lvm2-pvscan@66:848.service
23.316s lvm2-pvscan@65:800.service
6>
devices/obtain_device_list_from_udev=0
global/event_activation=1
activation/udev_sync=0
36.222s lvm2-pvscan@134:912.service <===
36.222s lvm2-pvscan@134:816.service
36.222s lvm2-pvscan@134:784.service
36.221s lvm2-pvscan@133:816.service
36.221s lvm2-pvscan@133:848.service
36.220s lvm2-pvscan@133:928.service
36.220s lvm2-pvscan@133:768.service
36.219s lvm2-pvscan@133:992.service
36.218s lvm2-pvscan@133:784.service
36.218s lvm2-pvscan@134:800.service
36.218s lvm2-pvscan@133:864.service
36.217s lvm2-pvscan@133:896.service
36.209s lvm2-pvscan@133:960.service
36.197s lvm2-pvscan@134:1008.service
7>
devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=1
2min 6.736s lvm2-pvscan@70:384.service <===
2min 6.628s lvm2-pvscan@70:400.service
2min 6.554s lvm2-pvscan@69:432.service
2min 6.518s lvm2-pvscan@69:480.service
2min 6.478s lvm2-pvscan@69:416.service
2min 6.277s lvm2-pvscan@69:464.service
2min 5.791s lvm2-pvscan@69:544.service
8>
devices/obtain_device_list_from_udev=1
global/event_activation=1
activation/udev_sync=0
2min 27.091s lvm2-pvscan@129:944.service <===
2min 26.952s lvm2-pvscan@129:912.service
2min 26.950s lvm2-pvscan@129:880.service
2min 26.947s lvm2-pvscan@129:960.service
2min 26.947s lvm2-pvscan@129:928.service
2min 26.947s lvm2-pvscan@129:832.service
2min 26.938s lvm2-pvscan@129:848.service
2min 26.733s lvm2-pvscan@129:864.service
2min 16.241s lvm2-pvscan@66:976.service
2min 15.166s lvm2-pvscan@66:992.service
_______________________________________________
linux-lvm mailing list
linux-lvm@xxxxxxxxxx
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
