On Apr 10, 2024 / 07:51, Tejun Heo wrote:
> Hello,
>
> On Wed, Apr 10, 2024 at 05:45:31PM +0900, Shin'ichiro Kawasaki wrote:
> > Commit 5797b1c18919 ("workqueue: Implement system-wide nr_active
> > enforcement for unbound workqueues") modified the maximum number of
> > active works that an unbound workqueue can handle to at most
> > WQ_DFL_MIN_ACTIVE (8 by default). This commit thus limits the number of
>
> This shouldn't be the case. The default max_active remains the same at 256.
> MIN_ACTIVE is used only to guarantee minimum forward progress guarantee when
> @max_active too low in multi NUMA setups. It's unexpected that the commit
> caused significant behavior difference on a single NUMA machines. The limits
> and enforcement for single NUMA machien shouldn't have changed.
>
> > active dm-zoned chunk works that execute concurrently on a single NUMA
> > node machine. This reduction results in garbage collection performance
> > degradation which manifests itself with longer unmount time with the xfs
> > file system on dm-zoned devices.
> >
> > To restore unmount duration with dm-zoned devices, drop the WQ_UNBOUND
> > flag for the chunk workqueue, thus allowing more than WQ_DFL_MIN_ACTIVE
> > chunk works. Though this change bounds all chunk works to the same CPU,
> > it provides more parallelism and improved performance. The table below
> > shows the average xfs unmount time of 10 times measurements, using a
> > single NUMA node machine with 32 CPUs. The xfs volume was prepared on
> > dm-zoned devices on top of an SMR HDD with 26GB dm-linear clip, then
> > filled with data files before executing unmount.
> >
> > Kernel | Unmount time
> > ---------------------+--------------
> > v6.8 | 29m 3s
> > v6.9-rc2 | 34m 17s
> > v6.9-rc2 + this fix | 27m 12s
>
> Can you please run `drgn tools/workqueue/wq_monitor.py 'dmz_cwq.*'` while
> testing? It should show how many work items are in flight and how much CPU
> time the workqueues are consuming.
Sure. I manually ran the command below during the unmount operation.
$ sudo drgn ./tools/workqueue/wq_monitor.py -i 10 'dmz_cwq.*'
Please find the command outputs below. I captured the outputs with three kernel
variations. Each of them shows the output for a minute.
Kernel: v6.8
--------------------------------------------------------------------------------
...
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 825 76 17.1 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 825 75 17.6 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 826 74 18.2 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 826 73 18.6 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 828 73 19.1 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 829 71 19.6 - 0 0 0
...
--------------------------------------------------------------------------------
Kernel: v6.9-rc2
--------------------------------------------------------------------------------
...
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 580 7 3.9 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 581 7 3.9 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 584 8 3.9 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 585 8 3.9 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 586 8 3.9 - 0 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 587 8 4.0 - 0 0 0
...
--------------------------------------------------------------------------------
Kernel: v6.9-rc2 + the patch
--------------------------------------------------------------------------------
...
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1439 46 8.2 0 2812 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1441 44 8.3 0 2813 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1442 43 8.3 0 2817 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1443 42 8.3 0 2821 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1446 42 8.3 0 2824 0 0
total infl CPUtime CPUitsv CMW/RPR mayday rescued
dmz_cwq_dmz_dml_072 1447 41 8.4 0 2828 0 0
...
--------------------------------------------------------------------------------
