On Fri, May 03, 2024 at 02:58:56PM +0200, Jesper Dangaard Brouer wrote: > > > On 02/05/2024 21.44, Shakeel Butt wrote: > > On Wed, May 01, 2024 at 07:22:26PM +0200, Jesper Dangaard Brouer wrote: > > > > > [...] > > > > > > More data, the histogram of time spend under the lock have some strange > > > variation issues with a group in 4ms to 65ms area. Investigating what > > > can be causeing this... which next step depend in these tracepoints. > > > > > > @lock_cnt: 759146 > > > > > > @locked_ns: > > > [1K, 2K) 499 | | > > > [2K, 4K) 206928 > > > |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| > > > [4K, 8K) 147904 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > > > [8K, 16K) 64453 |@@@@@@@@@@@@@@@@ | > > > [16K, 32K) 135467 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | > > > [32K, 64K) 75943 |@@@@@@@@@@@@@@@@@@@ | > > > [64K, 128K) 38359 |@@@@@@@@@ | > > > [128K, 256K) 46597 |@@@@@@@@@@@ | > > > [256K, 512K) 32466 |@@@@@@@@ | > > > [512K, 1M) 3945 | | > > > [1M, 2M) 642 | | > > > [2M, 4M) 750 | | > > > [4M, 8M) 1932 | | > > > [8M, 16M) 2114 | | > > > [16M, 32M) 1039 | | > > > [32M, 64M) 108 | | > > > > > > > Am I understanding correctly that 1K is 1 microsecond and 1M is 1 > > millisecond? > > Correct. > > > Is it possible to further divide this table into update > > side and flush side? > > > > This is *only* flush side. > > You question indicate, that we are talking past each-other ;-) > > Measurements above is with (recently) accepted tracepoints (e.g. not the > proposed tracepoints in this patch). I'm arguing with existing > tracepoint that I'm seeing this data, and arguing I need per-CPU > tracepoints to dig deeper into this (as proposed in this patch). Ah my mistake, I just assumed that the data shown is with the given patchset. > > The "update side" can only be measured once we apply this patch. > > This morning I got 6 prod machines booted with new kernels, that contain > this proposed per-CPU lock tracepoint patch. And 3 of these machines have > the Mutex lock change also. No data to share yet... > Eagerly waiting for the results. Also I don't have any concerns with these new traces. > --Jesper
