On Wed, Mar 23, 2022 at 3:26 PM Clark Williams <williams@xxxxxxxxxx> wrote: > > Something that's not always obvious is that BIOS settings can affect your latency numbers. Look in the rt-tests project for a script called 'hwlatdetect' and try running that to see if control is transferring to the BIOS via an SMI or MCE. Here's a sample run on an untuned Intel NUC: > > $ sudo hwlatdetect --duration=30s OK, this was very useful. This has returned the following for me: node-0> sudo hwlatdetect --duration=30s hwlatdetect: test duration 30 seconds detector: tracer parameters: Latency threshold: 10us Sample window: 1000000us Sample width: 500000us Non-sampling period: 500000us Output File: None Starting test test finished Max Latency: 811us Samples recorded: 23 Samples exceeding threshold: 23 ts: 1648082981.696619376, inner:811, outer:19 ts: 1648082982.999941947, inner:545, outer:16 ts: 1648082985.249826218, inner:0, outer:426 ts: 1648082986.249810897, inner:409, outer:0 ts: 1648082987.249818630, inner:0, outer:415 ts: 1648082988.249822370, inner:417, outer:0 ts: 1648082989.249816168, inner:409, outer:0 ts: 1648082990.249820394, inner:412, outer:0 ts: 1648082991.249820365, inner:0, outer:410 ts: 1648082992.249825886, inner:0, outer:414 ts: 1648082993.249822956, inner:409, outer:0 ts: 1648082994.249822720, inner:407, outer:0 ts: 1648082995.249826191, inner:409, outer:5 ts: 1648082996.249832029, inner:413, outer:0 ts: 1648082997.249832007, inner:412, outer:0 ts: 1648082998.249842325, inner:0, outer:420 ts: 1648082999.249838335, inner:0, outer:415 ts: 1648083000.249836740, inner:412, outer:0 ts: 1648083001.249835774, inner:409, outer:0 ts: 1648083002.249832951, inner:0, outer:404 ts: 1648083003.249833781, inner:0, outer:404 ts: 1648083004.249837415, inner:0, outer:406 ts: 1648083005.249851829, inner:0, outer:418 node-0> I see one inner value of > 800usec, and several cases where sum of "outer" + (next) "inner" is > 800 usec. If I can reduce this to the 55 usec you see on your "untuned" Intel NUC (what is NUC BTW?) I would be quite happy. Is there a pointer to any documentation as to what I may try on the BIOS settings? And I am not familiar w/ "...since transitioning between c-states is another place you can see latency spikes", any pointers regarding that also will be appreciated. Gautam