On Tue, Jan 17, 2017 at 05:45:53PM +0200, Sagi Grimberg wrote: > > >-- > >[1] > >queue = b'nvme0q1' > > usecs : count distribution > > 0 -> 1 : 7310 |****************************************| > > 2 -> 3 : 11 | | > > 4 -> 7 : 10 | | > > 8 -> 15 : 20 | | > > 16 -> 31 : 0 | | > > 32 -> 63 : 0 | | > > 64 -> 127 : 1 | | > > > >[2] > >queue = b'nvme0q1' > > usecs : count distribution > > 0 -> 1 : 7309 |****************************************| > > 2 -> 3 : 14 | | > > 4 -> 7 : 7 | | > > 8 -> 15 : 17 | | > > > > Rrr, email made the histograms look funky (tabs vs. spaces...) > The count is what's important anyways... > > Just adding that I used an Intel P3500 nvme device. > > >We can see that most of the time our latency is pretty good (<1ns) but with > >huge tail latencies (some 8-15 ns and even one in 32-63 ns). > > Obviously is micro-seconds and not nano-seconds (I wish...) So to share yesterday's (and today's) findings: On AHCI I see only one completion polled as well. This probably is because in contrast to networking (with NAPI) in the block layer we do have a link between submission and completion whereas in networking RX and TX are decoupled. So if we're sending out one request we get the completion for it. What we'd need is a link to know "we've sent 10 requests out, now poll for the 10 completions after the 1st IRQ". So basically what NVMe already did with calling __nvme_process_cq() after submission. Maybe we should even disable IRQs when submitting and re-enable after submitting so the submission patch doesn't get preempted by a completion. Does this make sense? Byte, Johannes -- Johannes Thumshirn Storage jthumshirn@xxxxxxx +49 911 74053 689 SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850 -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
