On Tue, 2018-01-23 at 09:44 -0500, Mike Snitzer wrote: > On Tue, Jan 23 2018 at 9:27am -0500, > Ming Lei <ming.lei@xxxxxxxxxx> wrote: > > > Hello Martin, > > > > On Tue, Jan 23, 2018 at 08:30:41AM -0500, Martin K. Petersen wrote: > > > > > > Ming, > > > > > > > + * Block layer and block driver specific status, which is > > > > ususally returnd > > > > > > > > > ^^^^^^^ > > > > + * from driver to block layer in IO path. > > > > > > Given that the comment blurb is long and the flag not defined > > > until > > > later, it is not entirely obvious that you are documenting > > > BLK_STS_DEV_RESOURCE. So please make that clear at the beginning > > > of the > > > comment. > > > > OK, how about the following document? > > > > /* > > * BLK_STS_DEV_RESOURC: Block layer and block driver specific > > status, > > ^^^^^^^^^^^^^^^^^^^ typo > > > * which is usually returned from driver to block layer in IO path. > > * > > * BLK_STS_DEV_RESOURCE is returned from driver to block layer if > > device > > * related resource is run out of, but driver can guarantee that > > queue > > * will be rerun in future for dispatching the current request when > > this > > * resource is available. > > * > > * Difference with BLK_STS_RESOURCE: > > * If driver isn't sure if the queue can be run again for dealing > > with the > > * current request after this kind of resource is available, please > > return > > * BLK_STS_SOURCE, for example, when memory allocation, DMA Mapping > > or other > > ^^^^^^^^^^^^^^ typo > > > * system resource allocation fails and IO can't be submitted to > > device, > > * BLK_STS_RESOURCE should be used for avoiding IO hang. > > */ > > In general the 2nd paragraph is one big run-on sentence. Needs some > isolation. I built a new kernel from Mikes latest tree which has all Ming's latest patches so I could sanity test this on SRP. I do not have Bart's special parameter the (the can_queue param) but I am running Bart's 02-mq test over and over looking for a stall. I am running 5 parallel 02-mq runs via fio against 5 mpaths on which I have 5 xfs file systems. So far I have 20 loops of 5 parallel runs and I do not see any stalls or lock-ups. Will be good to know if Bart can test this too in his test bed. Top of Mikes tree is this afb07d0 Merge branch 'block-4.16_dm-4.16' of https://git.kernel.org/pub /scm/linux/kernel/git/snitzer/linux int o block-4.16_dm-4.16 16ce390 Merge branch 'dm-4.16' into block-4.16_dm-4.16 0b943d7 Merge branch 'block-4.16' into block-4.16_dm-4.16 f046ecc blk-mq: introduce BLK_STS_DEV_RESOURCE 4c8cf1c dm mpath selector: more evenly distribute ties 475a055 blk-throttle: use queue_is_rq_based 7923111 dm thin: fix trailing semicolon in __remap_and_issue_shared_cell 1b3c530 dm table: fix NVMe bio-based dm_table_determine_type() validation 2ba27c8 dm: various cleanups to md->queue initialization code e603071 dm mpath: delay the retry of a request if the target responded as busy 316a795 dm mpath: don't call blk_mq_delay_run_hw_queue() in case of BLK_STS_RESOURCE f5ced52 block: Remove kblockd_schedule_delayed_work{,_on}() ae943d2 blk-mq: Avoid that blk_mq_delay_run_hw_queue() introduces unintended delays c77ff7f blk-mq: Rename blk_mq_request_direct_issue() into blk_mq_request_issue_directly() 8c7a8d1 lib/scatterlist: Fix chaining support in sgl_alloc_order() 9b9c63f7 Merge branch 'nvme-4.16' of git://git.infradead.org/nvme into for-4.16/block b889bf6 blk-throttle: track read and write request individually 486e45f IB/srp: Add target_can_queue login parameter a13553c block: add bdev_read_only() checks to common helpers 721c7fc block: fail op_is_write() requests to read-only partitions 17534c6 blk-throttle: export io_serviced_recursive, io_service_bytes_recursive 2c2086a block: Protect less code with sysfs_lock in blk_{un,}register_queue() 14a2349 block: Document scheduler modification locking requirements 83d016a block: Unexport elv_register_queue() and elv_unregister_queue() 8a8747d block, bfq: limit sectors served with interactive weight raising a52a69e block, bfq: limit tags for writes and async I/O 2f53fbe Merge branch 'block-4.16_dm-4.16' of https://git.kernel.org/pub /scm/linux/kernel/git/snitzer/linux int o block-4.16_dm-4.16 7d089f98 Merge branch 'dm-4.16' into block-4.16_dm-4.16 Typical collectl capture looks like this, solid I/O until completion ZERO gaps at 1s interval captures. # <----------Disks-----------> #Time KBRead Reads KBWrit Writes 10:11:50 0 4 333332 20777 10:11:51 0 1 961009 56736 10:11:52 0 0 1400K 61694 10:11:53 0 0 1595K 53154 10:11:54 0 0 1838K 40027 10:11:55 0 0 1736K 39865 10:11:56 0 0 2010K 19867 10:11:57 0 0 1746K 36227 10:11:58 0 0 1256K 104195 10:11:59 0 0 928996 126062 10:12:00 29152 7288 773564 79886 10:12:01 171924 42981 0 0 # <----------Disks-----------> #Time KBRead Reads KBWrit Writes 10:12:02 181500 45375 0 0 10:12:03 213976 53493 0 0 10:12:04 206808 51703 4 1 10:12:05 197260 49315 0 0 10:12:06 207252 51813 0 0 10:12:07 202348 50587 0 0 10:12:08 214412 53603 0 0 10:12:09 210232 52558 0 0 10:12:10 206756 51689 0 0 10:12:11 194708 48677 0 0 10:12:12 188132 47032 0 0 10:12:13 193272 48319 36 5 10:12:14 200372 50093 4 1 10:12:15 200088 50022 0 0 10:12:16 181816 45454 0 0 10:12:17 179116 44779 0 0 10:12:18 182212 45553 0 0 10:12:19 191192 47798 0 0 10:12:20 186580 46649 4 4 10:12:21 191496 47875 1 1 10:12:22 182752 45688 0 0 10:12:23 186272 46567 0 0 # <----------Disks-----------> #Time KBRead Reads KBWrit Writes 10:12:24 188660 47166 8 2 10:12:25 195072 48768 0 0 10:12:26 188696 47174 0 0 10:12:27 195560 48890 61 6 10:12:28 201228 50307 0 0 10:12:29 202776 50694 0 0 10:12:30 204900 51225 0 0 10:12:31 198860 49715 0 0 10:12:32 203500 50875 0 0 10:12:33 207860 51965 0 0 10:12:34 200608 50152 4 1 10:12:35 181780 45445 0 0 10:12:36 202020 50505 0 0 10:12:37 199856 49963 0 0 10:12:38 198744 49687 0 0 10:12:39 200156 50039 0 0 10:12:40 205280 51320 0 0 10:12:41 205308 51327 0 0 10:12:42 200236 50059 0 0 10:12:43 197364 49341 0 0 10:12:44 203864 50966 8 2 10:12:45 198548 49638 0 0 # <----------Disks-----------> #Time KBRead Reads KBWrit Writes 10:12:46 196468 49115 0 0 10:12:47 161360 40341 0 0 10:12:48 155240 38810 0 0 10:12:49 202264 50566 0 0 10:12:50 197448 49361 0 0 10:12:51 191276 47820 80 5 10:12:52 202460 50615 0 0 10:12:53 197224 49306 0 0 10:12:54 189064 47266 4 1 10:12:55 201244 50311 0 0 10:12:56 196640 49160 0 0 10:12:57 190344 47586 72 8 10:12:58 196880 49220 0 0 10:12:59 193896 48474 0 0 10:13:00 178112 44528 0 0 10:13:01 193904 48476 0 0 10:13:02 188084 47021 0 0 10:13:03 187304 46826 0 0 10:13:04 176236 44059 4 1 10:13:05 193848 48462 0 0 10:13:06 189856 47464 0 0 10:13:07 172788 43197 0 0 # <----------Disks-----------> #Time KBRead Reads KBWrit Writes 10:13:08 144608 36152 0 0 10:13:09 133972 33493 0 0 10:13:10 158800 39700 0 0 10:13:11 193416 48354 0 0 10:13:12 196384 49096 0 0 10:13:13 188396 47098 0 0 10:13:14 200520 50131 4 1 10:13:15 195560 48890 0 0 10:13:16 198440 49610 0 0 10:13:17 192260 48065 0 0 10:13:18 200164 50041 0 0 10:13:19 202188 50547 0 0 10:13:20 198104 49526 0 0 10:13:21 181892 45482 29 18 10:13:22 101488 25376 21 8 10:13:23 10004 2503 11 4
