Hi all Thanks for reporting about this. I did a test in my environment. time blkdiscard /dev/nvme5n1 (477GB) real 0m0.398s time blkdiscard /dev/md0 real 9m16.569s I'm not familiar with the block layer codes. I'll try to understand the codes related with discard request and try to fix this problem. I have a question for raid5 discard, it needs to consider more than raid0 and raid10. For example, there is a raid5 with 3 disks. D11 D21 P1 (stripe size is 4KB) D12 D22 P2 D13 D23 P3 D14 D24 P4 ... (chunk size is 512KB) If there is a discard request on D13 and D14, and there is no discard request on D23 D24. It can't send discard request to D13 and D14, right? P3 = D23 xor D13. If we discard D13 and disk2 is broken, it can't get the right data from D13 and P3. The discard request on D13 can write 0 to the discard region, right? If so, it can handle a discard bio at a time that is big enough at least to contain the data. (data disks * chunk size). In this case the size is 1024KB (512KB*2). Regards Xiao On Wed, Jun 9, 2021 at 10:40 AM Wang Shanker <shankerwangmiao@xxxxxxxxx> wrote: > > > > 2021年06月09日 08:41，Ming Lei <ming.lei@xxxxxxxxxx> 写道： > > > > On Tue, Jun 08, 2021 at 11:49:04PM +0800, Wang Shanker wrote: > >> > >> > >> Actually, what are received by the nvme controller are discard requests > >> with 128 segments of 4k, instead of one segment of 512k. > > > > Right, I am just wondering if this way makes a difference wrt. single > > range/segment discard request from device viewpoint, but anyway it is > > better to send less segment. > It would be meaningful if more than queue_max_discard_segments() bio's > are sent and merged into big segments. > > > >> > >>> > >>>> > >>>> Similarly, the problem with scsi devices can be emulated using the following > >>>> options for qemu: > >>>> > >>>> -device virtio-scsi,id=scsi \ > >>>> -device scsi-hd,drive=nvme1,bus=scsi.0,logical_block_size=4096,discard_granularity=2097152,physical_block_size=4096,serial=NVME1 \ > >>>> -device scsi-hd,drive=nvme2,bus=scsi.0,logical_block_size=4096,discard_granularity=2097152,physical_block_size=4096,serial=NVME2 \ > >>>> -device scsi-hd,drive=nvme3,bus=scsi.0,logical_block_size=4096,discard_granularity=2097152,physical_block_size=4096,serial=NVME3 \ > >>>> -trace scsi_disk_emulate_command_UNMAP,file=scsitrace.log > >>>> > >>>> > >>>> Despite the discovery, I cannot come up with a proper fix of this issue due > >>>> to my lack of familiarity of the block subsystem. I expect your kind feedback > >>>> on this. Thanks in advance. > >>> > >>> In the above setting and raid456 test, I observe that rq->nr_phys_segments can > >>> reach 128, but queue_max_discard_segments() reports 256. So discard > >>> request size can be 512KB, which is the max size when you run 1MB discard on > >>> raid456. However, if the discard length on raid456 is increased, the > >>> current way will become inefficient. > >> > >> Exactly. > >> > >> I suggest that bio's can be merged and be calculated as one segment if they are > >> contiguous and contain no data. > > > > Fine. > > > >> > >> And I also discovered later that, even normal long write requests, e.g. > >> a 10m write, will be split into 4k bio's. The maximum number of bio's which can > >> be merged into one request is limited by queue_max_segments, regardless > >> of whether those bio's are contiguous. In my test environment, for scsi devices, > >> queue_max_segments can be 254, which means about 1m size of requests. For nvme > >> devices(e.g. Intel DC P4610), queue_max_segments is only 33 since their mdts is 5, > >> which results in only 132k of requests. > > > > Here what matters is queue_max_discard_segments(). > Here I was considering normal write/read bio's, since I first took it for granted > that normal write/read IOs would be optimal in raid456, and finally discovered > that those 4k IOs can only be merged into not-so-big requests. > > > >> > >> So, I would also suggest that raid456 should be improved to issue bigger bio's to > >> underlying drives. > > > > Right, that should be root solution. > > > > Cc Xiao, I remembered that he worked on this area. > > Many thanks for looking into this issue. > > Cheers, > > Miao Wang >
