-----Original Message----- From: fio-owner@xxxxxxxxxxxxxxx [mailto:fio-owner@xxxxxxxxxxxxxxx] On Behalf Of Jens Axboe Sent: Sunday, September 16, 2018 8:42 PM To: smitha sunder <sundersmitha@xxxxxxxxx> Cc: Sitsofe Wheeler <sitsofe@xxxxxxxxx>; fio <fio@xxxxxxxxxxxxxxx> Subject: Re: No I/O performed by windowsaio/get_io_u: zero buflen On 9/16/18 9:34 PM, smitha sunder wrote: > > >> On Sep 16, 2018, at 8:31 PM, Jens Axboe <axboe@xxxxxxxxx> wrote: >> >>> On 9/16/18 9:13 PM, smitha sunder wrote: >>> >>> >>>>> On Sep 16, 2018, at 8:02 PM, Jens Axboe <axboe@xxxxxxxxx> wrote: >>>>> >>>>>> On 9/16/18 5:24 PM, smitha sunder wrote: >>>>>>> On Sun, Sep 16, 2018 at 3:02 PM, Sitsofe Wheeler <sitsofe@xxxxxxxxx> wrote: >>>>>>> On Sat, 15 Sep 2018 at 23:14, smitha sunder <sundersmitha@xxxxxxxxx> wrote: >>>>>>> >>>>>>> Hello all, >>>>>>> >>>>>>> I have a 30TB drive and I am running into an issue with random writes. >>>>>>> I went through this thread : >>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics >>>>>>> .net_lists_fio_msg06294.html&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=6QDNQBLcbJUsiXnq84y2HKG98aEmhvabF7uK-yKTbhk&e= that seems to be fixed already. >>>>>> >>>>>> I think that was something different regarding different >>>>>> blocksizes per direction. >>>>>> >>>>>>> I see the same issue with random reads as well. >>>>>>> So not sure what is the issue here in my case; any help is greatly appreciated. >>>>>>> >>>>>>> >>>>>>> Read Capacity results: >>>>>>> Protection: prot_en=1, p_type=1, p_i_exponent=0 [type 2 >>>>>>> protection] Logical block provisioning: lbpme=1, lbprz=1 Last >>>>>>> logical block address=58781073407 (0xdaf9fffff), Number of >>>>>>> logical blocks=58781073408 Logical block length=512 bytes >>>>>>> Logical blocks per physical block exponent=3 [so physical block >>>>>>> length=4096 bytes] >>>>>>> Lowest aligned logical block address=0 >>>>>>> Hence: >>>>>>> Device size: 30095909584896 bytes, 2.87017e+007 MiB, 30095.9 GB >>>>>>> >>>>>>> >>>>>>> C:\Program Files (x86)\fio>fio --ioengine=windowsaio >>>>>>> --group_reporting >>>>>>> --direct=1 --size=100% --bs=4K --thread >>>>>>> --filename=\\.\PhysicalDrive1 --name=precond --rw=randwrite >>>>>>> --iodepth=1 --numjobs=1 --debug=io,random >>>>>> >>>>>> <snip> >>>>>> >>>>>>> io 3372 fill: io_u 0A458780: >>>>>>> off=0x144365e7d000,len=0x0,ddir=1,file=\\.\PhysicalDrive1 >>>>>>> io 3372 get_io_u: zero buflen on 0A458780 >>>>>>> io 3372 get_io_u failed >>>>>>> io 3372 drop page cache \\.\PhysicalDrive1 >>>>>>> random 3372 off rand 17311067694306724737 >>>>>> >>>>>> That offset is crazy big - I'm sure it's bigger than a petabyte >>>>>> so my guess is that something is overflowing. If you use >>>>>> --size=27g does the job go through? >>>>>> >>>>>> [...] >>>>>> >>>>>>> I don’t see this issue if I use bs=8K or I use ba=512,8K, etc. >>>>>> >>>>>> -- >>>>>> Sitsofe | >>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__sucs.org_-7Es >>>>>> its_&d=DwIDaQ&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgR >>>>>> BDyeV4hVQQBEgL2&r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=N >>>>>> aPx5NYD-_SBHGUzJNi4czQueVX-2LvLGD_SEwee41c&s=ZnZsmte_bg5yC3adbwXI >>>>>> U4EmuEn97cA9o1YpZxDU4m0&e= >>>>> Hi Sitsofe, >>>>> >>>>> Thanks for the reply! >>>>> >>>>> Yes; If I use --size=27G or if provide the exact size that the OS >>>>> displays, then the job goes through. >>>> >>>> Are you running a 32-bit or 64-bit build of fio? >>>> >>>> -- >>>> Jens Axboe >>>> >>> 32 bit. >> >> I thought so. I see a few 32-bit issues with huge devices. One of >> them is this one: >> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__git.kernel.dk_cgi >> t_fio_commit_-3Fid-3D604d3f5bd9f2b985568593c23f8292cbc7f4044c&d=DwIDa >> Q&c=JfeWlBa6VbDyTXraMENjy_b_0yKWuqQ4qY-FPhxK4x8w-TfgRBDyeV4hVQQBEgL2& >> r=fLVkx46f6GJPZippnZkCtH9m_d1P5HX6rKxJgT7l7Us&m=NaPx5NYD-_SBHGUzJNi4c >> zQueVX-2LvLGD_SEwee41c&s=AJCs8E23SmJj-z7BbkuP_yxTNDg3g6BZ8FYIcV3EmAY& >> e= >> >> but I'm sure there are others, I'll try and reproduce and get this fixed. >> >> -- >> Jens Axboe >> > I see the same issue even with 64 bit fio build. The difference is that an unsigned long on 64-bit windows builds is still 32-bit, whereas it's 64-bit on linux. So the easiest for me is to test on 32-bit linux, which _probably_ then hits the same thing... -- Jens Axboe Jens, Got the latest build from https://ci.appveyor.com/project/axboe/fio/build/1.0.891/job/awu51avmepn1kvgf/artifacts. With size=100%, bs=4K , I no longer see the issue; thanks for the fix! precond: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=windowsaio, iodepth=32 fio-3.10 Starting 1 thread fio: file \\.\PhysicalDrive1 exceeds 32-bit tausworthe random generator. fio: Switching to tausworthe64. Use the random_generator= option to get rid of this warning. precond: (groupid=0, jobs=1): err= 0: pid=2164: Mon Sep 17 09:30:52 2018 write: IOPS=126k, BW=491MiB/s (515MB/s)(28.8GiB/60001msec) slat (nsec): min=1283, max=234980, avg=2264.27, stdev=1230.37 clat (usec): min=55, max=33282, avg=240.68, stdev=46.86 lat (usec): min=57, max=33295, avg=242.94, stdev=46.91 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 184], 10.00th=[ 194], 20.00th=[ 206], | 30.00th=[ 217], 40.00th=[ 227], 50.00th=[ 235], 60.00th=[ 245], | 70.00th=[ 258], 80.00th=[ 273], 90.00th=[ 297], 95.00th=[ 318], | 99.00th=[ 367], 99.50th=[ 388], 99.90th=[ 461], 99.95th=[ 611], | 99.99th=[ 1172] bw ( KiB/s): min=470928, max=608768, per=100.00%, avg=503204.03, stdev=36189.78, samples=119 iops : min=117732, max=152192, avg=125801.01, stdev=9047.44, samples=119 lat (usec) : 100=0.01%, 250=64.69%, 500=35.24%, 750=0.04%, 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 50=0.01% cpu : usr=8.33%, sys=20.00%, ctx=0, majf=0, minf=0 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=81.1%, 32=18.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=92.8%, 8=3.7%, 16=3.3%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=0,7545152,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: bw=491MiB/s (515MB/s), 491MiB/s-491MiB/s (515MB/s-515MB/s), io=28.8GiB (30.9GB), run=60001-60001msec Thanks smitha
