On Thu, Jan 16, 2020 at 11:04 AM Jared Walton <jawalking@xxxxxxxxx> wrote:
>
> Correct, I pre-condition for IOPS testing by utilizing the the last if
> block, only using randwrite. which will run random writes for about
> 45min, until a steady state is achieved.
>
> On Thu, Jan 16, 2020 at 11:40 AM Andrey Kuzmin
> <andrey.v.kuzmin@xxxxxxxxx> wrote:
> >
> > On Thu, Jan 16, 2020 at 9:31 PM Jared Walton <jawalking@xxxxxxxxx> wrote:
> > >
> > > Not sure if this will help, but I use the following to prep multiple
> > > 4TB drives at the same time in a little over an hour.
> >
> > You seem to be preconditioning with sequential writes only, and
> > further doing so
> > with essentially single write frontier.
> >
> > That doesn't stress FTL maps enough and doesn't trigger any substantial garbage
> > collection since SSD is intelligent enough to spot sequential write
> > workload with
> > 128K sequential (re)writes.
> >
> > So what you're doing is only good for bandwidth measurements, and if
> > this steady
> > state is applied to random IOPS profiling, you'd be getting highly
> > inflated results.
> >
> > Regards,
> > Andrey
> >
> > > Is it inelegant, yes, but it works for me.
> > >
> > > globalFIOParameters="--offset=0 --ioengine=libaio --invalidate=1
> > > --group_reporting --direct=1 --thread --refill_buffers --norandommap
> > > --randrepeat=0 --allow_mounted_write=1 --output-format=json,normal"
> > >
> > > # Drives should be FOB or LLF'd (if it's good to do that)
> > > # LLF logic
> > >
> > > # 128k Pre-Condition
> > > # Write to entire disk
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > > size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass1of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=write --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > # Read entire disk
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > > size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass2of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=read --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > # Write to entire disk one last time
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > > size=$(fdisk -l | grep ${i} | awk -F "," '{ print $2 }' | awk '{
> > > print $1 }')
> > > ./fio --name=PreconditionPass3of3 --filename=${i} --iodepth=$iodepth
> > > --bs=128k --rw=write --size=${size} --fill_device=1
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > >
> > > # Check 128k steady-state
> > > for i in `ls -1 /dev/nvme*n1`
> > > do
> > > ./fio --name=SteadyState --filename=${i} --iodepth=16 --numjobs=16
> > > --bs=4k --rw=read --ss_dur=1800 --ss=iops_slope:0.3% --runtime=24h
> > > $globalFIOParameters &
> > > done
> > > wait
> > >
> > > On Thu, Jan 16, 2020 at 9:13 AM Mauricio Tavares <raubvogel@xxxxxxxxx> wrote:
> > > >
> > > > On Thu, Jan 16, 2020 at 2:00 AM Andrey Kuzmin <andrey.v.kuzmin@xxxxxxxxx> wrote:
> > > > >
> > > > > On Wed, Jan 15, 2020 at 11:36 PM Mauricio Tavares <raubvogel@xxxxxxxxx> wrote:
> > > > > >
> > > > > > On Wed, Jan 15, 2020 at 2:00 PM Andrey Kuzmin <andrey.v.kuzmin@xxxxxxxxx> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 15, 2020 at 9:29 PM Mauricio Tavares <raubvogel@xxxxxxxxx> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jan 15, 2020 at 1:04 PM Andrey Kuzmin <andrey.v.kuzmin@xxxxxxxxx> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Jan 15, 2020 at 8:29 PM Gruher, Joseph R
> > > > > > > > > <joseph.r.gruher@xxxxxxxxx> wrote:
> > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: fio-owner@xxxxxxxxxxxxxxx <fio-owner@xxxxxxxxxxxxxxx> On Behalf Of
> > > > > > > > > > > Mauricio Tavares
> > > > > > > > > > > Sent: Wednesday, January 15, 2020 7:51 AM
> > > > > > > > > > > To: fio@xxxxxxxxxxxxxxx
> > > > > > > > > > > Subject: CPUs, threads, and speed
> > > > > > > > > > >
> > > > > > > > > > > Let's say I have a config file to preload drive that looks like this (stolen from
> > > > > > > > > > > https://github.com/intel/fiovisualizer/blob/master/Workloads/Precondition/fill
> > > > > > > > > > > _4KRandom_NVMe.ini)
> > > > > > > > > > >
> > > > > > > > > > > [global]
> > > > > > > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > > > ioengine=libaio
> > > > > > > > > > > direct=1
> > > > > > > > > > > bs=4k
> > > > > > > > > > > rw=randwrite
> > > > > > > > > > > iodepth=4
> > > > > > > > > > > numjobs=32
> > > > > > > > > > > buffered=0
> > > > > > > > > > > size=100%
> > > > > > > > > > > loops=2
> > > > > > > > > > > randrepeat=0
> > > > > > > > > > > norandommap
> > > > > > > > > > > refill_buffers
> > > > > > > > > > >
> > > > > > > > > > > [job1]
> > > > > > > > > > >
> > > > > > > > > > > That is taking a ton of time, like days to go. Is there anything I can do to speed it
> > > > > > > > > > > up?
> > > > > > > > > >
> > > > > > > > > > When you say preload, do you just want to write in the full capacity of the drive?
> > > > > > > > >
> > > > > > > > > I believe that preload here means what in SSD world is called drive
> > > > > > > > > preconditioning. It means bringing a fresh drive into steady mode
> > > > > > > > > where it gives you the true performance in production over months of
> > > > > > > > > use rather than the unrealistic fresh drive random write IOPS.
> > > > > > > > >
> > > > > > > > > > A sequential workload with larger blocks will be faster,
> > > > > > > > >
> > > > > > > > > No, you cannot get the job done by sequential writes since it doesn't
> > > > > > > > > populate FTL translation tables like random writes do.
> > > > > > > > >
> > > > > > > > > As to taking a ton, the rule of thumb is to give the SSD 2xcapacity
> > > > > > > > > worth of random writes. At today speeds, that should take just a
> > > > > > > > > couple of hours.
> > > > > > > > >
> > > > > > > > When you say 2xcapacity worth of random writes, do you mean just
> > > > > > > > setting size=200%?
> > > > > > >
> > > > > > > Right.
> > > > > > >
> > > > > > Then I wonder what I am doing wrong now. I changed the config file to
> > > > > >
> > > > > > [root@testbox tests]# cat preload.conf
> > > > > > [global]
> > > > > > name=4k random write 4 ios in the queue in 32 queues
> > > > > > ioengine=libaio
> > > > > > direct=1
> > > > > > bs=4k
> > > > > > rw=randwrite
> > > > > > iodepth=4
> > > > > > numjobs=32
> > > > > > buffered=0
> > > > > > size=200%
> > > > > > loops=2
> > > > > > random_generator=tausworthe64
> > > > > > thread=1
> > > > > >
> > > > > > [job1]
> > > > > > filename=/dev/nvme0n1
> > > > > > [root@testbox tests]#
> > > > > >
> > > > > > but when I run it, now it spits out much larger eta times:
> > > > > >
> > > > > > Jobs: 32 (f=32): [w(32)][0.0%][w=382MiB/s][w=97.7k IOPS][eta
> > > > > > 16580099d:14h:55m:27s]]
> > > > >
> > > > > Size is set on per thread basis, so you're doing 32x200%x2 loops=128
> > > > > drive capacities here.
> > > > >
> > > > > Also, using 32 threads doesn't improve anything. 2 (and even one)
> > > > > threads with qd=128 will push the drive
> > > > > to its limits.
> > > > >
> > > > Update: so I redid the config file a bit to pass some of the
> > > > arguments from command line, and cut down number of jobs and loops.
> > > > And I ran it again, this time sequential write to the drive I have not
> > > > touched to see how fast it was going to go. My eta is still
> > > > astronomical:
> > > >
> > > > [root@testbox tests]# cat preload_fio.conf
> > > > [global]
> > > > name=4k random
> > > > ioengine=${ioengine}
> > > > direct=1
> > > > bs=${bs_size}
> > > > rw=${iotype}
> > > > iodepth=4
> > > > numjobs=1
> > > > buffered=0
> > > > size=200%
> > > > loops=1
> > > >
> > > > [job1]
> > > > filename=${devicename}
> > > > [root@testbox tests]# devicename=/dev/nvme1n1 ioengine=libaio
> > > > iotype=write bs_size=128k ~/dev/fio/fio ./preload_fio.conf
> > > > job1: (g=0): rw=write, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T)
> > > > 128KiB-128KiB, ioengine=libaio, iodepth=4
> > > > fio-3.17-68-g3f1e
> > > > Starting 1 process
> > > > Jobs: 1 (f=1): [W(1)][0.0%][w=1906MiB/s][w=15.2k IOPS][eta 108616d:00h:00m:24s]
> > > >
> > > > > Regards,
> > > > > Andrey
> > > > > >
> > > > > > Compare with what I was getting with size=100%
> > > > > >
> > > > > > Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]
> > > > > >
> > > > > > > Regards,
> > > > > > > Andrey
> > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Andrey
> > > > > > > > >
> > > > > > > > > > like:
> > > > > > > > > >
> > > > > > > > > > [global]
> > > > > > > > > > ioengine=libaio
> > > > > > > > > > thread=1
> > > > > > > > > > direct=1
> > > > > > > > > > bs=128k
> > > > > > > > > > rw=write
> > > > > > > > > > numjobs=1
> > > > > > > > > > iodepth=128
> > > > > > > > > > size=100%
> > > > > > > > > > loops=2
> > > > > > > > > > [job00]
> > > > > > > > > > filename=/dev/nvme0n1
> > > > > > > > > >
> > > > > > > > > > Or if you have a use case where you specifically want to write it in with 4K blocks, you could probably increase your queue depth way beyond 4 and see improvement in performance, and you probably don't want to specify norandommap if you're trying to hit every block on the device.
> > > > > > > > > >
> > > > > > > > > > -Joe
I have pretty much standardized on two sequential drive writes and
four random drive writes to get to steady state. It may be overkill
but has worked well for me since we test a variety of SSDs and some
reach steady state faster than others.
