On Fri, Aug 21, 2020 at 02:12:32PM +0200, Alberto Garcia wrote: > On Fri 21 Aug 2020 01:42:52 PM CEST, Alberto Garcia wrote: > > On Fri 21 Aug 2020 01:05:06 PM CEST, Brian Foster <bfoster@xxxxxxxxxx> wrote: > >>> > 1) off: for every write request QEMU initializes the cluster (64KB) > >>> > with fallocate(ZERO_RANGE) and then writes the 4KB of data. > >>> > > >>> > 2) off w/o ZERO_RANGE: QEMU writes the 4KB of data and fills the rest > >>> > of the cluster with zeroes. > >>> > > >>> > 3) metadata: all clusters were allocated when the image was created > >>> > but they are sparse, QEMU only writes the 4KB of data. > >>> > > >>> > 4) falloc: all clusters were allocated with fallocate() when the image > >>> > was created, QEMU only writes 4KB of data. > >>> > > >>> > 5) full: all clusters were allocated by writing zeroes to all of them > >>> > when the image was created, QEMU only writes 4KB of data. > >>> > > >>> > As I said in a previous message I'm not familiar with xfs, but the > >>> > parts that I don't understand are > >>> > > >>> > - Why is (4) slower than (1)? > >>> > >>> Because fallocate() is a full IO serialisation barrier at the > >>> filesystem level. If you do: > >>> > >>> fallocate(whole file) > >>> <IO> > >>> <IO> > >>> <IO> > >>> ..... > >>> > >>> The IO can run concurrent and does not serialise against anything in > >>> the filesysetm except unwritten extent conversions at IO completion > >>> (see answer to next question!) > >>> > >>> However, if you just use (4) you get: > >>> > >>> falloc(64k) > >>> <wait for inflight IO to complete> > >>> <allocates 64k as unwritten> > >>> <4k io> > >>> .... > >>> falloc(64k) > >>> <wait for inflight IO to complete> > >>> .... > >>> <4k IO completes, converts 4k to written> > >>> <allocates 64k as unwritten> > >>> <4k io> > >>> falloc(64k) > >>> <wait for inflight IO to complete> > >>> .... > >>> <4k IO completes, converts 4k to written> > >>> <allocates 64k as unwritten> > >>> <4k io> > >>> .... > >>> > >> > >> Option 4 is described above as initial file preallocation whereas > >> option 1 is per 64k cluster prealloc. Prealloc mode mixup aside, Berto > >> is reporting that the initial file preallocation mode is slower than > >> the per cluster prealloc mode. Berto, am I following that right? > > After looking more closely at the data I can see that there is a peak of > ~30K IOPS during the first 5 or 6 seconds and then it suddenly drops to > ~7K for the rest of the test. > > I was running fio with --ramp_time=5 which ignores the first 5 seconds > of data in order to let performance settle, but if I remove that I can > see the effect more clearly. I can observe it with raw files (in 'off' > and 'prealloc' modes) and qcow2 files in 'prealloc' mode. With qcow2 and > preallocation=off the performance is stable during the whole test. > That's interesting. I ran your fio command (without --ramp_time and with --runtime=5m) against a file on XFS (so no qcow2, no zero_range) once with sparse file with a 64k extent size hint and again with a fully preallocated 25GB file and I saw similar results in terms of the delta. This was just against an SSD backed vdisk in my local dev VM, but I saw ~5800 iops for the full preallocation test and ~6200 iops with the extent size hint. I do notice an initial iops burst as described for both tests, so I switched to use a 60s ramp time and 60s runtime. With that longer ramp up time, I see ~5000 iops with the 64k extent size hint and ~5500 iops with the full 25GB prealloc. Perhaps the unexpected performance delta with qcow2 is similarly transient towards the start of the test and the runtime is short enough that it skews the final results..? Brian > Berto >
