Thanks for reply. After performing an additional testing with SSD. I have more questions.
Firstly, about the additional testing with SSD:
I tested it with SSD (in Linux software raid level 10 setup). The result shown using dm-integrity is faster than using XFS directly. For using dm-integrity, fio shows lots of I/O merges by the scheduler. Please find the attachment for the result.
Finally, please find the questions below:
1) So after the dm-integrity journal is written to the actual back end storage (hard drive), then fsync would then report completed?
2) To my understanding, for using dm-integrity with journal mode. Data has to written into the storage device twice (one part is the dm-integrity journal, the other one is the actual data). For the fio test, the data should be random and sustained for 60 seconds. But using dm-integrity with journal mode is still faster.
Thanks,
Patrick
On Sat, Jan 4, 2020 at 1:14 AM Mikulas Patocka <mpatocka@xxxxxxxxxx> wrote:
On Fri, 3 Jan 2020, Patrick Dung wrote:
> Hello,
>
> A quick question on dm-integrity. Does dm-integrity layer honors fsync?
Yes it does.
However, it writes data into the journal and when the journal is flushed,
it reports the fsync function as finished.
On a mechanical disk, writes to contiguous space (i.e. the journal) are
faster than random writes all over the disk, that's why you see better
performance with dm-integrity than without it.
Mikulas
> I was testing dm-integrity and performance. It had a strange result that using dm-integrity with journal is faster than a normal file system or dm-integrity with
> bitmap (no journal). fio is used for testing the storage performance. The device is a SATA hard disk drive. Then I created a 100GB partition for testing.
>
> Below is the test cases:
>
> 1) XFS on a partition directly test case
>
> 2) dm-integrity: crc32c on a partition with default setting journal commit interval is 10 seconds. Then create XFS on it. test case
>
> 3) dm-integrity: crc32c on a partition default setting journal commit interval set to 5 seconds. Then create XFS on it.
>
> 4) dm-integrity: crc32c on a partition default setting but using bitmap instead of journal. Then create XFS on it.
>
> FIO command:
>
> fio --filename=./t1 --direct=1 --rw=randrw --refill_buffers --norandommap --randrepeat=0 --ioengine=sync --bs=4k --rwmixread=75 --iodepth=16 --numjobs=8 --runtime=60
> --group_reporting --fsync=1 --name=4ktest --size=4G
>
> Result:
>
> 1. Read/Write IOPS: 117/41. Read/Write Speed 481KB/s 168KB/s
> 2. Read/Write IOPS: 178/59. Read/Write Speed 732KB/s 244KB/s
> 3. Read/Write IOPS: 169/57. Read/Write Speed 695KB/s 236KB/s
> 4. Read/Write IOPS: 97/32. Read/Write Speed 400K/s 131KB/s
> The original discussion in: https://gitlab.com/cryptsetup/cryptsetup/issues/513 . Milan Broz said the dm-devel mailing list is a suitable place to discuss the probem.
>
> Thanks in advance.
>
> Patrick
>
>
Testing with SSD
1) Without dm-integrity
4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16
...
fio-3.14
Starting 8 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020
read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec)
clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33
lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33
clat percentiles (usec):
| 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145],
| 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363],
| 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128],
| 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484],
| 99.99th=[18744]
bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944
iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944
write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets
clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30
lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97],
| 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461],
| 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130],
| 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570],
| 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581],
| 99.95th=[ 17957], 99.99th=[1333789]
bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944
iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944
lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52%
lat (usec) : 1000=0.20%
lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01%
lat (msec) : 500=0.01%, 2000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09
sync percentiles (usec):
| 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195],
| 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997],
| 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765],
| 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695],
| 99.99th=[20579]
cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec
WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec
Disk stats (read/write):
dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70%
sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48%
sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70%
md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36%
sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36%
sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55%
2) With dm-integrity
4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16
...
fio-3.14
Starting 8 processes
4ktest: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s]
4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020
read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec)
clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33
lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33
clat percentiles (usec):
| 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145],
| 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363],
| 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128],
| 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484],
| 99.99th=[18744]
bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944
iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944
write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets
clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30
lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97],
| 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461],
| 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130],
| 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570],
| 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581],
| 99.95th=[ 17957], 99.99th=[1333789]
bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944
iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944
lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52%
lat (usec) : 1000=0.20%
lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01%
lat (msec) : 500=0.01%, 2000=0.01%
fsync/fdatasync/sync_file_range:
sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09
sync percentiles (usec):
| 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195],
| 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997],
| 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765],
| 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695],
| 99.99th=[20579]
cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108
IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec
WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec
Disk stats (read/write):
dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70%
sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48%
sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70%
md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36%
sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36%
sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55%
-- dm-devel mailing list dm-devel@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/dm-devel
