[PATCH V2 0/7] null_blk: allow REQ_OP_WRITE_ZEROES and cleanup

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

In order to test the non-trivial I/O path in the block layer for
REQ_OP_WRITE_ZEROES allow write-zeroes on null_blk so we can write
testcases with and without non-memory backed mode followed by
few cleanup patches.

Below is the test report with ext2/ext4 mkfs and a blktest
waiting to get upstream for this patch-series.

-ck

v1->v2:-

* Merge patch 1 and 2 (Damien).
* Change SECT_INX_IN_PAGE->SECT_OFFSET_IN_PAGE. (Damien)
* Add reviewed-by tags. (Damien)
* Add discard/write-zeroes module param to set the respective
  max sector length.

Chaitanya Kulkarni (7):
  null_blk: allow REQ_OP_WRITE_ZEROES
  null_blk: code cleaup
  null_blk: initialize cmd->bio in __alloc_cmd()
  null_blk: don't use magic numbers in the code
  null_blk: remove extra space in switch condition
  null_blk: add param to set max disacrd sectors
  null_blk: add param to set max write-zeroes sects

 drivers/block/null_blk/main.c     | 149 ++++++++++++++++++++++--------
 drivers/block/null_blk/null_blk.h |   3 +
 2 files changed, 116 insertions(+), 36 deletions(-)

linux-block (for-next) # gitlog -8
ab66a9484551 (HEAD -> for-next) null_blk: add param to set max write-zeroes sects
4ad2d08f4ea4 null_blk: add param to set max disacrd sectors
008e19cd64e6 null_blk: remove extra space in switch condition
b92a9bf280bb null_blk: don't use magic numbers in the code
cc56f23fb6bc null_blk: initialize cmd->bio in __alloc_cmd()
36037b231dee null_blk: code cleaup
d23c0fdf1ee2 null_blk: allow REQ_OP_WRITE_ZEROES
86e551dc35c9 Merge branch 'for-next' of git://git.kernel.dk/linux-block into for-next
linux-block (for-next) # ./compile_nullb.sh 
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block modules
  CC [M]  drivers/block/null_blk/main.o
  CC [M]  drivers/block/null_blk/trace.o
  CC [M]  drivers/block/null_blk/zoned.o
  LD [M]  drivers/block/null_blk/null_blk.o
  MODPOST drivers/block/Module.symvers
  LD [M]  drivers/block/null_blk/null_blk.ko
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/6.0.0-rc7blk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/6.0.0-rc7blk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/6.0.0-rc7blk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.2M Oct  5 19:55 /lib/modules/6.0.0-rc7blk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c

blktests (master) # ./check block
block/001 (stress device hotplugging)                        [passed]
    runtime  53.159s  ...  49.759s
block/002 (remove a device while running blktrace)           [passed]
    runtime  0.798s  ...  1.267s
block/006 (run null-blk in blocking mode)                    [passed]
    read iops  8867450  ...  8810329
    runtime    9.669s   ...  9.850s
block/007 => nullb1 (test classic and hybrid IO polling)     [passed]
    auto hybrid poll read iops       619725      ...  615361
    auto hybrid poll read lat mean   1.366826    ...  1.371562
    auto hybrid poll system cpu      76.066351%  ...  74.352941%
    classic poll read iops           618264      ...  615361
    classic poll read lat mean       1.366413    ...  1.371557
    classic poll system cpu          73.995272%  ...  72.470588%
    fixed hybrid poll read iops      625642      ...  622669
    fixed hybrid poll read lat mean  1.352212    ...  1.357977
    fixed hybrid poll system cpu     75.598086%  ...  73.809524%
    no poll read iops                630153      ...  619725
    no poll read lat mean            1.345745    ...  1.362713
    no poll system cpu               73.975904%  ...  72.037915%
    runtime                          2.499s      ...  2.515s
block/009 (check page-cache coherency after BLKDISCARD)      [passed]
    runtime  0.590s  ...  0.560s
block/010 (run I/O on null_blk with shared and non-shared tags) [passed]
    Individual tags read iops  7911915  ...  7883103
    runtime                    21.139s  ...  20.894s
    Shared tags read iops      7837073  ...  7849908
block/014 (run null-blk with blk-mq and timeout injection configured) [passed]
    runtime  86.076s  ...  60.765s
block/015 (run null-blk on different schedulers with requeue injection configured) [passed]
    runtime  44.143s  ...  43.989s
block/016 (send a signal to a process waiting on a frozen queue) [passed]
    runtime  7.085s  ...  7.091s
block/017 (do I/O and check the inflight counter)            [passed]
    runtime  1.703s  ...  1.692s
block/018 (do I/O and check iostats times)                   [passed]
    runtime  5.111s  ...  5.107s
block/020 (run null-blk on different schedulers with only one hardware tag) [passed]
    runtime  41.501s  ...  41.558s
block/021 (read/write nr_requests on null-blk with different schedulers) [passed]
    runtime  5.854s  ...  5.854s
block/022 (Test hang caused by freeze/unfreeze sequence)     [passed]
    runtime  30.137s  ...  30.144s
block/023 (do I/O on all null_blk queue modes)               [passed]
    runtime  0.274s  ...  0.267s
block/024 (do I/O faster than a jiffy and check iostats times) [passed]
    runtime  2.657s  ...  2.660s
block/025 (do a huge discard with 4k sector size)            [passed]
    runtime  3.968s  ...  3.948s
block/027 (stress device hotplugging with running fio jobs and different schedulers) [passed]
    runtime  18.602s  ...  18.900s
block/028 (do I/O on scsi_debug with DIF/DIX enabled)        [passed]
    runtime  13.869s  ...  13.742s
block/029 (trigger blk_mq_update_nr_hw_queues())             [passed]
    runtime  30.283s  ...  30.297s
block/030 (trigger the blk_mq_realloc_hw_ctxs() error path)  [passed]
    runtime  2.142s  ...  2.141s
block/031 (do IO on null-blk with a host tag set)            [passed]
    runtime  30.324s  ...  30.327s


linux-block (for-next) # ./zeroout.sh 1
#################### BLKISZ 512 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
└── zone_size

0 directories, 31 files
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 1b2daa8b-59f3-40d7-9e06-fa9f5347d08f
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[18136.386533] null_blk: disk nullb0 created
[18137.411780] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[18137.412380] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1091MiB/s][r=279k IOPS][eta 00m:05s]                   
write-and-verify: (groupid=0, jobs=1): err= 0: pid=8936: Wed Oct  5 19:55:38 2022
  read: IOPS=278k, BW=1088MiB/s (1141MB/s)(2529MiB/2325msec)
    slat (nsec): min=2164, max=57528, avg=2545.52, stdev=505.53
    clat (nsec): min=1102, max=145345, avg=54147.07, stdev=2689.73
     lat (usec): min=3, max=148, avg=56.74, stdev= 2.76
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   52], 10.00th=[   52], 20.00th=[   53],
     | 30.00th=[   53], 40.00th=[   53], 50.00th=[   54], 60.00th=[   54],
     | 70.00th=[   55], 80.00th=[   55], 90.00th=[   59], 95.00th=[   61],
     | 99.00th=[   63], 99.50th=[   65], 99.90th=[   74], 99.95th=[   83],
     | 99.99th=[  108]
  write: IOPS=78.3k, BW=306MiB/s (321MB/s)(4000MiB/13070msec); 0 zone resets
    slat (usec): min=3, max=486, avg=12.06, stdev=10.45
    clat (nsec): min=781, max=937009, avg=191772.55, stdev=46098.76
     lat (usec): min=7, max=1274, avg=203.90, stdev=47.92
    clat percentiles (usec):
     |  1.00th=[  128],  5.00th=[  143], 10.00th=[  151], 20.00th=[  163],
     | 30.00th=[  172], 40.00th=[  180], 50.00th=[  186], 60.00th=[  194],
     | 70.00th=[  200], 80.00th=[  208], 90.00th=[  219], 95.00th=[  306],
     | 99.00th=[  363], 99.50th=[  383], 99.90th=[  537], 99.95th=[  594],
     | 99.99th=[  701]
   bw (  KiB/s): min=45520, max=368544, per=96.81%, avg=303407.41, stdev=59205.58, samples=27
   iops        : min=11380, max=92136, avg=75851.85, stdev=14801.40, samples=27
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.73%
  lat (usec)   : 250=57.66%, 500=3.50%, 750=0.11%, 1000=0.01%
  cpu          : usr=22.24%, sys=75.62%, ctx=3208, majf=0, minf=15193
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647410,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1088MiB/s (1141MB/s), 1088MiB/s-1088MiB/s (1141MB/s-1141MB/s), io=2529MiB (2652MB), run=2325-2325msec
  WRITE: bw=306MiB/s (321MB/s), 306MiB/s-306MiB/s (321MB/s-321MB/s), io=4000MiB (4194MB), run=13070-13070msec

Disk stats (read/write):
  nullb0: ios=645166/1024000, merge=0/0, ticks=714/1821, in_queue=2534, util=99.36%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
	last mounted on /mnt/nullb0 on Wed Oct  5 19:55:22 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 524cb8db-9768-493b-b6ad-abb1db199bcd
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[18153.154196] EXT4-fs (nullb0): unmounting filesystem.
[18153.182343] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=0): [f(1)][100.0%][r=1043MiB/s][r=267k IOPS][eta 00m:00s]                 
write-and-verify: (groupid=0, jobs=1): err= 0: pid=8951: Wed Oct  5 19:55:51 2022
  read: IOPS=273k, BW=1066MiB/s (1118MB/s)(2528MiB/2370msec)
    slat (nsec): min=2174, max=97034, avg=2606.58, stdev=551.65
    clat (nsec): min=1112, max=151898, avg=55202.08, stdev=3056.22
     lat (usec): min=3, max=155, avg=57.85, stdev= 3.15
    clat percentiles (nsec):
     |  1.00th=[52480],  5.00th=[52992], 10.00th=[52992], 20.00th=[53504],
     | 30.00th=[54016], 40.00th=[54016], 50.00th=[54528], 60.00th=[54528],
     | 70.00th=[55040], 80.00th=[55552], 90.00th=[59136], 95.00th=[61696],
     | 99.00th=[66048], 99.50th=[70144], 99.90th=[79360], 99.95th=[82432],
     | 99.99th=[93696]
  write: IOPS=97.6k, BW=381MiB/s (400MB/s)(4000MiB/10489msec); 0 zone resets
    slat (usec): min=3, max=3626, avg= 9.62, stdev= 9.41
    clat (nsec): min=982, max=3790.8k, avg=153946.33, stdev=38887.01
     lat (usec): min=8, max=3800, avg=163.62, stdev=40.33
    clat percentiles (usec):
     |  1.00th=[  119],  5.00th=[  128], 10.00th=[  133], 20.00th=[  139],
     | 30.00th=[  145], 40.00th=[  149], 50.00th=[  151], 60.00th=[  155],
     | 70.00th=[  159], 80.00th=[  165], 90.00th=[  172], 95.00th=[  180],
     | 99.00th=[  269], 99.50th=[  318], 99.90th=[  351], 99.95th=[  494],
     | 99.99th=[  947]
   bw (  KiB/s): min=357048, max=428968, per=99.90%, avg=390095.24, stdev=23668.21, samples=21
   iops        : min=89262, max=107242, avg=97523.81, stdev=5917.05, samples=21
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=38.72%
  lat (usec)   : 250=60.59%, 500=0.66%, 750=0.02%, 1000=0.01%
  lat (msec)   : 4=0.01%
  cpu          : usr=26.27%, sys=73.56%, ctx=29, majf=0, minf=15182
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647042,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1066MiB/s (1118MB/s), 1066MiB/s-1066MiB/s (1118MB/s-1118MB/s), io=2528MiB (2650MB), run=2370-2370msec
  WRITE: bw=381MiB/s (400MB/s), 381MiB/s-381MiB/s (400MB/s-400MB/s), io=4000MiB (4194MB), run=10489-10489msec

Disk stats (read/write):
  nullb0: ios=587867/1024234, merge=0/5709, ticks=651/1908, in_queue=2559, util=99.27%




#################### BLKISZ 1024 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
└── zone_size

0 directories, 31 files
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 66fa20c2-4bd7-4dd5-9002-8f832b198068
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[18166.870587] null_blk: disk nullb0 created
[18167.892430] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[18167.893033] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][71.4%][r=1127MiB/s][r=289k IOPS][eta 00m:06s]                  
write-and-verify: (groupid=0, jobs=1): err= 0: pid=8976: Wed Oct  5 19:56:09 2022
  read: IOPS=288k, BW=1123MiB/s (1178MB/s)(2528MiB/2251msec)
    slat (nsec): min=2014, max=92396, avg=2422.41, stdev=512.19
    clat (nsec): min=1303, max=167628, avg=52433.36, stdev=2464.96
     lat (usec): min=3, max=170, avg=54.90, stdev= 2.53
    clat percentiles (usec):
     |  1.00th=[   50],  5.00th=[   51], 10.00th=[   51], 20.00th=[   51],
     | 30.00th=[   52], 40.00th=[   52], 50.00th=[   52], 60.00th=[   52],
     | 70.00th=[   53], 80.00th=[   53], 90.00th=[   57], 95.00th=[   59],
     | 99.00th=[   61], 99.50th=[   62], 99.90th=[   67], 99.95th=[   69],
     | 99.99th=[  106]
  write: IOPS=75.7k, BW=296MiB/s (310MB/s)(4000MiB/13530msec); 0 zone resets
    slat (usec): min=3, max=466, avg=12.47, stdev=10.01
    clat (nsec): min=1052, max=903526, avg=198530.45, stdev=47804.47
     lat (usec): min=27, max=1064, avg=211.08, stdev=49.81
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  147], 10.00th=[  157], 20.00th=[  167],
     | 30.00th=[  178], 40.00th=[  186], 50.00th=[  194], 60.00th=[  200],
     | 70.00th=[  206], 80.00th=[  215], 90.00th=[  229], 95.00th=[  314],
     | 99.00th=[  375], 99.50th=[  404], 99.90th=[  562], 99.95th=[  611],
     | 99.99th=[  701]
   bw (  KiB/s): min=16080, max=355544, per=96.64%, avg=292571.43, stdev=60750.67, samples=28
   iops        : min= 4020, max=88886, avg=73142.86, stdev=15187.66, samples=28
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.50%, 100=38.23%
  lat (usec)   : 250=57.21%, 500=3.95%, 750=0.11%, 1000=0.01%
  cpu          : usr=23.21%, sys=74.54%, ctx=3346, majf=0, minf=15186
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647231,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1123MiB/s (1178MB/s), 1123MiB/s-1123MiB/s (1178MB/s-1178MB/s), io=2528MiB (2651MB), run=2251-2251msec
  WRITE: bw=296MiB/s (310MB/s), 296MiB/s-296MiB/s (310MB/s-310MB/s), io=4000MiB (4194MB), run=13530-13530msec

Disk stats (read/write):
  nullb0: ios=605621/1024000, merge=0/0, ticks=602/1664, in_queue=2266, util=99.36%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
	last mounted on /mnt/nullb0 on Wed Oct  5 19:55:53 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: eeac79fa-2a6f-497c-a04a-e22e48867b4a
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[18184.019532] EXT4-fs (nullb0): unmounting filesystem.
[18184.043893] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1123MiB/s][r=287k IOPS][eta 00m:04s]                   
write-and-verify: (groupid=0, jobs=1): err= 0: pid=8990: Wed Oct  5 19:56:22 2022
  read: IOPS=287k, BW=1119MiB/s (1174MB/s)(2530MiB/2260msec)
    slat (nsec): min=2024, max=94349, avg=2444.00, stdev=532.18
    clat (nsec): min=1383, max=148101, avg=52612.45, stdev=2521.26
     lat (usec): min=3, max=150, avg=55.10, stdev= 2.58
    clat percentiles (nsec):
     |  1.00th=[50432],  5.00th=[50432], 10.00th=[50944], 20.00th=[51456],
     | 30.00th=[51456], 40.00th=[51456], 50.00th=[51968], 60.00th=[51968],
     | 70.00th=[52480], 80.00th=[52992], 90.00th=[56064], 95.00th=[58624],
     | 99.00th=[60672], 99.50th=[61696], 99.90th=[66048], 99.95th=[76288],
     | 99.99th=[98816]
  write: IOPS=101k, BW=396MiB/s (415MB/s)(4000MiB/10107msec); 0 zone resets
    slat (usec): min=3, max=3934, avg= 9.24, stdev= 8.67
    clat (nsec): min=792, max=7280.8k, avg=148351.59, stdev=40152.32
     lat (usec): min=6, max=7292, avg=157.65, stdev=41.65
    clat percentiles (usec):
     |  1.00th=[  114],  5.00th=[  123], 10.00th=[  128], 20.00th=[  135],
     | 30.00th=[  139], 40.00th=[  143], 50.00th=[  147], 60.00th=[  151],
     | 70.00th=[  155], 80.00th=[  161], 90.00th=[  167], 95.00th=[  176],
     | 99.00th=[  202], 99.50th=[  223], 99.90th=[  277], 99.95th=[  433],
     | 99.99th=[  486]
   bw (  KiB/s): min=93312, max=450696, per=96.26%, avg=390095.24, stdev=73136.38, samples=21
   iops        : min=23328, max=112674, avg=97523.81, stdev=18284.09, samples=21
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.16%, 100=38.59%
  lat (usec)   : 250=61.09%, 500=0.16%, 750=0.01%
  lat (msec)   : 4=0.01%, 10=0.01%
  cpu          : usr=27.71%, sys=72.12%, ctx=29, majf=0, minf=15196
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647570,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1119MiB/s (1174MB/s), 1119MiB/s-1119MiB/s (1174MB/s-1174MB/s), io=2530MiB (2652MB), run=2260-2260msec
  WRITE: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=4000MiB (4194MB), run=10107-10107msec

Disk stats (read/write):
  nullb0: ios=583689/1024237, merge=0/5798, ticks=571/1672, in_queue=2244, util=99.23%




#################### BLKISZ 2048 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
└── zone_size

0 directories, 31 files
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 95af17f3-1251-4eb3-bb08-ef59d64de711
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[18197.234676] null_blk: disk nullb0 created
[18198.256145] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[18198.256835] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][75.0%][r=1176MiB/s][r=301k IOPS][eta 00m:05s]                  
write-and-verify: (groupid=0, jobs=1): err= 0: pid=9014: Wed Oct  5 19:56:39 2022
  read: IOPS=300k, BW=1172MiB/s (1229MB/s)(2526MiB/2156msec)
    slat (nsec): min=1883, max=95211, avg=2289.83, stdev=538.98
    clat (nsec): min=1102, max=144034, avg=50260.51, stdev=3111.72
     lat (usec): min=3, max=146, avg=52.59, stdev= 3.20
    clat percentiles (usec):
     |  1.00th=[   48],  5.00th=[   49], 10.00th=[   49], 20.00th=[   49],
     | 30.00th=[   49], 40.00th=[   49], 50.00th=[   50], 60.00th=[   50],
     | 70.00th=[   50], 80.00th=[   51], 90.00th=[   55], 95.00th=[   57],
     | 99.00th=[   62], 99.50th=[   66], 99.90th=[   79], 99.95th=[   84],
     | 99.99th=[  104]
  write: IOPS=77.2k, BW=301MiB/s (316MB/s)(4000MiB/13272msec); 0 zone resets
    slat (usec): min=3, max=392, avg=12.21, stdev= 9.32
    clat (nsec): min=792, max=996792, avg=194754.81, stdev=45154.41
     lat (usec): min=6, max=1032, avg=207.04, stdev=47.09
    clat percentiles (usec):
     |  1.00th=[  127],  5.00th=[  143], 10.00th=[  151], 20.00th=[  163],
     | 30.00th=[  174], 40.00th=[  182], 50.00th=[  190], 60.00th=[  198],
     | 70.00th=[  206], 80.00th=[  215], 90.00th=[  227], 95.00th=[  289],
     | 99.00th=[  363], 99.50th=[  375], 99.90th=[  529], 99.95th=[  619],
     | 99.99th=[  725]
   bw (  KiB/s): min=192784, max=369664, per=98.30%, avg=303385.96, stdev=40667.67, samples=27
   iops        : min=48196, max=92416, avg=75846.48, stdev=10166.92, samples=27
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=28.00%, 100=10.71%
  lat (usec)   : 250=57.62%, 500=3.59%, 750=0.08%, 1000=0.01%
  cpu          : usr=23.52%, sys=74.41%, ctx=3289, majf=0, minf=15175
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=646749,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1172MiB/s (1229MB/s), 1172MiB/s-1172MiB/s (1229MB/s-1229MB/s), io=2526MiB (2649MB), run=2156-2156msec
  WRITE: bw=301MiB/s (316MB/s), 301MiB/s-301MiB/s (316MB/s-316MB/s), io=4000MiB (4194MB), run=13272-13272msec

Disk stats (read/write):
  nullb0: ios=634659/1024000, merge=0/0, ticks=558/1600, in_queue=2157, util=99.38%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
	last mounted on /mnt/nullb0 on Wed Oct  5 19:56:23 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: b3f69fcb-abd2-41eb-aaf5-621503259a75
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[18214.019199] EXT4-fs (nullb0): unmounting filesystem.
[18214.041644] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][80.0%][r=1168MiB/s][r=299k IOPS][eta 00m:03s]                   
write-and-verify: (groupid=0, jobs=1): err= 0: pid=9027: Wed Oct  5 19:56:52 2022
  read: IOPS=297k, BW=1161MiB/s (1217MB/s)(2528MiB/2178msec)
    slat (nsec): min=1903, max=61776, avg=2321.68, stdev=544.98
    clat (nsec): min=1102, max=111392, avg=50739.27, stdev=3167.99
     lat (usec): min=3, max=113, avg=53.10, stdev= 3.27
    clat percentiles (usec):
     |  1.00th=[   48],  5.00th=[   49], 10.00th=[   49], 20.00th=[   49],
     | 30.00th=[   50], 40.00th=[   50], 50.00th=[   50], 60.00th=[   50],
     | 70.00th=[   51], 80.00th=[   51], 90.00th=[   55], 95.00th=[   58],
     | 99.00th=[   65], 99.50th=[   71], 99.90th=[   77], 99.95th=[   80],
     | 99.99th=[  104]
  write: IOPS=101k, BW=396MiB/s (415MB/s)(4000MiB/10105msec); 0 zone resets
    slat (usec): min=3, max=3540, avg= 9.23, stdev= 8.10
    clat (nsec): min=831, max=3715.1k, avg=148313.45, stdev=34493.81
     lat (usec): min=17, max=3722, avg=157.60, stdev=35.79
    clat percentiles (usec):
     |  1.00th=[  113],  5.00th=[  122], 10.00th=[  127], 20.00th=[  135],
     | 30.00th=[  139], 40.00th=[  143], 50.00th=[  147], 60.00th=[  151],
     | 70.00th=[  155], 80.00th=[  161], 90.00th=[  169], 95.00th=[  178],
     | 99.00th=[  215], 99.50th=[  243], 99.90th=[  281], 99.95th=[  334],
     | 99.99th=[  404]
   bw (  KiB/s): min=89824, max=454648, per=96.24%, avg=390095.24, stdev=74144.52, samples=21
   iops        : min=22456, max=113662, avg=97523.81, stdev=18536.13, samples=21
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=21.93%, 100=16.81%
  lat (usec)   : 250=61.03%, 500=0.22%
  lat (msec)   : 4=0.01%
  cpu          : usr=28.16%, sys=71.64%, ctx=191, majf=0, minf=15190
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647285,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1161MiB/s (1217MB/s), 1161MiB/s-1161MiB/s (1217MB/s-1217MB/s), io=2528MiB (2651MB), run=2178-2178msec
  WRITE: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=4000MiB (4194MB), run=10105-10105msec

Disk stats (read/write):
  nullb0: ios=606388/1024235, merge=0/5718, ticks=528/1602, in_queue=2130, util=99.24%




#################### BLKISZ 4096 #####################
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_discard_sectors
├── max_sectors
├── max_write_zeroes_sectors
├── mbps
├── memory_backed
├── no_sched
├── poll_queues
├── power
├── queue_mode
├── shared_tag_bitmap
├── size
├── submit_queues
├── use_per_node_hctx
├── virt_boundary
├── write_zeroes
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
└── zone_size

0 directories, 31 files
MKFS:-
mke2fs 1.45.6 (20-Mar-2020)
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 90636218-f980-4d5f-a306-d0b38fd30e92
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext2 (rw,relatime,seclabel)
[18227.145657] null_blk: disk nullb0 created
[18228.166279] EXT4-fs (nullb0): mounting ext2 file system using the ext4 subsystem
[18228.166990] EXT4-fs (nullb0): mounted filesystem without journal. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][78.9%][r=1174MiB/s][r=301k IOPS][eta 00m:04s]                   
write-and-verify: (groupid=0, jobs=1): err= 0: pid=9053: Wed Oct  5 19:57:08 2022
  read: IOPS=298k, BW=1165MiB/s (1221MB/s)(2530MiB/2172msec)
    slat (nsec): min=1923, max=112183, avg=2305.07, stdev=500.91
    clat (nsec): min=1342, max=256057, avg=50578.91, stdev=2797.70
     lat (usec): min=3, max=261, avg=52.93, stdev= 2.88
    clat percentiles (usec):
     |  1.00th=[   48],  5.00th=[   49], 10.00th=[   49], 20.00th=[   49],
     | 30.00th=[   50], 40.00th=[   50], 50.00th=[   50], 60.00th=[   50],
     | 70.00th=[   51], 80.00th=[   51], 90.00th=[   55], 95.00th=[   57],
     | 99.00th=[   60], 99.50th=[   61], 99.90th=[   68], 99.95th=[   81],
     | 99.99th=[  109]
  write: IOPS=79.2k, BW=309MiB/s (325MB/s)(4000MiB/12925msec); 0 zone resets
    slat (usec): min=3, max=527, avg=11.92, stdev= 9.64
    clat (nsec): min=942, max=974140, avg=189663.16, stdev=44574.80
     lat (usec): min=8, max=1353, avg=201.65, stdev=46.43
    clat percentiles (usec):
     |  1.00th=[  127],  5.00th=[  143], 10.00th=[  151], 20.00th=[  161],
     | 30.00th=[  169], 40.00th=[  178], 50.00th=[  186], 60.00th=[  192],
     | 70.00th=[  198], 80.00th=[  206], 90.00th=[  219], 95.00th=[  273],
     | 99.00th=[  363], 99.50th=[  400], 99.90th=[  545], 99.95th=[  611],
     | 99.99th=[  807]
   bw (  KiB/s): min=251544, max=368624, per=99.42%, avg=315076.92, stdev=30985.16, samples=26
   iops        : min=62886, max=92156, avg=78769.23, stdev=7746.29, samples=26
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=21.02%, 100=17.72%
  lat (usec)   : 250=58.00%, 500=3.16%, 750=0.09%, 1000=0.01%
  cpu          : usr=23.65%, sys=74.12%, ctx=3203, majf=0, minf=15194
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647556,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1165MiB/s (1221MB/s), 1165MiB/s-1165MiB/s (1221MB/s-1221MB/s), io=2530MiB (2652MB), run=2172-2172msec
  WRITE: bw=309MiB/s (325MB/s), 309MiB/s-309MiB/s (325MB/s-325MB/s), io=4000MiB (4194MB), run=12925-12925msec

Disk stats (read/write):
  nullb0: ios=583450/1024000, merge=0/0, ticks=517/1562, in_queue=2079, util=99.33%
mke2fs 1.45.6 (20-Mar-2020)
/dev/nullb0 contains a ext2 file system
	last mounted on /mnt/nullb0 on Wed Oct  5 19:56:53 2022
Creating filesystem with 1310720 4k blocks and 327680 inodes
Filesystem UUID: 03970406-594a-4d78-b966-d6dbdc697b16
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

/dev/nullb0 on /mnt/nullb0 type ext4 (rw,relatime,seclabel)
[18243.612194] EXT4-fs (nullb0): unmounting filesystem.
[18243.634860] EXT4-fs (nullb0): mounted filesystem with ordered data mode. Quota mode: none.
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
fio-3.27
Starting 1 process
write-and-verify: Laying out IO file (1 file / 4000MiB)
Jobs: 1 (f=1): [V(1)][80.0%][r=1161MiB/s][r=297k IOPS][eta 00m:03s]                   
write-and-verify: (groupid=0, jobs=1): err= 0: pid=9066: Wed Oct  5 19:57:21 2022
  read: IOPS=295k, BW=1151MiB/s (1207MB/s)(2528MiB/2196msec)
    slat (nsec): min=1944, max=76366, avg=2349.04, stdev=512.68
    clat (nsec): min=1092, max=132292, avg=51163.41, stdev=2525.10
     lat (usec): min=3, max=134, avg=53.56, stdev= 2.60
    clat percentiles (usec):
     |  1.00th=[   49],  5.00th=[   49], 10.00th=[   49], 20.00th=[   50],
     | 30.00th=[   50], 40.00th=[   51], 50.00th=[   51], 60.00th=[   51],
     | 70.00th=[   52], 80.00th=[   52], 90.00th=[   55], 95.00th=[   58],
     | 99.00th=[   60], 99.50th=[   61], 99.90th=[   65], 99.95th=[   68],
     | 99.99th=[  109]
  write: IOPS=101k, BW=396MiB/s (416MB/s)(4000MiB/10093msec); 0 zone resets
    slat (usec): min=3, max=3218, avg= 9.23, stdev= 7.92
    clat (nsec): min=772, max=3407.3k, avg=148142.35, stdev=32842.67
     lat (usec): min=6, max=3419, avg=157.43, stdev=34.05
    clat percentiles (usec):
     |  1.00th=[  114],  5.00th=[  123], 10.00th=[  128], 20.00th=[  135],
     | 30.00th=[  139], 40.00th=[  143], 50.00th=[  147], 60.00th=[  151],
     | 70.00th=[  155], 80.00th=[  161], 90.00th=[  167], 95.00th=[  176],
     | 99.00th=[  198], 99.50th=[  210], 99.90th=[  273], 99.95th=[  338],
     | 99.99th=[  457]
   bw (  KiB/s): min=78848, max=452720, per=96.12%, avg=390095.24, stdev=76125.82, samples=21
   iops        : min=19712, max=113180, avg=97523.81, stdev=19031.45, samples=21
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.01%, 50=12.29%, 100=26.44%
  lat (usec)   : 250=61.14%, 500=0.13%
  lat (msec)   : 4=0.01%
  cpu          : usr=27.40%, sys=72.43%, ctx=28, majf=0, minf=15187
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=647188,1024000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1151MiB/s (1207MB/s), 1151MiB/s-1151MiB/s (1207MB/s-1207MB/s), io=2528MiB (2651MB), run=2196-2196msec
  WRITE: bw=396MiB/s (416MB/s), 396MiB/s-396MiB/s (416MB/s-416MB/s), io=4000MiB (4194MB), run=10093-10093msec

Disk stats (read/write):
  nullb0: ios=604283/1024239, merge=0/5858, ticks=528/1597, in_queue=2125, util=99.24%



* Without max discard and write-zeroes set :-

linux-block (for-next) # for i in `ls /sys/block/nullb0/queue/*discard*`;
do echo -n $i:   ;cat $i; echo "";  done 
/sys/block/nullb0/queue/discard_granularity:4096

/sys/block/nullb0/queue/discard_max_bytes:4294966784

/sys/block/nullb0/queue/discard_max_hw_bytes:4294966784

/sys/block/nullb0/queue/discard_zeroes_data:0

/sys/block/nullb0/queue/max_discard_segments:1

linux-block (for-next) # for i in `ls /sys/block/nullb0/queue/*zeroes*`;
do echo -n $i:   ;cat $i; echo "";  done 
/sys/block/nullb0/queue/discard_zeroes_data:0

/sys/block/nullb0/queue/write_zeroes_max_bytes:4294966784
* With max discard and write-zeroes set to 1  :-

linux-block (for-next) # for i in `ls /sys/block/nullb0/queue/*discard*`;
do echo -n $i:   ;cat $i; echo "";  done 
/sys/block/nullb0/queue/discard_granularity:4096

/sys/block/nullb0/queue/discard_max_bytes:512

/sys/block/nullb0/queue/discard_max_hw_bytes:512

/sys/block/nullb0/queue/discard_zeroes_data:0

/sys/block/nullb0/queue/max_discard_segments:1

linux-block (for-next) # for i in `ls /sys/block/nullb0/queue/*zeroes*`; do echo -n $i:   ;cat $i; echo "";  done /sys/block/nullb0/queue/discard_zeroes_data:0

/sys/block/nullb0/queue/write_zeroes_max_bytes:512

linux-block (for-next) # 





-- 
2.29.0




[Index of Archives]     [Linux RAID]     [Linux SCSI]     [Linux ATA RAID]     [IDE]     [Linux Wireless]     [Linux Kernel]     [ATH6KL]     [Linux Bluetooth]     [Linux Netdev]     [Kernel Newbies]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Device Mapper]

  Powered by Linux