On 5/12/23 19:41, Ming Lin wrote:
On Thu, May 11, 2023 at 11:56 AM Hannes Reinecke <hare@xxxxxxx> wrote:
On 5/11/23 20:41, Ming Lin wrote:
Hi list,
I have an application that needs to use buffered_io to access SMR disk
for good performance.
From "ZBD Support Restrictions" at https://zonedstorage.io/docs/linux/overview
"
Direct IO Writes The kernel page cache does not guarantee that cached
dirty pages will be flushed to a block device in sequential sector
order. This can lead to unaligned write errors if an application uses
buffered writes to write to the sequential write required zones of a
device. To avoid this pitfall, applications that directly use a zoned
block device without a file system should always use direct I/O
operations to write to the sequential write required zones of a
host-managed disk (that is, they should issue write() system calls
with a block device "file open" that uses the O_DIRECT flag).
"
Raw zbd disk only supports direct_io.
Does dm-zoned support buffered io (without O_DIRECT)?
Yes. But I _think_ the above paragraph is ever so slightly outdated, as
we've spent quite a lot of time fixing sequential writes (cf blk-zoned
etc). So while dm-zoned is using bufferet writes there won't be any
sequential write issues.
At least, I have not uncovered any of those during testing.
Hi Hannes,
I use 5.10.90 kernel and smr disk capacity is 24T.
I followed the below guide to create dm_zone device on top of smr disk.
https://zonedstorage.io/docs/linux/dm
Then mkfs.ext4 /dev/dm-0, but it seems hung.
Any ideas?
[37552.217472] device-mapper: uevent: version 1.0.3
[37552.217549] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01)
initialised: dm-devel@xxxxxxxxxx
[37575.608500] device-mapper: zoned metadata: (dmz-5000cca2bfc0db21):
DM-Zoned metadata version 2
[37575.608502] device-mapper: zoned metadata: (sdx): Host-managed
zoned block device
[37575.608503] device-mapper: zoned metadata: (sdx): 50782535680
512-byte logical sectors (offset 0)
[37575.608503] device-mapper: zoned metadata: (sdx): 96860 zones of
524288 512-byte logical sectors (offset 0)
[37575.608504] device-mapper: zoned metadata: (dmz-5000cca2bfc0db21):
96860 zones of 524288 512-byte logical sectors
[37575.609204] device-mapper: zoned: (dmz-5000cca2bfc0db21): Target
device: 50771001344 512-byte logical sectors (6346375168 blocks)
[38101.543353] INFO: task mkfs.ext4:1411791 blocked for more than 122 seconds.
[38101.543380] Tainted: G OE 5.10.90.bm.1-amd64+ #2
[38101.543395] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[38101.543411] task:mkfs.ext4 state:D stack: 0 pid:1411791
ppid:1388660 flags:0x00004000
[38101.543415] Call Trace:
[38101.543422] __schedule+0x3fd/0x760
[38101.543425] schedule+0x46/0xb0
[38101.543426] io_schedule+0x12/0x40
[38101.543429] wait_on_page_bit+0x133/0x270
[38101.543431] ? __page_cache_alloc+0xa0/0xa0
[38101.543432] wait_on_page_writeback+0x25/0x70
[38101.543434] __filemap_fdatawait_range+0x86/0xf0
[38101.543435] file_write_and_wait_range+0x74/0xb0
[38101.543438] blkdev_fsync+0x16/0x40
[38101.543441] do_fsync+0x38/0x60
[38101.543442] __x64_sys_fsync+0x10/0x20
[38101.543445] do_syscall_64+0x2d/0x70
[38101.543446] entry_SYSCALL_64_after_hwframe+0x44/0xa9
===
Below are the steps I did:
root@smr_dev:~# blkzone reset /dev/sdx
root@smr_dev:~# dmzadm --format /dev/sdx
/dev/sdx: 50782535680 512-byte sectors (24215 GiB)
Host-managed device
96860 zones, offset 0
96860 zones of 524288 512-byte sectors (256 MiB)
65536 4KB data blocks per zone
Resetting sequential zones
Writing primary metadata set
Writing mapping table
Writing bitmap blocks
Writing super block to sdx block 0
Writing secondary metadata set
Writing mapping table
Writing bitmap blocks
Writing super block to sdx block 196608
Syncing disk
Done.
Hmm. I don't actually see how many CMR zones the drive has.
root@smr_dev:~# dmzadm --start /dev/sdx
/dev/sdx: 50782535680 512-byte sectors (24215 GiB)
Host-managed device
96860 zones, offset 0
96860 zones of 524288 512-byte sectors (256 MiB)
65536 4KB data blocks per zone
sdx: starting dmz-5000cca2bfc0db21, metadata ver. 2, uuid
7495e21a-23d9-49f4-832a-76b32136078b
root@smr_dev:~# mkfs.ext4 /dev/dm-0
mke2fs 1.44.5 (15-Dec-2018)
Discarding device blocks: done
Creating filesystem with 6346375168 4k blocks and 396648448 inodes
Filesystem UUID: c47de06d-6cf6-4a85-9502-7830ca2f4526
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432, 5804752896
Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information:
===
At another terminal,
root@smr_dev:~# ps aux | grep mkfs.ext4
root 1411791 2.8 0.0 30992 19864 pts/1 D+ 01:30 0:01
mkfs.ext4 /dev/dm-0
root 1413640 0.0 0.0 13972 2496 pts/0 S+ 01:31 0:00
grep mkfs.ext4
root@smr_dev:~# cat /proc/1411791/stack
[<0>] wait_on_page_bit+0x133/0x270
[<0>] wait_on_page_writeback+0x25/0x70
[<0>] __filemap_fdatawait_range+0x86/0xf0
[<0>] file_write_and_wait_range+0x74/0xb0
[<0>] blkdev_fsync+0x16/0x40
[<0>] do_fsync+0x38/0x60
[<0>] __x64_sys_fsync+0x10/0x20
[<0>] do_syscall_64+0x2d/0x70
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
But that just means that we're waiting for I/O to complete; there must
be another thread processing the I/O.
If this is the only active thread in you system something is seriously
hosed.
But I guess I don't need to tell _you_ that :-)
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@xxxxxxx +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
--
dm-devel mailing list
dm-devel@xxxxxxxxxx
https://listman.redhat.com/mailman/listinfo/dm-devel
