(Sorry for sending this mail again, this one add nbd@xxxxxxxxxxxxxxxx) Hi kernel、ceph comunity: We run into an issue that mainly related to the (kernel) nbd driver and (ceph) rbd-nbd. After some investigations, I found that the root cause of the problem seems to be related to the change in the block size of nbd. I am not sure whether it is the nbd driver or rbd-nbd bug, however there is such a problem. What happened： It will always hang when accessing the mount point of nbd device with ext4 after nbd resized. Environment information: - kernel: v4.19.25 or master - rbd-nbd(ceph): v12.2.0 Luminous or master - the fs of nbd: ext4 Steps to reproduce: 1. rbd create --size 2G rbdpool/foo # create a 2G size rbd image 2. rbd-nbd map rbdpool/foo # map the rbd image as a local block device /dev/nbd0, block size is 512(the default block size is set in rbd-nbd code when nbd mapped). 3. mkfs.ext4 /dev/nbd0 # mkfs.ext4 on nbd0, only nbd + ext4 can reproduce the problem 4. mount /dev/nbd0 /mnt # mount nbd0 on /mnt 5. rbd resize --size 4G rbdpool/foo # expand the nbd backend image from 2G to 4G size 6. ls /mnt # `ls` stuck here forever ln@ubuntu:linux>$ ps -ef |grep mnt root 8670 7519 98 10:16 pts/5 00:28:46 ls --color=auto /mnt/ ln 9508 9293 0 10:45 pts/6 00:00:00 grep --color=auto mnt ln@ubuntu:linux>$ sudo cat /proc/8670/stack [<0>] io_schedule+0x1a/0x40 [<0>] __lock_page+0x105/0x150 [<0>] pagecache_get_page+0x199/0x2c0 [<0>] __getblk_gfp+0xef/0x290 [<0>] ext4_getblk+0x83/0x1a0 [<0>] ext4_bread+0x26/0xb0 [<0>] __ext4_read_dirblock+0x34/0x2c0 [<0>] htree_dirblock_to_tree+0x56/0x1c0 [<0>] ext4_htree_fill_tree+0xad/0x330 [<0>] ext4_readdir+0x6a3/0x980 [<0>] iterate_dir+0x9e/0x1a0 [<0>] ksys_getdents64+0xa0/0x130 [<0>] __x64_sys_getdents64+0x1e/0x30 [<0>] do_syscall_64+0x5e/0x110 [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [<0>] 0xffffffffffffffff Some investigations on the kernel side: By git bisect, I found the problem is related to this commit: https://github.com/torvalds/linux/commit/9a9c3c02eacecf4bfde74b08ed32749a4929a2cf . The kernel with this commit (9a9c3c02) can reproduce the problem, revert the commit and the problem disappears. Some Logical analysis about the nbd block size changing: 1. rbd-nbd map rbdpool/foo => ioctl NBD_BLKSZSET 512 => nbd_size_set() => nbd_size_update(nbd) =>{ bdev = bdget_disk(nbd->disk, 0); bd_set_size(bdev, 512) set_blocksize(bdev, 512) } 2. mkfs.ext4 /dev/nbd0 3. mount /dev/nbd0 /mnt => vfs mount => ext4_mount() => … => sb_set_blocksize() => set_blocksize(bdev, 4096) <= mount ext4 will set the nbd blocksize to 4096 4. rbd resize –size 4G rbdpool/foo => ioctl NBD_SET_SIZE 4G <= rbd-nbd will update the latest total size of nbd device => nbd_size_set() => nbd_size_update(nbd) =>{ bdev = bdget_disk(nbd->disk, 0); bd_set_size(bdev, 512) set_blocksize(bdev, 512) <= the blocksize is set back to 512 [code line: set_blocksize(bdev, config->blksize); ]. It seems to be the root cause. }
