Are you using krbd directly within the VM or librbd via virtio-blk/scsi? Ticket #9071 is against krbd. On Mon, Jan 16, 2017 at 11:34 AM, Vincent Godin <vince.mlist@xxxxxxxxx> wrote: > In fact, we can reproduce the problem from VM with CentOS 6.7, 7.2 or 7.3. > We can reproduce it each time with this config : one VM (here in CentOS 6.7) > with 16 RBD volumes of 100GB attached. When we launch in serial mkfs.ext4 on > each of these volumes, we allways encounter the problem on one of them. We > tried with the option -E nodiscard but we still have the problem. It' look > exactly like the bug #9071 with the same dmesg message : > > vdh: unknown partition table > EXT4-fs (vdf): mounted filesystem with ordered data mode. Opts: > EXT4-fs (vdg): mounted filesystem with ordered data mode. Opts: > INFO: task flush-252:112:2903 blocked for more than 120 seconds. > Not tainted 2.6.32-573.18.1.el6.x86_64 #1 > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > flush-252:112 D 0000000000000000 0 2903 2 0x00000080 > ffff8808328bf6e0 0000000000000046 ffff8808ffffffff 000000003d697f73 > 0000000000000000 ffff88082fbd7ec0 0000000000021454 ffffffffa78356ec > 000000002b9db4fe ffffffff81aa6700 ffff88082efc9ad8 ffff8808328bffd8 > Call Trace: > [<ffffffff81539673>] io_schedule+0x73/0xc0 > [<ffffffff81276598>] get_request_wait+0x108/0x1d0 > [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 > [<ffffffff812766f9>] blk_queue_bio+0x99/0x610 > [<ffffffff81274ec0>] generic_make_request+0x240/0x5a0 > [<ffffffff81129cf5>] ? mempool_alloc_slab+0x15/0x20 > [<ffffffff81129e93>] ? mempool_alloc+0x63/0x140 > [<ffffffff81275290>] submit_bio+0x70/0x120 > [<ffffffff811c7dcd>] submit_bh+0x11d/0x1f0 > [<ffffffff811ca588>] __block_write_full_page+0x1c8/0x330 > [<ffffffff811c9550>] ? end_buffer_async_write+0x0/0x190 > [<ffffffff811ce450>] ? blkdev_get_block+0x0/0x20 > [<ffffffff811ce450>] ? blkdev_get_block+0x0/0x20 > [<ffffffff811ca7d0>] block_write_full_page_endio+0xe0/0x120 > [<ffffffff81126ff0>] ? find_get_pages_tag+0x40/0x130 > [<ffffffff811ca825>] block_write_full_page+0x15/0x20 > [<ffffffff811cf5e8>] blkdev_writepage+0x18/0x20 > [<ffffffff8113b387>] __writepage+0x17/0x40 > [<ffffffff8113c64d>] write_cache_pages+0x1fd/0x4c0 > [<ffffffff8113b370>] ? __writepage+0x0/0x40 > [<ffffffff8113c934>] generic_writepages+0x24/0x30 > [<ffffffff8113c961>] do_writepages+0x21/0x40 > [<ffffffff811bf01d>] writeback_single_inode+0xdd/0x290 > [<ffffffff811bf41d>] writeback_sb_inodes+0xbd/0x170 > [<ffffffff811bf57b>] writeback_inodes_wb+0xab/0x1b0 > [<ffffffff811bf973>] wb_writeback+0x2f3/0x410 > [<ffffffff811bfb4b>] wb_do_writeback+0xbb/0x240 > [<ffffffff811bfd33>] bdi_writeback_task+0x63/0x1b0 > [<ffffffff810a12e7>] ? bit_waitqueue+0x17/0xd0 > [<ffffffff8114b760>] ? bdi_start_fn+0x0/0x100 > [<ffffffff8114b7e6>] bdi_start_fn+0x86/0x100 > [<ffffffff8114b760>] ? bdi_start_fn+0x0/0x100 > [<ffffffff810a0fce>] kthread+0x9e/0xc0 > [<ffffffff8100c28a>] child_rip+0xa/0x20 > [<ffffffff810a0f30>] ? kthread+0x0/0xc0 > [<ffffffff8100c280>] ? child_rip+0x0/0x20 > INFO: task mkfs.ext4:3040 blocked for more than 120 seconds. > Not tainted 2.6.32-573.18.1.el6.x86_64 #1 > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > mkfs.ext4 D 0000000000000002 0 3040 3038 0x00000080 > ffff88075e79f4d8 0000000000000082 ffff8808ffffffff 000000003d697f73 > 0000000000000000 ffff88082fb73130 0000000000021472 ffffffffa78356ec > 000000002b9db4fe ffffffff81aa6700 ffff88082e787068 ffff88075e79ffd8 > Call Trace: > [<ffffffff81539673>] io_schedule+0x73/0xc0 > [<ffffffff81276598>] get_request_wait+0x108/0x1d0 > [<ffffffff810a1460>] ? autoremove_wake_function+0x0/0x40 > [<ffffffff812766f9>] blk_queue_bio+0x99/0x610 > > Ceph version is Jewel 10.2.3 > Ceph clients, mons and servers have the kernel 3.10.0-327.36.3.el7.x86_64 > on CentOS 7.2 > > 2017-01-13 20:07 GMT+01:00 Jason Dillaman <jdillama@xxxxxxxxxx>: >> >> You might be hitting this issue [1] where mkfs is issuing lots of >> discard operations. If you get a chance, can you retest w/ the "-E >> nodiscard" option? >> >> Thanks >> >> [1] http://tracker.ceph.com/issues/16689 >> >> On Fri, Jan 13, 2017 at 12:57 PM, Vincent Godin <vince.mlist@xxxxxxxxx> >> wrote: >> > Thanks Jason, >> > >> > We observed a curious behavior : we have some VMs on CentOS 6.x hosted >> > on >> > our Openstack computes which are in CentOS 7.2. If we try to make a >> > mkfs.ext4 on a volume create with the Jewel default (61) on the VM it's >> > hung >> > and we have to reboot the VM to get a responsive system. This is strange >> > because the libvirt process is launched from the host which is in CentOS >> > 7.2. If a disable some features, the mkfs.ext4 succeed. If the VM is in >> > CentOS 7.x, there is no probleme at all. Maybe the kernel of the CentOS >> > 6.X >> > is unable to use the exclusive-lock feature ? >> > I think we will have to stay in a very conservative rbd_default_features >> > such 1 because we don't use stripping and the others features are not >> > compatible with our old CentOS 6.x VMs .. >> > >> > A last question : is the rbd object-map rebuild a long process ? in an >> > other >> > way, does it cost the same time as a delete (which read all the blocks >> > possible for an image without omap feature). Is it a good idea to enable >> > omap feature on an already used image ? (I know that during the rebuild >> > process, the VM will have to be stopped) >> > >> > >> > >> > 2017-01-13 15:09 GMT+01:00 Jason Dillaman <jdillama@xxxxxxxxxx>: >> >> >> >> On Fri, Jan 13, 2017 at 5:11 AM, Vincent Godin <vince.mlist@xxxxxxxxx> >> >> wrote: >> >> > We are using a production cluster which started in Firefly, then >> >> > moved >> >> > to >> >> > Giant, Hammer and finally Jewel. So our images have different >> >> > features >> >> > correspondind to the value of "rbd_default_features" of the version >> >> > when >> >> > they were created. >> >> > We have actually three pack of features activated : >> >> > image with : >> >> > - layering ~ 1 >> >> > - layering, striping ~3 >> >> > - layering, exclusive-lock, object-map, fast-diff, deep-flatten ~ 61 >> >> > >> >> > 1) Is it a good idea to try to give all images the same features ? >> >> >> >> It isn't needed. >> >> >> >> > 2) Is it possible to disable the striping feature on an already >> >> > created >> >> > image (we never specify any stripe-unit nor stripe-count) ? >> >> >> >> Negative -- striping cannot be dynamically disabled because it would >> >> result in potentially altering the structure and placement of the data >> >> within the image. If your stripe-unit is the object size and the >> >> stripe count is 1, that's a special case where the flag is essentially >> >> ignored. >> >> >> >> > 3) What is the behaviour of an already created image on which we >> >> > activate >> >> > the object-map feature ? Will a process try to rebuild a index of >> >> > used >> >> > blocks - if no, if we delete later the image, will ceph try to remove >> >> > all >> >> > the blocks or only the blocks refered by object-map index ? >> >> >> >> You would need to run "rbd object-map rebuild <image-spec>" to rebuild >> >> the object map. Until it is rebuilt, it will be considered invalid and >> >> won't be used for reference. You can determine the object map state by >> >> running "rbd info <image-spec>" >> >> >> >> > 4) We are on Jewel but with tunables set to hammer (Centos 7.2). What >> >> > are >> >> > the best default features to set in that case ? (we use Ceph under >> >> > an >> >> > Openstack for glance, nova and cinder >> >> >> >> We feel like the current defaults are a good mix of features for >> >> everyday use of non-shared images or non-krbd images. Most >> >> importantly, all the default features can be dynamically disabled if >> >> your needs for the image change. >> >> >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > ceph-users mailing list >> >> > ceph-users@xxxxxxxxxxxxxx >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> >> >> >> >> >> >> >> -- >> >> Jason >> > >> > >> >> >> >> -- >> Jason > > -- Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |