Hi,
I was able to get a dmesg output from the centos Machine with kernel 3.16:
kworker/3:2:9521 blocked for more than 120 seconds.
Not tainted 3.16.2-1.el6.elrepo.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/3:2 D 0000000000000003 0 9521 2 0x00000080
Workqueue: events handle_timeout [libceph]
ffff8801228cfcd8 0000000000000046 0000000300000000 ffff8801228cc010
0000000000014400 0000000000014400 ffff8800ba01c250 ffff880234ed3070
0000000000000000 ffff8800baf237c8 ffff8800baf237cc ffff8800ba01c250
Call Trace:
[<ffffffff81647da9>] schedule+0x29/0x70
[<ffffffff81647f0e>] schedule_preempt_disabled+0xe/0x10
[<ffffffff8164987b>] __mutex_lock_slowpath+0xdb/0x1d0
[<ffffffff81649993>] mutex_lock+0x23/0x40
[<ffffffffa0348c73>] handle_timeout+0x63/0x1c0 [libceph]
[<ffffffff8108d60c>] process_one_work+0x17c/0x420
[<ffffffff8108e7d3>] worker_thread+0x123/0x420
[<ffffffff8108e6b0>] ? maybe_create_worker+0x180/0x180
[<ffffffff810943be>] kthread+0xce/0xf0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff8164b5bc>] ret_from_fork+0x7c/0xb0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
INFO: task kworker/3:1:62 blocked for more than 120 seconds.
Not tainted 3.16.2-1.el6.elrepo.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/3:1 D 0000000000000003 0 62 2 0x00000000
Workqueue: events handle_osds_timeout [libceph]
ffff880037907ce8 0000000000000046 0000000000000000 ffff880037904010
0000000000014400 0000000000014400 ffff880232389130 ffff880234ed3070
ffffffff8101d833 ffff8800baf237c8 ffff8800baf237cc ffff880232389130
Call Trace:
[<ffffffff8101d833>] ? native_sched_clock+0x33/0xd0
[<ffffffff81647da9>] schedule+0x29/0x70
[<ffffffff81647f0e>] schedule_preempt_disabled+0xe/0x10
[<ffffffff8164987b>] __mutex_lock_slowpath+0xdb/0x1d0
[<ffffffff810afbbf>] ? put_prev_entity+0x2f/0x400
[<ffffffff81649993>] mutex_lock+0x23/0x40
[<ffffffffa0347003>] handle_osds_timeout+0x53/0x120 [libceph]
[<ffffffff8108d60c>] process_one_work+0x17c/0x420
[<ffffffff8108e7d3>] worker_thread+0x123/0x420
[<ffffffff8108e6b0>] ? maybe_create_worker+0x180/0x180
[<ffffffff810943be>] kthread+0xce/0xf0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff8164b5bc>] ret_from_fork+0x7c/0xb0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
INFO: task kworker/u8:0:9486 blocked for more than 120 seconds.
Not tainted 3.16.2-1.el6.elrepo.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/u8:0 D 0000000000000002 0 9486 2 0x00000080
Workqueue: writeback bdi_writeback_workfn (flush-253:7)
ffff8802337cf368 0000000000000046 00000000ae5d42c1 ffff8802337cc010
0000000000014400 0000000000014400 ffff880232554fb0 ffff8800ba4be210
ffff8802337cc010 ffff8800ba5579b8 ffff880232fa0250 ffff880232554fb0
Call Trace:
[<ffffffff81647da9>] schedule+0x29/0x70
[<ffffffff81647f0e>] schedule_preempt_disabled+0xe/0x10
[<ffffffff81649962>] __mutex_lock_slowpath+0x1c2/0x1d0
[<ffffffff81649993>] mutex_lock+0x23/0x40
[<ffffffffa033fc2d>] ceph_con_send+0x4d/0x150 [libceph]
[<ffffffffa0348bc4>] __send_queued+0x134/0x180 [libceph]
[<ffffffffa0349e7b>] __ceph_osdc_start_request+0x5b/0xb0 [libceph]
[<ffffffffa0349f21>] ceph_osdc_start_request+0x51/0x80 [libceph]
[<ffffffffa037b2a0>] rbd_img_obj_request_submit+0xb0/0x110 [rbd]
[<ffffffffa037b349>] rbd_img_request_submit+0x49/0x60 [rbd]
[<ffffffffa037bcd8>] rbd_request_fn+0x248/0x2b0 [rbd]
[<ffffffff812b22e7>] __blk_run_queue+0x37/0x50
[<ffffffff812b296e>] queue_unplugged+0x4e/0xb0
[<ffffffff812b2b2e>] blk_flush_plug_list+0x15e/0x200
[<ffffffff81647e65>] io_schedule+0x75/0xd0
[<ffffffff812b3f87>] get_request+0x167/0x340
[<ffffffff810b6220>] ? bit_waitqueue+0xe0/0xe0
[<ffffffff812ae78b>] ? elv_merge+0xeb/0xf0
[<ffffffff812b4228>] blk_queue_bio+0xc8/0x340
[<ffffffff812b30f0>] generic_make_request+0xc0/0x100
[<ffffffff812b31b0>] submit_bio+0x80/0x170
[<ffffffff812abdf1>] ? bio_alloc_bioset+0xa1/0x1e0
[<ffffffff811ff4a6>] _submit_bh+0x146/0x220
[<ffffffff811ff590>] submit_bh+0x10/0x20
[<ffffffff81202ed3>] __block_write_full_page.clone.0+0x1a3/0x340
[<ffffffff81203790>] ? I_BDEV+0x10/0x10
[<ffffffff81203790>] ? I_BDEV+0x10/0x10
[<ffffffff81203246>] block_write_full_page+0xc6/0x100
[<ffffffff81204848>] blkdev_writepage+0x18/0x20
[<ffffffff81163be7>] __writepage+0x17/0x50
[<ffffffff81164fe4>] write_cache_pages+0x244/0x510
[<ffffffff81163bd0>] ? set_page_dirty+0x60/0x60
[<ffffffff81165301>] generic_writepages+0x51/0x80
[<ffffffff81165350>] do_writepages+0x20/0x40
[<ffffffff811f6309>] __writeback_single_inode+0x49/0x230
[<ffffffff810b665f>] ? wake_up_bit+0x2f/0x40
[<ffffffff811f7149>] writeback_sb_inodes+0x279/0x390
[<ffffffff811d03d5>] ? put_super+0x25/0x40
[<ffffffff811f72fe>] __writeback_inodes_wb+0x9e/0xd0
[<ffffffff811f752b>] wb_writeback+0x1fb/0x2c0
[<ffffffff811f76f0>] wb_do_writeback+0x100/0x1f0
[<ffffffff811f7a60>] bdi_writeback_workfn+0x70/0x210
[<ffffffff8108d60c>] process_one_work+0x17c/0x420
[<ffffffff8108e7d3>] worker_thread+0x123/0x420
[<ffffffff8108e6b0>] ? maybe_create_worker+0x180/0x180
[<ffffffff810943be>] kthread+0xce/0xf0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
[<ffffffff8164b5bc>] ret_from_fork+0x7c/0xb0
[<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
Micha Krause
Am 23.09.2014 um 15:37 schrieb Micha Krause:
> bump
>
> I have observed this crash on ubuntu with kernel 3.13 and centos with 3.16 as well now.
> rbd hangs, and iostat shows something similar to the Output below.
>
>
> Micha Krause
>
> Am 19.09.2014 um 09:22 schrieb Micha Krause:
>> Hi,
>>
>> > I have build an NFS Server based on Sebastiens Blog Post here:
>>> http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/
>>>
>>> Im using Kernel 3.14-0.bpo.1-amd64 on Debian wheezy, the host is a VM on Vmware.
>>>
>>> Using rsync im writing data via nfs from one client to this Server.
>>>
>>> The NFS Server crashes multiple times per day, I can't even login to the Server then.
>>> After a reset, there is no kernel log about the crash, so I guess something is blocking
>>> all I/Os.
>>
>> Ok, it seems that I just can't get a shell, but I can run commands via ssh directly.
>>
>> I was able to get the following informations:
>>
>> dmesg:
>>
>> [18102.981064] INFO: task nfsd:2769 blocked for more than 120 seconds.
>> [18102.981112] Not tainted 3.14-0.bpo.1-amd64 #1
>> [18102.981150] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> [18102.981216] nfsd D ffff88003fc14340 0 2769 2 0x00000000
>> [18102.981218] ffff88003bac6e20 0000000000000046 0000000000000000 ffff88003d47ada0
>> [18102.981219] 0000000000014340 ffff88003ce31fd8 0000000000014340 ffff88003bac6e20
>> [18102.981221] ffff88003ce31728 ffff8800029539f0 7fffffffffffffff 7fffffffffffffff
>> [18102.981223] Call Trace:
>> [18102.981225] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250
>> [18102.981231] [<ffffffffa04b0f92>] ? _xfs_buf_find+0xd2/0x280 [xfs]
>> [18102.981234] [<ffffffff8117fc2c>] ? kmem_cache_alloc+0x1bc/0x1f0
>> [18102.981236] [<ffffffff814f193c>] ? __down_common+0x97/0xea
>> [18102.981241] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs]
>> [18102.981243] [<ffffffff810aa697>] ? down+0x37/0x40
>> [18102.981247] [<ffffffffa04b0e02>] ? xfs_buf_lock+0x32/0xf0 [xfs]
>> [18102.981252] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs]
>> [18102.981257] [<ffffffffa04b1215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs]
>> [18102.981263] [<ffffffffa04b2153>] ? xfs_buf_read_map+0x33/0x130 [xfs]
>> [18102.981269] [<ffffffffa05161da>] ? xfs_trans_read_buf_map+0x34a/0x4f0 [xfs]
>> [18102.981275] [<ffffffffa05036f9>] ? xfs_imap_to_bp+0x69/0xf0 [xfs]
>> [18102.981281] [<ffffffffa0503bcd>] ? xfs_iread+0x7d/0x3f0 [xfs]
>> [18102.981284] [<ffffffff810e8939>] ? make_kgid+0x9/0x10
>> [18102.981286] [<ffffffff811b148e>] ? inode_init_always+0x10e/0x1d0
>> [18102.981292] [<ffffffffa04ba11a>] ? xfs_iget+0x2ba/0x810 [xfs]
>> [18102.981298] [<ffffffffa04fd9a6>] ? xfs_ialloc+0xe6/0x740 [xfs]
>> [18102.981305] [<ffffffffa04ca1ee>] ? kmem_zone_alloc+0x6e/0xf0 [xfs]
>> [18102.981311] [<ffffffffa04fe083>] ? xfs_dir_ialloc+0x83/0x300 [xfs]
>> [18102.981317] [<ffffffffa04c8e43>] ? xfs_trans_reserve+0x213/0x220 [xfs]
>> [18102.981323] [<ffffffffa04fe87e>] ? xfs_create+0x4fe/0x720 [xfs]
>> [18102.981329] [<ffffffffa04bfd02>] ? xfs_vn_mknod+0xd2/0x200 [xfs]
>> [18102.981331] [<ffffffff811a6b54>] ? vfs_create+0xe4/0x160
>> [18102.981335] [<ffffffffa0400d9e>] ? do_nfsd_create+0x53e/0x610 [nfsd]
>> [18102.981339] [<ffffffffa0407f4d>] ? nfsd3_proc_create+0x16d/0x250 [nfsd]
>> [18102.981342] [<ffffffffa03f9d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd]
>> [18102.981347] [<ffffffffa035dd64>] ? svc_process_common+0x354/0x690 [sunrpc]
>> [18102.981349] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280
>> [18102.981353] [<ffffffffa035e3fb>] ? svc_process+0x10b/0x160 [sunrpc]
>> [18102.981359] [<ffffffffa03f96d7>] ? nfsd+0xb7/0x130 [nfsd]
>> [18102.981363] [<ffffffffa03f9620>] ? nfsd_destroy+0x70/0x70 [nfsd]
>> [18102.981365] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0
>> [18102.981367] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
>> [18102.981369] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0
>> [18102.981371] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0
>>
>> iostat:
>> avg-cpu: %user %nice %system %iowait %steal %idle
>> 0.00 0.00 1.00 99.00 0.00 0.00
>>
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> rbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 46.00 0.00 0.00 0.00 0.00 100.00
>> rbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.00 0.00 0.00 0.00 0.00 100.00
>> rbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 136.00 0.00 0.00 0.00 0.00 100.00
>> rbd3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> rbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11.00 0.00 0.00 0.00 0.00 100.00
>> rbd5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 57.00 0.00 0.00 0.00 0.00 100.00
>> emcpowerig 0.00 0.00 0.00 0.00 0.00 0.00 0.00 32.00 0.00 0.00 0.00 0.00 100.00
>> emcpowerhq 0.00 0.00 0.00 0.00 0.00 0.00 0.00 38.00 0.00 0.00 0.00 0.00 100.00
>>
>> (for some reason rbd6 and rbd7 are shown as emcpower in iostat, no idea why)
>>
>> cat /sys/kernel/debug/ceph/*/*
>>
>> have osdmap 39226
>> want next osdmap
>> epoch 10
>> mon0 10.210.32.11:6789
>> mon1 10.210.33.11:6789
>> mon2 10.210.34.11:6789
>> 582277 osd19 37.e52208ae rb.0.1676178.2ae8944a.000000005ffd write
>> 582278 osd2 37.f52b9433 rb.0.1676178.2ae8944a.000000006000 write
>> 582279 osd2 37.b3f0aae3 rb.0.1676178.2ae8944a.00000000641b write
>> 582280 osd28 37.d8768bba rb.0.1676178.2ae8944a.00000000641c write
>> 582282 osd29 37.a923b4c6 rb.0.1676178.2ae8944a.000000008032 write
>> 582283 osd28 37.a2510620 rb.0.1676178.2ae8944a.000000008034 write
>> 582289 osd18 37.c96bc19d rb.0.1345def.2ae8944a.0000001401cf write
>> 582290 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 write
>> 582291 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee write
>> 582292 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee write
>> 582293 osd2 37.d34be89b rb.0.1345def.2ae8944a.00000003c961 read
>> 582294 osd4 37.c611292f rb.0.1345def.2ae8944a.00000007a19a read
>> 582295 osd20 37.b1ae9634 rb.0.1345def.2ae8944a.00000003def7 read
>> 582296 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106 write
>> 582297 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 write
>> 582298 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 write
>> 582299 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149 write
>> 582300 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582301 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582302 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582303 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582304 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582305 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582306 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582307 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb write
>> 582308 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582309 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 write
>> 582310 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 write
>> 582311 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582312 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582313 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582314 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582315 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582316 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582317 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582318 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 write
>> 582319 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba write
>> 582320 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582321 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582322 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582323 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582324 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582325 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582326 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582327 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc write
>> 582328 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd write
>> 582329 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106 write
>> 582330 osd23 37.834e04c8 rb.0.1689cf4.238e1f29.00000002485f write
>> 582331 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea write
>> 582332 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea write
>> 582333 osd29 37.e3741a18 rb.0.1689cf4.238e1f29.00000002c0f6 write
>> 582334 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7 write
>> 582335 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7 write
>> 582336 osd26 37.3f51e7ac rb.0.1689cf4.238e1f29.0000000320e7 write
>> 582337 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6 write
>> 582338 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6 write
>> 582339 osd18 37.e3d0fce6 rb.0.1689cf4.238e1f29.00000002005f write
>> 582340 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba write
>> 582341 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba write
>> 582342 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba write
>> 582343 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba write
>> 582344 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd write
>> 582345 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd write
>> 582346 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd write
>> 582347 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582348 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582349 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582350 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582351 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582352 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 write
>> 582353 osd4 37.1067846f rb.0.1676178.2ae8944a.00000003a43c write
>> 582354 osd26 37.ecccc929 rb.0.1676178.2ae8944a.00000003c703 write
>> 582355 osd18 37.ed91673e rb.0.1676178.2ae8944a.00000003c704 write
>> 582356 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1 write
>> 582357 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1 write
>> 582358 osd23 37.a050d40f rb.0.1676178.2ae8944a.00000003e0a4 write
>> 582359 osd1 37.c3f1f60c rb.0.1676178.2ae8944a.00000003e24f write
>> 582360 osd18 37.8ad6d326 rb.0.1676178.2ae8944a.00000003e50b write
>> 582361 osd28 37.cdf2c9ba rb.0.1676178.2ae8944a.00000003e50c write
>> 582362 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d write
>> 582363 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d write
>> 582364 osd16 37.1eaec3ff rb.0.1676178.2ae8944a.00000000641a write
>> 582365 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc write
>> 582366 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc write
>> 582367 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc write
>> 582368 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c write
>> 582369 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c write
>> 582370 osd18 37.876fd4d7 rb.0.1676178.2ae8944a.000000008430 write
>> 582371 osd19 37.e08b5547 rb.0.1676178.2ae8944a.000000008431 write
>> 582372 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e write
>> 582373 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e write
>> 582374 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5 write
>> 582375 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5 write
>> 582376 osd17 37.82f3d395 rb.0.1676178.2ae8944a.000000016173 write
>> 582377 osd19 37.9af0f044 rb.0.1676178.2ae8944a.000000016389 write
>> 582378 osd29 37.6d121a86 rb.0.1676178.2ae8944a.00000002004a write
>> 582379 osd2 37.ee45629b rb.0.1689d69.238e1f29.00000000484e write
>> 582380 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd write
>> 582381 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd write
>> 582382 osd27 37.56c53271 rb.0.1689d69.238e1f29.00000000618d write
>> 582383 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc write
>> 582384 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc write
>> 582385 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc write
>> 582386 osd1 37.947c5bcc rb.0.1689d69.238e1f29.0000000080bd write
>> 582387 osd26 37.7ae1d612 rb.0.1689d69.238e1f29.000000008112 write
>> 582388 osd18 37.9bf4cf3c rb.0.1689d69.238e1f29.0000000081eb write
>> 582389 osd1 37.9d5cf7f0 rb.0.1689d69.238e1f29.0000000081ec write
>> 582390 osd19 37.4f750fee rb.0.1689d69.238e1f29.0000000081f0 write
>> 582391 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb write
>> 582392 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb write
>> 582393 osd26 37.94b385c rb.0.1689d69.238e1f29.00000000a1d1 write
>> 582394 osd28 37.fd40607a rb.0.1689d69.238e1f29.000000022197 write
>> 582395 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 write
>> 582396 osd27 37.c42eab1 rb.0.1689d69.238e1f29.000000036160 write
>> 582397 osd2 37.2ac07662 rb.0.1689d69.238e1f29.00000001fffc write
>> 582398 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149 write
>> 582399 osd18 37.5c0f9c25 rb.0.167614e.2ae8944a.00000002a133 write
>> 582400 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 write
>> 582401 osd26 37.9be1215c rb.0.167614e.2ae8944a.00000000042b write
>> 582402 osd23 37.a64e7d0f rb.0.167614e.2ae8944a.00000000238f write
>> 582403 osd20 37.da28f8ab rb.0.167614e.2ae8944a.000000003ffe write
>> 582404 osd24 37.b7af1f40 rb.0.167614e.2ae8944a.000000004005 write
>> 582405 osd20 37.cecda9b4 rb.0.167614e.2ae8944a.000000004b8a write
>> 582406 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd write
>> 582407 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd write
>> 582408 osd27 37.f29b1b01 rb.0.167614e.2ae8944a.000000006242 write
>> 582409 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc write
>> 582410 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc write
>> 582411 osd24 37.87cc6a8e rb.0.167614e.2ae8944a.0000000082f2 write
>> 582412 osd15 37.a91eb70d rb.0.167614e.2ae8944a.000000009ffb write
>> 582413 osd15 37.3b20fd05 rb.0.167614e.2ae8944a.00000000a03a write
>> 582414 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 write
>> 582415 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 write
>> 582416 osd26 37.f0bb2a2a rb.0.167614e.2ae8944a.0000000301cc write
>> 582417 osd26 37.3b4ebe12 rb.0.167614e.2ae8944a.0000000301cd write
>> 582418 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7 write
>> 582419 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7 write
>> 582420 osd27 37.3f46e175 rb.0.167614e.2ae8944a.00000003215c write
>> 582421 osd18 37.796dd926 rb.0.167614e.2ae8944a.00000003217a write
>> 582422 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6 write
>> 582423 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6 write
>> 582424 osd26 37.84b7afe9 rb.0.167614e.2ae8944a.000000034112 write
>> 582425 osd24 37.f9a234b8 rb.0.167614e.2ae8944a.000000034113 write
>> 582426 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5 write
>> 582427 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5 write
>> 582428 osd23 37.1ae6123b rb.0.167614e.2ae8944a.00000003611a write
>> 582429 osd18 37.8202a597 rb.0.167614e.2ae8944a.00000003611b write
>> 582430 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4 write
>> 582431 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4 write
>> 582432 osd4 37.7caa0b rb.0.167614e.2ae8944a.00000003800a write
>> 582433 osd4 37.5917feaf rb.0.167614e.2ae8944a.0000000380cd write
>> 582434 osd29 37.abe267c6 rb.0.167614e.2ae8944a.0000000380cf write
>> 582435 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3 write
>> 582436 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3 write
>> 582437 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f write
>> 582438 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f write
>> 582439 osd27 37.33f8b641 rb.0.167614e.2ae8944a.00000003cc91 write
>> 582440 osd22 37.be13ad3d rb.0.167614e.2ae8944a.00000003e40d write
>> 582441 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035 write
>> 582442 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035 write
>> 582443 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 write
>> 582444 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 write
>> 582445 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 write
>> 582446 osd16 37.4800911f rb.0.165171e.238e1f29.00000003a085 write
>> 582447 osd2 37.bdec20a3 rb.0.165171e.238e1f29.00000003a12c write
>> 582448 osd23 37.33d51508 rb.0.165171e.238e1f29.00000003a12d write
>> 582449 osd16 37.8a8ec527 rb.0.165171e.238e1f29.00000003a138 write
>> 582450 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139 write
>> 582451 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139 write
>> 582452 osd15 37.7e4b4f9e rb.0.165171e.238e1f29.00000003a13a write
>> 582453 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b write
>> 582454 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b write
>> 582455 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c write
>> 582456 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c write
>> 582457 osd19 37.dcc1f244 rb.0.165171e.238e1f29.00000003a13d write
>> 582458 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e write
>> 582459 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e write
>> 582460 osd18 37.d7371b26 rb.0.165171e.238e1f29.00000003a13f write
>> 582461 osd18 37.89ec9be5 rb.0.165171e.238e1f29.00000003a140 write
>> 582462 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2 write
>> 582463 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2 write
>> 582464 osd26 37.54a4fd50 rb.0.165171e.238e1f29.00000003bfe9 write
>> 582465 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136 write
>> 582466 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136 write
>> 582467 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1 write
>> 582468 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1 write
>> 582469 osd24 37.685a8638 rb.0.165171e.238e1f29.00000003e08c write
>> 582470 osd26 37.adfd8b12 rb.0.165171e.238e1f29.00000003e14a write
>> 582471 osd2 37.a67386a2 rb.0.165171e.238e1f29.00000003e14b write
>> 582472 osd18 37.688ac754 rb.0.165171e.238e1f29.00000002802c write
>> 582473 osd15 37.d2bed74d rb.0.1676160.238e1f29.00000002000d write
>> 582474 osd23 37.9c9a1a8f rb.0.165171e.238e1f29.000000020002 write
>> epoch 39226
>> flags
>> pg_pool 0 pg_num 256 / 255
>> pg_pool 1 pg_num 128 / 127
>> pg_pool 4 pg_num 32 / 31
>> pg_pool 19 pg_num 512 / 511
>> pg_pool 25 pg_num 8 / 7
>> pg_pool 27 pg_num 1 / 0
>> pg_pool 28 pg_num 1 / 0
>> pg_pool 29 pg_num 1 / 0
>> pg_pool 30 pg_num 1 / 0
>> pg_pool 31 pg_num 1 / 0
>> pg_pool 32 pg_num 1 / 0
>> pg_pool 33 pg_num 1 / 0
>> pg_pool 34 pg_num 1 / 0
>> pg_pool 35 pg_num 2 / 1
>> pg_pool 36 pg_num 1 / 0
>> pg_pool 37 pg_num 64 / 63
>> pg_pool 40 pg_num 2 / 1
>> pg_pool 41 pg_num 1 / 0
>> osd0 10.210.33.22:6815 100% (exists, up)
>> osd1 10.210.33.22:6800 100% (exists, up)
>> osd2 10.210.33.22:6805 100% (exists, up)
>> osd3 10.210.32.22:6800 0% (doesn't exist)
>> osd4 10.210.33.22:6810 100% (exists, up)
>> osd5 10.210.33.22:6820 100% (doesn't exist)
>> osd6 10.210.33.22:6805 100% (doesn't exist)
>> osd7 10.210.33.22:6825 100% (doesn't exist)
>> osd8 10.210.33.22:6830 100% (doesn't exist)
>> osd9 10.210.33.22:6835 100% (doesn't exist)
>> osd10 10.210.33.22:6805 100% (doesn't exist)
>> osd11 10.210.33.22:6845 100% (doesn't exist)
>> osd12 10.210.33.22:6850 100% (doesn't exist)
>> osd13 10.210.33.22:6855 100% (doesn't exist)
>> osd14 10.210.33.22:6860 100% (doesn't exist)
>> osd15 10.210.32.23:6800 100% (exists, up)
>> osd16 10.210.32.23:6807 100% (exists, up)
>> osd17 10.210.32.23:6801 100% (exists, up)
>> osd18 10.210.32.23:6816 100% (exists, up)
>> osd19 10.210.32.23:6812 100% (exists, up)
>> osd20 10.210.34.21:6800 100% (exists, up)
>> osd21 10.210.34.21:6804 100% (exists, up)
>> osd22 10.210.34.21:6809 100% (exists, up)
>> osd23 10.210.34.21:6814 100% (exists, up)
>> osd24 10.210.34.21:6819 100% (exists, up)
>> osd25 10.210.33.21:6800 100% (exists, up)
>> osd26 10.210.33.21:6805 100% (exists, up)
>> osd27 10.210.33.21:6810 100% (exists, up)
>> osd28 10.210.33.21:6815 100% (exists, up)
>> osd29 10.210.33.21:6820 100% (exists, up)
>> osd30 10.210.33.22:6865 100% (doesn't exist)
>> osd31 10.210.33.22:6870 0% (doesn't exist)
>> osd32 10.210.33.21:6800 0% (doesn't exist)
>>
>> I don't know how to interpret this, the doesn't exist lines are correct, these osds where removed.
>> Why are they still known to the rbd client? The OSDs where removed before the client was booted.
>>
>>
>>
>> Micha Krause
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users at lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users at lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
