On Fri, Sep 19, 2014 at 11:22 AM, Micha Krause <micha at krausam.de> wrote: > Hi, > >> I have build an NFS Server based on Sebastiens Blog Post here: >> >> http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/ >> >> Im using Kernel 3.14-0.bpo.1-amd64 on Debian wheezy, the host is a VM on >> Vmware. >> >> Using rsync im writing data via nfs from one client to this Server. >> >> The NFS Server crashes multiple times per day, I can't even login to the >> Server then. >> After a reset, there is no kernel log about the crash, so I guess >> something is blocking >> all I/Os. > > > Ok, it seems that I just can't get a shell, but I can run commands via ssh > directly. So does it actually crash or it's just the blocked I/Os? If it doesn't crash, you should be able to get everything off dmesg. > > I was able to get the following informations: > > dmesg: > > [18102.981064] INFO: task nfsd:2769 blocked for more than 120 seconds. > [18102.981112] Not tainted 3.14-0.bpo.1-amd64 #1 > [18102.981150] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables > this message. > [18102.981216] nfsd D ffff88003fc14340 0 2769 2 > 0x00000000 > [18102.981218] ffff88003bac6e20 0000000000000046 0000000000000000 > ffff88003d47ada0 > [18102.981219] 0000000000014340 ffff88003ce31fd8 0000000000014340 > ffff88003bac6e20 > [18102.981221] ffff88003ce31728 ffff8800029539f0 7fffffffffffffff > 7fffffffffffffff > [18102.981223] Call Trace: > [18102.981225] [<ffffffff814eedbd>] ? schedule_timeout+0x1ed/0x250 > [18102.981231] [<ffffffffa04b0f92>] ? _xfs_buf_find+0xd2/0x280 [xfs] > [18102.981234] [<ffffffff8117fc2c>] ? kmem_cache_alloc+0x1bc/0x1f0 > [18102.981236] [<ffffffff814f193c>] ? __down_common+0x97/0xea > [18102.981241] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs] > [18102.981243] [<ffffffff810aa697>] ? down+0x37/0x40 > [18102.981247] [<ffffffffa04b0e02>] ? xfs_buf_lock+0x32/0xf0 [xfs] > [18102.981252] [<ffffffffa04b0faa>] ? _xfs_buf_find+0xea/0x280 [xfs] > [18102.981257] [<ffffffffa04b1215>] ? xfs_buf_get_map+0x35/0x1a0 [xfs] > [18102.981263] [<ffffffffa04b2153>] ? xfs_buf_read_map+0x33/0x130 [xfs] > [18102.981269] [<ffffffffa05161da>] ? xfs_trans_read_buf_map+0x34a/0x4f0 > [xfs] > [18102.981275] [<ffffffffa05036f9>] ? xfs_imap_to_bp+0x69/0xf0 [xfs] > [18102.981281] [<ffffffffa0503bcd>] ? xfs_iread+0x7d/0x3f0 [xfs] > [18102.981284] [<ffffffff810e8939>] ? make_kgid+0x9/0x10 > [18102.981286] [<ffffffff811b148e>] ? inode_init_always+0x10e/0x1d0 > [18102.981292] [<ffffffffa04ba11a>] ? xfs_iget+0x2ba/0x810 [xfs] > [18102.981298] [<ffffffffa04fd9a6>] ? xfs_ialloc+0xe6/0x740 [xfs] > [18102.981305] [<ffffffffa04ca1ee>] ? kmem_zone_alloc+0x6e/0xf0 [xfs] > [18102.981311] [<ffffffffa04fe083>] ? xfs_dir_ialloc+0x83/0x300 [xfs] > [18102.981317] [<ffffffffa04c8e43>] ? xfs_trans_reserve+0x213/0x220 [xfs] > [18102.981323] [<ffffffffa04fe87e>] ? xfs_create+0x4fe/0x720 [xfs] > [18102.981329] [<ffffffffa04bfd02>] ? xfs_vn_mknod+0xd2/0x200 [xfs] > [18102.981331] [<ffffffff811a6b54>] ? vfs_create+0xe4/0x160 > [18102.981335] [<ffffffffa0400d9e>] ? do_nfsd_create+0x53e/0x610 [nfsd] > [18102.981339] [<ffffffffa0407f4d>] ? nfsd3_proc_create+0x16d/0x250 [nfsd] > [18102.981342] [<ffffffffa03f9d74>] ? nfsd_dispatch+0xe4/0x230 [nfsd] > [18102.981347] [<ffffffffa035dd64>] ? svc_process_common+0x354/0x690 > [sunrpc] > [18102.981349] [<ffffffff81096ab0>] ? try_to_wake_up+0x280/0x280 > [18102.981353] [<ffffffffa035e3fb>] ? svc_process+0x10b/0x160 [sunrpc] > [18102.981359] [<ffffffffa03f96d7>] ? nfsd+0xb7/0x130 [nfsd] > [18102.981363] [<ffffffffa03f9620>] ? nfsd_destroy+0x70/0x70 [nfsd] > [18102.981365] [<ffffffff81086d6c>] ? kthread+0xbc/0xe0 > [18102.981367] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0 > [18102.981369] [<ffffffff814faecc>] ? ret_from_fork+0x7c/0xb0 > [18102.981371] [<ffffffff81086cb0>] ? flush_kthread_worker+0xa0/0xa0 Is that the only hung task in dmesg? > > iostat: > avg-cpu: %user %nice %system %iowait %steal %idle > 0.00 0.00 1.00 99.00 0.00 0.00 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz > avgqu-sz await r_await w_await svctm %util > sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > dm-4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > rbd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 46.00 0.00 0.00 0.00 0.00 100.00 > rbd1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 12.00 0.00 0.00 0.00 0.00 100.00 > rbd2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 136.00 0.00 0.00 0.00 0.00 100.00 > rbd3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 0.00 0.00 0.00 0.00 0.00 0.00 > rbd4 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 11.00 0.00 0.00 0.00 0.00 100.00 > rbd5 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 57.00 0.00 0.00 0.00 0.00 100.00 > emcpowerig 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 32.00 0.00 0.00 0.00 0.00 100.00 > emcpowerhq 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > 38.00 0.00 0.00 0.00 0.00 100.00 > > (for some reason rbd6 and rbd7 are shown as emcpower in iostat, no idea why) > > cat /sys/kernel/debug/ceph/*/* > > have osdmap 39226 > want next osdmap > epoch 10 > mon0 10.210.32.11:6789 > mon1 10.210.33.11:6789 > mon2 10.210.34.11:6789 > 582277 osd19 37.e52208ae rb.0.1676178.2ae8944a.000000005ffd > write > 582278 osd2 37.f52b9433 rb.0.1676178.2ae8944a.000000006000 > write > 582279 osd2 37.b3f0aae3 rb.0.1676178.2ae8944a.00000000641b > write > 582280 osd28 37.d8768bba rb.0.1676178.2ae8944a.00000000641c > write > 582282 osd29 37.a923b4c6 rb.0.1676178.2ae8944a.000000008032 > write > 582283 osd28 37.a2510620 rb.0.1676178.2ae8944a.000000008034 > write > 582289 osd18 37.c96bc19d rb.0.1345def.2ae8944a.0000001401cf > write > 582290 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 > write > 582291 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee > write > 582292 osd20 37.89f3f734 rb.0.1676160.238e1f29.0000000002ee > write > 582293 osd2 37.d34be89b rb.0.1345def.2ae8944a.00000003c961 > read > 582294 osd4 37.c611292f rb.0.1345def.2ae8944a.00000007a19a > read > 582295 osd20 37.b1ae9634 rb.0.1345def.2ae8944a.00000003def7 > read > 582296 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106 > write > 582297 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 > write > 582298 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 > write > 582299 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149 > write > 582300 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582301 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582302 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582303 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582304 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582305 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582306 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582307 osd18 37.fe0a5354 rb.0.1689d69.238e1f29.0000000221bb > write > 582308 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582309 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 > write > 582310 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 > write > 582311 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582312 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582313 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582314 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582315 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582316 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582317 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582318 osd24 37.3c5f1cf8 rb.0.1689d69.238e1f29.0000000221b9 > write > 582319 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba > write > 582320 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582321 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582322 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582323 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582324 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582325 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582326 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582327 osd2 37.d142a662 rb.0.1689d69.238e1f29.0000000221bc > write > 582328 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd > write > 582329 osd19 37.3928a207 rb.0.1689cf4.238e1f29.000000034106 > write > 582330 osd23 37.834e04c8 rb.0.1689cf4.238e1f29.00000002485f > write > 582331 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea > write > 582332 osd2 37.a7fb0062 rb.0.1689cf4.238e1f29.00000002bfea > write > 582333 osd29 37.e3741a18 rb.0.1689cf4.238e1f29.00000002c0f6 > write > 582334 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7 > write > 582335 osd19 37.dec7ad07 rb.0.1689cf4.238e1f29.000000031fe7 > write > 582336 osd26 37.3f51e7ac rb.0.1689cf4.238e1f29.0000000320e7 > write > 582337 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6 > write > 582338 osd18 37.536a8f6 rb.0.1689cf4.238e1f29.000000033fe6 > write > 582339 osd18 37.e3d0fce6 rb.0.1689cf4.238e1f29.00000002005f > write > 582340 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba > write > 582341 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba > write > 582342 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba > write > 582343 osd1 37.36930f4c rb.0.1689d69.238e1f29.0000000221ba > write > 582344 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd > write > 582345 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd > write > 582346 osd27 37.886d4a41 rb.0.1689d69.238e1f29.0000000221bd > write > 582347 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582348 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582349 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582350 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582351 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582352 osd16 37.9764b7df rb.0.1676178.2ae8944a.000000016392 > write > 582353 osd4 37.1067846f rb.0.1676178.2ae8944a.00000003a43c > write > 582354 osd26 37.ecccc929 rb.0.1676178.2ae8944a.00000003c703 > write > 582355 osd18 37.ed91673e rb.0.1676178.2ae8944a.00000003c704 > write > 582356 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1 > write > 582357 osd23 37.58fdaf5a rb.0.1676178.2ae8944a.00000003dfe1 > write > 582358 osd23 37.a050d40f rb.0.1676178.2ae8944a.00000003e0a4 > write > 582359 osd1 37.c3f1f60c rb.0.1676178.2ae8944a.00000003e24f > write > 582360 osd18 37.8ad6d326 rb.0.1676178.2ae8944a.00000003e50b > write > 582361 osd28 37.cdf2c9ba rb.0.1676178.2ae8944a.00000003e50c > write > 582362 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d > write > 582363 osd27 37.99cdab71 rb.0.1676178.2ae8944a.00000003e50d > write > 582364 osd16 37.1eaec3ff rb.0.1676178.2ae8944a.00000000641a > write > 582365 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc > write > 582366 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc > write > 582367 osd24 37.d881c8e rb.0.1676178.2ae8944a.000000007ffc > write > 582368 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c > write > 582369 osd27 37.d054a135 rb.0.1676178.2ae8944a.00000000842c > write > 582370 osd18 37.876fd4d7 rb.0.1676178.2ae8944a.000000008430 > write > 582371 osd19 37.e08b5547 rb.0.1676178.2ae8944a.000000008431 > write > 582372 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e > write > 582373 osd26 37.cd501c rb.0.1676178.2ae8944a.00000000843e > write > 582374 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5 > write > 582375 osd2 37.c5381d37 rb.0.1676178.2ae8944a.000000015ff5 > write > 582376 osd17 37.82f3d395 rb.0.1676178.2ae8944a.000000016173 > write > 582377 osd19 37.9af0f044 rb.0.1676178.2ae8944a.000000016389 > write > 582378 osd29 37.6d121a86 rb.0.1676178.2ae8944a.00000002004a > write > 582379 osd2 37.ee45629b rb.0.1689d69.238e1f29.00000000484e > write > 582380 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd > write > 582381 osd28 37.b24e1c3a rb.0.1689d69.238e1f29.000000005ffd > write > 582382 osd27 37.56c53271 rb.0.1689d69.238e1f29.00000000618d > write > 582383 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc > write > 582384 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc > write > 582385 osd4 37.a5d8fbaf rb.0.1689d69.238e1f29.000000007ffc > write > 582386 osd1 37.947c5bcc rb.0.1689d69.238e1f29.0000000080bd > write > 582387 osd26 37.7ae1d612 rb.0.1689d69.238e1f29.000000008112 > write > 582388 osd18 37.9bf4cf3c rb.0.1689d69.238e1f29.0000000081eb > write > 582389 osd1 37.9d5cf7f0 rb.0.1689d69.238e1f29.0000000081ec > write > 582390 osd19 37.4f750fee rb.0.1689d69.238e1f29.0000000081f0 > write > 582391 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb > write > 582392 osd19 37.3db853ad rb.0.1689d69.238e1f29.000000009ffb > write > 582393 osd26 37.94b385c rb.0.1689d69.238e1f29.00000000a1d1 > write > 582394 osd28 37.fd40607a rb.0.1689d69.238e1f29.000000022197 > write > 582395 osd15 37.1011005e rb.0.1689d69.238e1f29.000000034614 > write > 582396 osd27 37.c42eab1 rb.0.1689d69.238e1f29.000000036160 > write > 582397 osd2 37.2ac07662 rb.0.1689d69.238e1f29.00000001fffc > write > 582398 osd24 37.80a8ba78 rb.0.167614e.2ae8944a.000000026149 > write > 582399 osd18 37.5c0f9c25 rb.0.167614e.2ae8944a.00000002a133 > write > 582400 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 > write > 582401 osd26 37.9be1215c rb.0.167614e.2ae8944a.00000000042b > write > 582402 osd23 37.a64e7d0f rb.0.167614e.2ae8944a.00000000238f > write > 582403 osd20 37.da28f8ab rb.0.167614e.2ae8944a.000000003ffe > write > 582404 osd24 37.b7af1f40 rb.0.167614e.2ae8944a.000000004005 > write > 582405 osd20 37.cecda9b4 rb.0.167614e.2ae8944a.000000004b8a > write > 582406 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd > write > 582407 osd19 37.29bcdcae rb.0.167614e.2ae8944a.000000005ffd > write > 582408 osd27 37.f29b1b01 rb.0.167614e.2ae8944a.000000006242 > write > 582409 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc > write > 582410 osd18 37.d68a6565 rb.0.167614e.2ae8944a.000000007ffc > write > 582411 osd24 37.87cc6a8e rb.0.167614e.2ae8944a.0000000082f2 > write > 582412 osd15 37.a91eb70d rb.0.167614e.2ae8944a.000000009ffb > write > 582413 osd15 37.3b20fd05 rb.0.167614e.2ae8944a.00000000a03a > write > 582414 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 > write > 582415 osd2 37.89e0f5b3 rb.0.167614e.2ae8944a.00000002ffe8 > write > 582416 osd26 37.f0bb2a2a rb.0.167614e.2ae8944a.0000000301cc > write > 582417 osd26 37.3b4ebe12 rb.0.167614e.2ae8944a.0000000301cd > write > 582418 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7 > write > 582419 osd24 37.346ed74e rb.0.167614e.2ae8944a.000000031fe7 > write > 582420 osd27 37.3f46e175 rb.0.167614e.2ae8944a.00000003215c > write > 582421 osd18 37.796dd926 rb.0.167614e.2ae8944a.00000003217a > write > 582422 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6 > write > 582423 osd19 37.4287ec84 rb.0.167614e.2ae8944a.000000033fe6 > write > 582424 osd26 37.84b7afe9 rb.0.167614e.2ae8944a.000000034112 > write > 582425 osd24 37.f9a234b8 rb.0.167614e.2ae8944a.000000034113 > write > 582426 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5 > write > 582427 osd18 37.6bd0b876 rb.0.167614e.2ae8944a.000000035fe5 > write > 582428 osd23 37.1ae6123b rb.0.167614e.2ae8944a.00000003611a > write > 582429 osd18 37.8202a597 rb.0.167614e.2ae8944a.00000003611b > write > 582430 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4 > write > 582431 osd17 37.4d252e13 rb.0.167614e.2ae8944a.000000037fe4 > write > 582432 osd4 37.7caa0b rb.0.167614e.2ae8944a.00000003800a > write > 582433 osd4 37.5917feaf rb.0.167614e.2ae8944a.0000000380cd > write > 582434 osd29 37.abe267c6 rb.0.167614e.2ae8944a.0000000380cf > write > 582435 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3 > write > 582436 osd16 37.dce75fe7 rb.0.167614e.2ae8944a.000000039fe3 > write > 582437 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f > write > 582438 osd1 37.f04f6991 rb.0.167614e.2ae8944a.00000003a02f > write > 582439 osd27 37.33f8b641 rb.0.167614e.2ae8944a.00000003cc91 > write > 582440 osd22 37.be13ad3d rb.0.167614e.2ae8944a.00000003e40d > write > 582441 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035 > write > 582442 osd18 37.bbf96366 rb.0.167614e.2ae8944a.000000020035 > write > 582443 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 > write > 582444 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 > write > 582445 osd1 37.3edba98c rb.0.165171e.238e1f29.000000039fe3 > write > 582446 osd16 37.4800911f rb.0.165171e.238e1f29.00000003a085 > write > 582447 osd2 37.bdec20a3 rb.0.165171e.238e1f29.00000003a12c > write > 582448 osd23 37.33d51508 rb.0.165171e.238e1f29.00000003a12d > write > 582449 osd16 37.8a8ec527 rb.0.165171e.238e1f29.00000003a138 > write > 582450 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139 > write > 582451 osd17 37.d7435c13 rb.0.165171e.238e1f29.00000003a139 > write > 582452 osd15 37.7e4b4f9e rb.0.165171e.238e1f29.00000003a13a > write > 582453 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b > write > 582454 osd26 37.ae80f3ac rb.0.165171e.238e1f29.00000003a13b > write > 582455 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c > write > 582456 osd15 37.fc3cadcd rb.0.165171e.238e1f29.00000003a13c > write > 582457 osd19 37.dcc1f244 rb.0.165171e.238e1f29.00000003a13d > write > 582458 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e > write > 582459 osd28 37.c5ce907a rb.0.165171e.238e1f29.00000003a13e > write > 582460 osd18 37.d7371b26 rb.0.165171e.238e1f29.00000003a13f > write > 582461 osd18 37.89ec9be5 rb.0.165171e.238e1f29.00000003a140 > write > 582462 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2 > write > 582463 osd4 37.5032c82f rb.0.165171e.238e1f29.00000003bfe2 > write > 582464 osd26 37.54a4fd50 rb.0.165171e.238e1f29.00000003bfe9 > write > 582465 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136 > write > 582466 osd23 37.2929897b rb.0.165171e.238e1f29.00000003c136 > write > 582467 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1 > write > 582468 osd20 37.b9aff419 rb.0.165171e.238e1f29.00000003dfe1 > write > 582469 osd24 37.685a8638 rb.0.165171e.238e1f29.00000003e08c > write > 582470 osd26 37.adfd8b12 rb.0.165171e.238e1f29.00000003e14a > write > 582471 osd2 37.a67386a2 rb.0.165171e.238e1f29.00000003e14b > write > 582472 osd18 37.688ac754 rb.0.165171e.238e1f29.00000002802c > write > 582473 osd15 37.d2bed74d rb.0.1676160.238e1f29.00000002000d > write > 582474 osd23 37.9c9a1a8f rb.0.165171e.238e1f29.000000020002 > write Have you looked at Ceph servers? krbd is really just a client, so if OSDs don't reply to its requests it can't do much. From a quick look this doesn't look like a krbd bug. > epoch 39226 > flags > pg_pool 0 pg_num 256 / 255 > pg_pool 1 pg_num 128 / 127 > pg_pool 4 pg_num 32 / 31 > pg_pool 19 pg_num 512 / 511 > pg_pool 25 pg_num 8 / 7 > pg_pool 27 pg_num 1 / 0 > pg_pool 28 pg_num 1 / 0 > pg_pool 29 pg_num 1 / 0 > pg_pool 30 pg_num 1 / 0 > pg_pool 31 pg_num 1 / 0 > pg_pool 32 pg_num 1 / 0 > pg_pool 33 pg_num 1 / 0 > pg_pool 34 pg_num 1 / 0 > pg_pool 35 pg_num 2 / 1 > pg_pool 36 pg_num 1 / 0 > pg_pool 37 pg_num 64 / 63 > pg_pool 40 pg_num 2 / 1 > pg_pool 41 pg_num 1 / 0 > osd0 10.210.33.22:6815 100% (exists, up) > osd1 10.210.33.22:6800 100% (exists, up) > osd2 10.210.33.22:6805 100% (exists, up) > osd3 10.210.32.22:6800 0% (doesn't exist) > osd4 10.210.33.22:6810 100% (exists, up) > osd5 10.210.33.22:6820 100% (doesn't exist) > osd6 10.210.33.22:6805 100% (doesn't exist) > osd7 10.210.33.22:6825 100% (doesn't exist) > osd8 10.210.33.22:6830 100% (doesn't exist) > osd9 10.210.33.22:6835 100% (doesn't exist) > osd10 10.210.33.22:6805 100% (doesn't exist) > osd11 10.210.33.22:6845 100% (doesn't exist) > osd12 10.210.33.22:6850 100% (doesn't exist) > osd13 10.210.33.22:6855 100% (doesn't exist) > osd14 10.210.33.22:6860 100% (doesn't exist) > osd15 10.210.32.23:6800 100% (exists, up) > osd16 10.210.32.23:6807 100% (exists, up) > osd17 10.210.32.23:6801 100% (exists, up) > osd18 10.210.32.23:6816 100% (exists, up) > osd19 10.210.32.23:6812 100% (exists, up) > osd20 10.210.34.21:6800 100% (exists, up) > osd21 10.210.34.21:6804 100% (exists, up) > osd22 10.210.34.21:6809 100% (exists, up) > osd23 10.210.34.21:6814 100% (exists, up) > osd24 10.210.34.21:6819 100% (exists, up) > osd25 10.210.33.21:6800 100% (exists, up) > osd26 10.210.33.21:6805 100% (exists, up) > osd27 10.210.33.21:6810 100% (exists, up) > osd28 10.210.33.21:6815 100% (exists, up) > osd29 10.210.33.21:6820 100% (exists, up) > osd30 10.210.33.22:6865 100% (doesn't exist) > osd31 10.210.33.22:6870 0% (doesn't exist) > osd32 10.210.33.21:6800 0% (doesn't exist) > > I don't know how to interpret this, the doesn't exist lines are correct, > these osds where removed. > Why are they still known to the rbd client? The OSDs where removed before > the client was booted. What procedure did you follow to remove those OSDs? Thanks, Ilya
 |