|
Hi all, Sorry to bother with this, I am trying my best to solve it, but I am quite stuck. I will continue to try to dig out more
information any way I can, but I am quite clueless right now when it comes to why my OSDs don´t come up. Any help is of course very appreciated. I increased the logging by setting the loglevel in my ceph.conf anf I got thousands of rows, but I have gone through them
to just paste the ones that are not variations of the same thing in my osd-log and I hope this does npt get too long for the list: 2015-12-07 19:00:04.508322 7f9294de8940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd,
pid 5904 2015-12-07 19:00:04.577887 7f9294de8940 0 filestore(/ceph/osd.0) backend xfs (magic 0x58465342) 2015-12-07 19:00:04.582495 7f9294de8940 0 genericfilestorebackend(/ceph/osd.0) detect_features: FIEMAP ioctl is disabled
via 'filestore fiemap' config option 2015-12-07 19:00:04.582544 7f9294de8940 0 genericfilestorebackend(/ceph/osd.0) detect_features: SEEK_DATA/SEEK_HOLE is
disabled via 'filestore seek data hole' config option 2015-12-07 19:00:04.582603 7f9294de8940 0 genericfilestorebackend(/ceph/osd.0) detect_features: splice is supported 2015-12-07 19:00:04.587960 7f9294de8940 0 genericfilestorebackend(/ceph/osd.0) detect_features: syncfs(2) syscall fully
supported (by glibc and kernel) 2015-12-07 19:00:04.588230 7f9294de8940 0 xfsfilestorebackend(/ceph/osd.0) detect_features: extsize is supported and
your kernel >= 3.5 2015-12-07 19:00:04.807548 7f9294de8940 0 filestore(/ceph/osd.0) mount: enabling WRITEAHEAD journal mode: checkpoint
is not enabled 2015-12-07 19:00:39.087821 7f9294de8940 1 journal _open /dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size
4096 bytes, directio = 1, aio = 1 2015-12-07 19:00:39.117817 7f9294de8940 1 journal _open /dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size
4096 bytes, directio = 1, aio = 1 2015-12-07 19:00:39.166074 7f9294de8940 1 filestore(/ceph/osd.0) upgrade 2015-12-07 19:00:39.184942 7f9294de8940 0 <cls> cls/cephfs/cls_cephfs.cc:136: loading cephfs_size_scan 2015-12-07 19:00:39.186470 7f9294de8940 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello 2015-12-07 19:00:39.205219 7f9294de8940 0 osd.0 39538 crush map has features 1107558400, adjusting msgr requires for
clients 2015-12-07 19:00:39.205261 7f9294de8940 0 osd.0 39538 crush map has features 1107558400 was 8705, adjusting msgr requires
for mons 2015-12-07 19:00:39.205273 7f9294de8940 0 osd.0 39538 crush map has features 1107558400, adjusting msgr requires for
osds 2015-12-07 19:01:19.311676 7f9294de8940 0 osd.0 39538 load_pgs 2015-12-07 19:02:08.559239 7f9294de8940 0 osd.0 39538 load_pgs opened 1230 pgs 2015-12-07 19:02:08.593503 7f9294de8940 -1 osd.0 39538 log_to_monitors {default=true} 2015-12-07 19:02:08.609909 7f9276a63700 0 osd.0 39538 ignoring osdmap until we have initialized 2015-12-07 19:02:09.403148 7f9268ef3700 1 osd.0 pg_epoch: 39538 pg[8.ff8( v 29836'11734 (15269'8693,29836'11734] local-les=19376
n=728 ec=4388 les/c/f 19376/19392/0 39491/39491/39491) [4] r=-1 lpr=39493 pi=19328-39490/4 crt=29836'11732 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.404974 7f92696f4700 1 osd.0 pg_epoch: 39538 pg[8.f7f( v 29322'12104 (15269'8967,29322'12104] local-les=20939
n=775 ec=4388 les/c/f 20939/20939/0 39492/39492/39492) [5] r=-1 lpr=39493 pi=19359-39491/4 crt=29322'12095 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.405181 7f9268ef3700 1 osd.0 pg_epoch: 39538 pg[8.fe6( v 29826'12135 (15269'9073,29826'12135] local-les=39037
n=801 ec=4388 les/c/f 39037/39038/0 39522/39522/39522) [6] r=-1 lpr=39522 pi=36394-39521/15 crt=29322'12125 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.407047 7f92696f4700 1 osd.0 pg_epoch: 39538 pg[8.f71( v 29826'11670 (15269'8574,29826'11670] local-les=38783
n=732 ec=4388 les/c/f 38783/38783/0 39513/39513/39513) [5] r=-1 lpr=39513 pi=38782-39512/9 crt=29322'11659 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.408078 7f92696f4700 1 osd.0 pg_epoch: 39538 pg[8.f68( v 29836'12136 (15269'9105,29836'12136] local-les=38443
n=795 ec=4388 les/c/f 38443/38445/0 39527/39527/39527) [5] r=-1 lpr=39527 pi=38282-39526/16 crt=29836'12129 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.685544 7f9268ef3700 1 osd.0 pg_epoch: 39538 pg[8.fe4( v 29836'12151 (15269'9151,29836'12151] local-les=37035
n=794 ec=4388 les/c/f 37035/37035/0 39493/39493/39493) [6] r=-1 lpr=39493 pi=35488-39492/10 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.688751 7f9268ef3700 1 osd.0 pg_epoch: 39538 pg[8.fdb( v 29826'11846 (15269'8846,29826'11846] local-les=38885
n=773 ec=4388 les/c/f 38885/38886/0 39513/39513/39513) [4] r=-1 lpr=39513 pi=38884-39512/9 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.691616 7f92696f4700 1 osd.0 pg_epoch: 39538 pg[8.f64( v 29826'11980 (15269'8980,29826'11980] lb MIN
(NIBBLEWISE) local-les=30663 n=0 ec=4388 les/c/f 30575/30579/0 39474/39474/39474) [] r=-1 lpr=39481 pi=30569-39473/60 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:09.694220 7f9268ef3700 1 osd.0 pg_epoch: 39538 pg[8.fd7( v 29836'11699 (15269'8699,29836'11699] local-les=38371
n=743 ec=4388 les/c/f 38371/38428/0 39492/39492/39492) [5] r=-1 lpr=39493 pi=35482-39491/16 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray ............... 2015-12-07 19:02:11.056343 7f9294de8940 0 osd.0 39538 done with init, starting boot process 2015-12-07 19:02:11.059896 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.ffe( v 29826'12261 (15269'9253,29826'12261] local-les=20071
n=798 ec=4388 les/c/f 20071/20073/0 39539/39539/39539) [4] r=-1 lpr=39539 pi=20026-39538/31 crt=29322'12252 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.060296 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.fea( v 29322'11835 (15269'8772,29322'11835] local-les=39403
n=786 ec=4388 les/c/f 39403/39453/0 39539/39539/39539) [] r=-1 lpr=39539 pi=30699-39538/31 crt=29322'11826 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.060665 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.fd2( v 29836'11729 (15269'8729,29836'11729] lb MIN
(NIBBLEWISE) local-les=35859 n=0 ec=4388 les/c/f 38373/35518/0 39539/39539/39539) [5] r=-1 lpr=39539 pi=35482-39538/45 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.060926 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.fbe( v 29836'11834 (15269'8834,29836'11834] local-les=37200
n=783 ec=4388 les/c/f 37200/37201/0 39539/39539/39539) [4] r=-1 lpr=39539 pi=37199-39538/43 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.061298 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.fa4( v 29836'11887 (15269'8855,29836'11887] local-les=39403
n=780 ec=4388 les/c/f 39403/39453/0 39539/39539/39539) [] r=-1 lpr=39539 pi=30699-39538/31 crt=29836'11881 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.061560 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.f9e( v 29836'11178 (15269'8080,29836'11178] local-les=39403
n=701 ec=4388 les/c/f 39403/39453/0 39539/39539/39539) [] r=-1 lpr=39539 pi=30699-39538/31 crt=29826'11169 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.061771 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.f97( v 29836'11738 (15269'8738,29836'11738] lb MIN
(NIBBLEWISE) local-les=38476 n=0 ec=4388 les/c/f 38476/38331/0 39539/39539/39539) [6] r=-1 lpr=39539 pi=38269-39538/41 crt=0'0 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.063913 7f9268ef3700 1 osd.0 pg_epoch: 39539 pg[8.efd( v 29826'12231 (15269'9230,29826'12231] local-les=19720
n=825 ec=4388 les/c/f 19720/19720/0 39539/39539/39539) [5] r=-1 lpr=39539 pi=19719-39538/32 crt=29826'12223 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.064463 7f9268ef3700 1 osd.0 pg_epoch: 39539 pg[8.ef3( v 29836'11462 (15269'8388,29836'11462] local-les=39094
n=752 ec=4388 les/c/f 39094/39096/0 39539/39539/39539) [5] r=-1 lpr=39539 pi=39093-39538/38 crt=29836'11460 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray 2015-12-07 19:02:11.062736 7f92696f4700 1 osd.0 pg_epoch: 39539 pg[8.f6b( v 29836'11680 (15269'8628,29836'11680] local-les=20072
n=792 ec=4388 les/c/f 20072/20073/0 39539/39539/39539) [4] r=-1 lpr=39539 pi=20067-39538/32 crt=29836'11676 lcod 0'0 inactive NOTIFY NIBBLEWISE] state<Start>: transitioning to Stray ............... 2015-12-07 19:02:11.262515 7f9269ef5700 1 osd.0 39539 osdmap indicates one or more pre-v0.94.4 hammer OSDs is running 2015-12-07 19:02:11.263866 7f9269ef5700 1 osd.0 39539 osdmap indicates one or more pre-v0.94.4 hammer OSDs is running 2015-12-07 19:05:34.951752 7f9269ef5700 1 osd.0 39540 osdmap indicates one or more pre-v0.94.4 hammer OSDs is running 2015-12-07 19:05:41.844302 7f925eedf700 -1 osd.0 39540 *** Got signal Terminated *** 2015-12-07 19:05:41.844356 7f925eedf700 0 osd.0 39540 prepare_to_stop starting shutdown 2015-12-07 19:05:41.844979 7f925eedf700 -1 osd.0 39540 shutdown 2015-12-07 19:05:41.848297 7f925eedf700 20 osd.0 39540 kicking pg 8.ffe ........... 2015-12-07 19:05:41.882677 7f925eedf700 20 osd.0 39540 kicking pg 8.acc 2015-12-07 19:05:41.882679 7f925eedf700 30 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] lock 2015-12-07 19:05:41.882694 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] on_shutdown 2015-12-07 19:05:41.882708 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] cancel_copy_ops 2015-12-07 19:05:41.882721 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] cancel_flush_ops 2015-12-07 19:05:41.882734 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] cancel_proxy_ops 2015-12-07 19:05:41.882748 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] on_change 2015-12-07 19:05:41.882762 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] clear_primary_state 2015-12-07 19:05:41.882776 7f925eedf700 20 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] agent_stop 2015-12-07 19:05:41.882789 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] cancel_recovery 2015-12-07 19:05:41.882802 7f925eedf700 10 osd.0 pg_epoch: 39540 pg[8.acc( v 29322'12257 (15269'9238,29322'12257] local-les=35775
n=823 ec=4388 les/c/f 35775/35859/0 39540/39540/39540) [6] r=-1 lpr=39540 pi=35643-39539/39 crt=29322'12254 lcod 0'0 inactive NOTIFY NIBBLEWISE] clear_recovery_state 2015-12-07 19:05:41.882849 7f925eedf700 20 osd.0 39540 kicking pg 8.ac8 .................. 2015-12-07 19:05:42.041877 7f928a36a700 5 osd.0 39540 tick 2015-12-07 19:05:42.041885 7f928a36a700 10 osd.0 39540 do_waiters -- start 2015-12-07 19:05:42.041888 7f928a36a700 10 osd.0 39540 do_waiters -- finish 2015-12-07 19:05:42.042119 7f925eedf700 10 osd.0 39540 recovery tp stopped 2015-12-07 19:05:42.043075 7f925eedf700 10 osd.0 39540 osd tp stopped 2015-12-07 19:05:42.669366 7f9289b69700 5 osd.0 39540 tick_without_osd_lock 2015-12-07 19:05:42.669413 7f9289b69700 20 osd.0 39540 scrub_random_backoff lost coin flip, randomly backing off From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Claes Sahlström Hi all, I have started to dig into this again, too much work stopped me from digging further and I had the machines shut down
since, anyway. My problem started when I upgraded to Infernalis from Hammer (I am quite certain it was at least 0.94.4) then my OSDs can´t join any more. I am running on Ubuntu 14.04. I guess rolling back is not an option? My setup is 3 HP microservers named black, orange and purple. All 3 servers have one mon each and four OSDs. It has been suggested to download a version released by Sage that might fix my problem, but I am not sure where the archives
are. Looking at the mon log I see: 2015-12-05 19:08:49.895681 7f68852a9700 10
mon.black@0(leader).pg v8253363 check_osd_map -- osdmap not readable, waiting Also this might give a clue to what is happening, when I try to change the logging for the OSD I get an error: ceph tell osd.0 injectargs '--debug-osd 0/5' Error ENXIO: problem getting command descriptions from osd.0 More from the logs, I marked osd.0 as in at
Dec 5 19:08:49 CET 2015 I hope my data is still ok, I am not in a hurry to get it back and prefer a safe solution to a quick one
J I am very thankful for any help or suggestions. Since all this have just worked for way over a year then I am quite rusty
when it comes to figuring out what is wrong and my Linux skills are not enough to figure out what has gone wrong. I tried to only keep what it in the logs from when I tried to set the osd.0 to in. In the log for the OSD when I start it and try to get it to join I get: 2015-12-05 19:08:18.319849 7feef41b5940 0 ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299), process ceph-osd,
pid 4249 2015-12-05 19:08:18.414650 7feef41b5940 0 filestore(/ceph/osd.0) backend xfs (magic 0x58465342) 2015-12-05 19:08:18.415996 7feef41b5940 0 genericfilestorebackend(/ceph/osd.0) detect_features: FIEMAP ioctl is disabled
via 'filestore fiemap' config option 2015-12-05 19:08:18.416033 7feef41b5940 0 genericfilestorebackend(/ceph/osd.0) detect_features: SEEK_DATA/SEEK_HOLE
is disabled via 'filestore seek data hole' config option 2015-12-05 19:08:18.416090 7feef41b5940 0 genericfilestorebackend(/ceph/osd.0) detect_features: splice is supported 2015-12-05 19:08:18.418819 7feef41b5940 0 genericfilestorebackend(/ceph/osd.0) detect_features: syncfs(2) syscall
fully supported (by glibc and kernel) 2015-12-05 19:08:18.419136 7feef41b5940 0 xfsfilestorebackend(/ceph/osd.0) detect_features: extsize is supported and
your kernel >= 3.5 2015-12-05 19:08:18.527107 7feef41b5940 0 filestore(/ceph/osd.0) mount: enabling WRITEAHEAD journal mode: checkpoint
is not enabled 2015-12-05 19:08:18.637889 7feef41b5940 1 journal _open /dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size
4096 bytes, directio = 1, aio = 1 2015-12-05 19:08:18.667396 7feef41b5940 1 journal _open /dev/black/journal-osd.0 fd 19: 23622320128 bytes, block size
4096 bytes, directio = 1, aio = 1 2015-12-05 19:08:18.720872 7feef41b5940 1 filestore(/ceph/osd.0) upgrade 2015-12-05 19:08:18.742447 7feef41b5940 0 <cls> cls/cephfs/cls_cephfs.cc:136: loading cephfs_size_scan 2015-12-05 19:08:18.744275 7feef41b5940 0 <cls> cls/hello/cls_hello.cc:305: loading cls_hello 2015-12-05 19:08:18.761847 7feef41b5940 0 osd.0 39530 crush map has features 1107558400, adjusting msgr requires for
clients 2015-12-05 19:08:18.761883 7feef41b5940 0 osd.0 39530 crush map has features 1107558400 was 8705, adjusting msgr requires
for mons 2015-12-05 19:08:18.761899 7feef41b5940 0 osd.0 39530 crush map has features 1107558400, adjusting msgr requires for
osds 2015-12-05 19:08:21.301437 7feef41b5940 0 osd.0 39530 load_pgs 2015-12-05 19:09:04.080813 7feef41b5940 0 osd.0 39530 load_pgs opened 1230 pgs 2015-12-05 19:09:04.118602 7feef41b5940 -1 osd.0 39530 log_to_monitors {default=true} 2015-12-05 19:09:04.135379 7feed5f0e700 0 osd.0 39530 ignoring osdmap until we have initialized 2015-12-05 19:09:05.867889 7feef41b5940 0 osd.0 39530 done with init, starting boot process In the logs of the mons I get: 2015-12-05 19:08:49.635074 7f688262e700 10
mon.black@0(leader).log v9157752 logging 2015-12-05 19:08:49.633694 mon.2 172.16.0.203:6789/0 27 : audit [INF] from='client.? 172.16.0.201:0/3894299556' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]:
dispatch 2015-12-05 19:08:49.635184 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218c40 for mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.635188 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.635196 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.635198 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.635199 7f688262e700 20 allow all 2015-12-05 19:08:49.635201 7f688262e700 10
mon.black@0(leader) e3 received forwarded message from client.524843 172.16.0.201:0/3894299556 via mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.635207 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.635209 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.635210 7f688262e700 20 allow all 2015-12-05 19:08:49.635214 7f688262e700 10
mon.black@0(leader) e3 caps are allow * 2015-12-05 19:08:49.635216 7f688262e700 10
mon.black@0(leader) e3 entity name 'client.admin' type 8 2015-12-05 19:08:49.635218 7f688262e700 10
mon.black@0(leader) e3 mesg 0x7f688fb10800 from 172.16.0.203:6789/0 2015-12-05 19:08:49.635240 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f21b480 for client.524843 :/0 2015-12-05 19:08:49.635243 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.635301 7f688262e700 0
mon.black@0(leader) e3 handle_command mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 2015-12-05 19:08:49.635337 7f688262e700 20 is_capable service=osd command=osd in read write on cap allow * 2015-12-05 19:08:49.635340 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.635341 7f688262e700 20 allow all 2015-12-05 19:08:49.635342 7f688262e700 10
mon.black@0(leader) e3 _allowed_command capable 2015-12-05 19:08:49.635353 7f688262e700 0 log_channel(audit) log [INF] : from='client.524843 :/0' entity='client.admin'
cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2015-12-05 19:08:49.635453 7f688262e700 10
mon.black@0(leader).osd e39530 preprocess_query mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 from client.524843 172.16.0.201:0/3894299556 2015-12-05 19:08:49.635502 7f688262e700 7
mon.black@0(leader).osd e39530 prepare_update mon_command({"prefix": "osd in", "ids": ["0"]} v 0) v1 from client.524843 172.16.0.201:0/3894299556 2015-12-05 19:08:49.637872 7f688262e700 10
mon.black@0(leader).osd e39530 should_propose 2015-12-05 19:08:49.637997 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218a80 for mon.0 172.16.0.201:6789/0 2015-12-05 19:08:49.638004 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.638034 7f688262e700 10
mon.black@0(leader).log v9157752 preprocess_query log(1 entries) v1 from mon.0 172.16.0.201:6789/0 2015-12-05 19:08:49.638060 7f688262e700 10
mon.black@0(leader).log v9157752 preprocess_log log(1 entries) v1 from mon.0 2015-12-05 19:08:49.638064 7f688262e700 20 is_capable service=log command= write on cap allow * 2015-12-05 19:08:49.638067 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.638069 7f688262e700 20 allow all 2015-12-05 19:08:49.638089 7f688262e700 10
mon.black@0(leader).log v9157752 prepare_update log(1 entries) v1 from mon.0 172.16.0.201:6789/0 2015-12-05 19:08:49.638100 7f688262e700 10
mon.black@0(leader).log v9157752 prepare_log log(1 entries) v1 from mon.0 2015-12-05 19:08:49.638103 7f688262e700 10
mon.black@0(leader).log v9157752 logging 2015-12-05 19:08:49.635355 mon.0 172.16.0.201:6789/0 29 : audit [INF] from='client.524843 :/0' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2015-12-05 19:08:49.685271 7f6882e2f700 10
mon.black@0(leader).log v9157752 encode_full log v 9157752 2015-12-05 19:08:49.685505 7f6882e2f700 10
mon.black@0(leader).log v9157752 encode_pending v9157753 2015-12-05 19:08:49.690050 7f6882e2f700 10
mon.black@0(leader).osd e39530 encode_pending e 39531 2015-12-05 19:08:49.690092 7f6882e2f700 2
mon.black@0(leader).osd e39530 osd.0 IN 2015-12-05 19:08:49.690698 7f6882e2f700 20
mon.black@0(leader).osd e39530 full_crc 3915856006 inc_crc 3792224963 2015-12-05 19:08:49.691048 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f2188c0 for mon.1 172.16.0.202:6789/0 2015-12-05 19:08:49.691055 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.691064 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.691068 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.691069 7f688262e700 20 allow all 2015-12-05 19:08:49.691071 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.691073 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.691074 7f688262e700 20 allow all 2015-12-05 19:08:49.878891 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218c40 for mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.878904 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.878921 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.878926 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.878929 7f688262e700 20 allow all 2015-12-05 19:08:49.878932 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.878935 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.878937 7f688262e700 20 allow all 2015-12-05 19:08:49.881064 7f68852a9700 10
mon.black@0(leader) e3 refresh_from_paxos 2015-12-05 19:08:49.881688 7f68852a9700 10
mon.black@0(leader).log v9157753 update_from_paxos 2015-12-05 19:08:49.881696 7f68852a9700 10
mon.black@0(leader).log v9157753 update_from_paxos version 9157753 summary v 9157752 2015-12-05 19:08:49.881747 7f68852a9700 10
mon.black@0(leader).log v9157753 update_from_paxos latest full 9157752 2015-12-05 19:08:49.881809 7f68852a9700 7
mon.black@0(leader).log v9157753 update_from_paxos applying incremental log 9157753 2015-12-05 19:08:49.633694 mon.2 172.16.0.203:6789/0 27 : audit [INF] from='client.? 172.16.0.201:0/3894299556' entity='client.admin'
cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2015-12-05 19:08:49.881840 7f68852a9700 20
mon.black@0(leader).log v9157753 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log' 2015-12-05 19:08:49.881882 7f68852a9700 7
mon.black@0(leader).log v9157753 update_from_paxos applying incremental log 9157753 2015-12-05 19:08:49.635355 mon.0 172.16.0.201:6789/0 29 : audit [INF] from='client.524843 :/0' entity='client.admin' cmd=[{"prefix":
"osd in", "ids": ["0"]}]: dispatch 2015-12-05 19:08:49.881901 7f68852a9700 20
mon.black@0(leader).log v9157753 update_from_paxos logging for channel 'audit' to file '/var/log/ceph/ceph.audit.log' 2015-12-05 19:08:49.881925 7f68852a9700 15
mon.black@0(leader).log v9157753 update_from_paxos logging for 1 channels 2015-12-05 19:08:49.881929 7f68852a9700 15
mon.black@0(leader).log v9157753 update_from_paxos channel 'audit' logging 353 bytes 2015-12-05 19:08:49.882845 7f68852a9700 10
mon.black@0(leader).log v9157753 check_subs 2015-12-05 19:08:49.883231 7f68852a9700 10
mon.black@0(leader).auth v14845 update_from_paxos 2015-12-05 19:08:49.883242 7f68852a9700 10
mon.black@0(leader).pg v8253363 map_pg_creates to 0 pgs -- no change 2015-12-05 19:08:49.883247 7f68852a9700 10
mon.black@0(leader).pg v8253363 send_pg_creates to 0 pgs 2015-12-05 19:08:49.883400 7f68852a9700 10
mon.black@0(leader).log v9157753 create_pending v 9157754 2015-12-05 19:08:49.883423 7f68852a9700 7
mon.black@0(leader).log v9157753 _updated_log for mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.883444 7f68852a9700 2
mon.black@0(leader) e3 send_reply 0x7f688f64ca80 0x7f688f3d5680 log(last 27) v1 2015-12-05 19:08:49.883450 7f68852a9700 15
mon.black@0(leader) e3 send_reply routing reply to 172.16.0.203:6789/0 via 172.16.0.203:6789/0 for request log(1 entries) v1 2015-12-05 19:08:49.883531 7f68852a9700 7
mon.black@0(leader).log v9157753 _updated_log for mon.0 172.16.0.201:6789/0 2015-12-05 19:08:49.883548 7f68852a9700 2
mon.black@0(leader) e3 send_reply 0x7f688f64cb60 0x7f688f3d6940 log(last 29) v1 2015-12-05 19:08:49.886246 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218a80 for mon.0 172.16.0.201:6789/0 2015-12-05 19:08:49.886261 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.886323 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.886330 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.886332 7f688262e700 20 allow all 2015-12-05 19:08:49.888142 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f2188c0 for mon.1 172.16.0.202:6789/0 2015-12-05 19:08:49.888346 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.888367 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.888373 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.888375 7f688262e700 20 allow all 2015-12-05 19:08:49.888379 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.888382 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.888384 7f688262e700 20 allow all 2015-12-05 19:08:49.888502 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218c40 for mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.888512 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.888524 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.888528 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.888530 7f688262e700 20 allow all 2015-12-05 19:08:49.888532 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.888535 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.888537 7f688262e700 20 allow all 2015-12-05 19:08:49.889676 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f2188c0 for mon.1 172.16.0.202:6789/0 2015-12-05 19:08:49.889688 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.889703 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.889709 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.889711 7f688262e700 20 allow all 2015-12-05 19:08:49.889714 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.889717 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.889719 7f688262e700 20 allow all 2015-12-05 19:08:49.890685 7f688262e700 20
mon.black@0(leader) e3 _ms_dispatch existing session 0x7f688f218c40 for mon.2 172.16.0.203:6789/0 2015-12-05 19:08:49.890697 7f688262e700 20
mon.black@0(leader) e3 caps allow * 2015-12-05 19:08:49.890712 7f688262e700 20 is_capable service=mon command= read on cap allow * 2015-12-05 19:08:49.890717 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.890720 7f688262e700 20 allow all 2015-12-05 19:08:49.890747 7f688262e700 20 is_capable service=mon command= exec on cap allow * 2015-12-05 19:08:49.890751 7f688262e700 20 allow so far , doing grant allow * 2015-12-05 19:08:49.890753 7f688262e700 20 allow all 2015-12-05 19:08:49.894322 7f68852a9700 10
mon.black@0(leader) e3 refresh_from_paxos 2015-12-05 19:08:49.894834 7f68852a9700 15
mon.black@0(leader).osd e39530 update_from_paxos paxos e 39531, my e 39530 2015-12-05 19:08:49.894899 7f68852a9700 7
mon.black@0(leader).osd e39530 update_from_paxos applying incremental 39531 2015-12-05 19:08:49.895218 7f68852a9700 1
mon.black@0(leader).osd e39531 e39531: 12 osds: 3 up, 4 in 2015-12-05 19:08:49.895663 7f68852a9700 10
mon.black@0(leader).osd e39531 adding osd.0 to down_pending_out map 2015-12-05 19:08:49.895681 7f68852a9700 10
mon.black@0(leader).pg v8253363 check_osd_map -- osdmap not readable, waiting From: Claes Sahlström
Yes I upgraded from Hammer 0.94.4. And "ceph-osd --version" gives the correct version 9.2.0, I think it is a problem with the communication between my OSDs and either the MONs or the other OSDs or maybe both. I will check out those archives from Sage also… I have probably done something wrong, but I cannot figure out what is is. All my upgrades before was smooth and this is quite an old installation I have at home. From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Josef Johansson And if you look through the archives Sage did release a version of Infernalis that fixed if you didn’t do it that way as well. |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
