Hello,
I've been having several issue with monitor processes achieving quorum
with 0.60. The most recent was an assert with the output mon/Monitor.cc:
1105: FAILED assert(0 == "We should never reach this"). I've pasted a
larger set of the log file at the bottom of this email.
Prior to this assert, I ran into trouble where one out of the three
monitors would drop out of quorum with the other two. Clocks are
synchronized properly. Over the past week, I've seen several instances
of a monitor dropping out of quorum, but the process continues to run in
the probing state without ever achieving recovery.
I've seen several odd behaviors as I try to recover from this state:
1) Restarting that monitor that is out of quorum does not fix the issue.
2) If I leave the OSDs running, then do a rolling restart of the
monitors, I can get the original monitor that was our of quorum to come
back into quorum, but then one of the other monitors falls out or
quorum. I've repeated this issue several times. In this state, I've
always seen one leader, one peon, and one probing. The order in which I
start the monitors seems to determine which will end up in the probing
state. I have had each of the three end up probing during my investigation.
3) If I stop all mon and osd daemons, I can bring the three mons back up
and they achieve quorum immediately. Then I can restart the OSDs. I've
sometimes seen one mon fall back out of quorum soon afterward.
4) I've seen a situation where mon.a and mon.b were concurrently
reporting as the leader while mon.c was a peon. Ceph -s would report one
of the leaders and the peon as part of the quorum.
Interestingly, after the assert shown in the log below, I restarted that
monitor, quorum was immediately achieved between all three, and it has
been stable for the past 18 hours.
Any ideas?
Thanks,
Mike
root@xxxx:~# tail -500 /var/log/ceph/ceph-mon.a.log
-349> 2013-04-19 20:46:40.206756 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (4181272333 0 0) 0x545ac00 con 0x3fa6b00
-348> 2013-04-19 20:46:40.206882 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.6:6800/30645 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1be00 con 0x3fa6b00
-347> 2013-04-19 20:46:40.206900 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 1 ==== auth(proto 0 26 bytes epoch 1) v1
==== 56+0+0 (600800006 0 0) 0x545ae00 con 0x3fa6f20
-346> 2013-04-19 20:46:40.206988 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.5:6803/14788 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1bc00 con 0x3fa6f20
-345> 2013-04-19 20:46:40.207006 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5808 1 ==== auth(proto 0 25 bytes epoch 0) v1
==== 55+0+0 (1030628714 0 0) 0x545b000 con 0x3fa7080
-344> 2013-04-19 20:46:40.207063 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 -- mon_map v1 -- ?+0 0x505cd20 con 0x3fa7080
-343> 2013-04-19 20:46:40.207162 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1ba00 con 0x3fa7080
-342> 2013-04-19 20:46:40.207176 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (1687163837 0 0) 0x545b200 con 0x3fa6420
-341> 2013-04-19 20:46:40.207244 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6800/5606 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1b800 con 0x3fa6420
-340> 2013-04-19 20:46:40.207263 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (1938252949 0 0) 0x545b400 con 0x3fa66e0
-339> 2013-04-19 20:46:40.207320 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6800/1216 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1b600 con 0x3fa66e0
-338> 2013-04-19 20:46:40.207336 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (4253028184 0 0) 0x545b600 con 0x3fa7600
-337> 2013-04-19 20:46:40.207407 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6806/1386 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1b400 con 0x3fa7600
-336> 2013-04-19 20:46:40.207432 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 1 ==== auth(proto 0 26 bytes epoch 1) v1
==== 56+0+0 (4019298683 0 0) 0x545b800 con 0x3fa7a20
-335> 2013-04-19 20:46:40.207555 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.3:6806/3242 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1b200 con 0x3fa7a20
-334> 2013-04-19 20:46:40.207571 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (1142887624 0 0) 0x3e20000 con 0x3fa7340
-333> 2013-04-19 20:46:40.207690 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6806/5787 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1b000 con 0x3fa7340
-332> 2013-04-19 20:46:40.207801 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (1368324502 0 0) 0x5897000 con 0x3fa7760
-331> 2013-04-19 20:46:40.207863 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.68:6803/30491 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e1ae00 con 0x3fa7760
-330> 2013-04-19 20:46:40.207881 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (16545660 0 0) 0x5897400 con 0x3edadc0
-329> 2013-04-19 20:46:40.208108 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.70:6800/17218 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897000 con 0x3edadc0
-328> 2013-04-19 20:46:40.208129 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5798 2 ==== auth(proto 2 32 bytes epoch 0) v1
==== 62+0+0 (2951758062 0 0) 0x609e000 con 0x3edb080
-327> 2013-04-19 20:46:40.208244 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5798 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897400 con 0x3edb080
-326> 2013-04-19 20:46:40.208266 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (2286058006 0 0) 0x5d81c00 con 0x3eda580
-325> 2013-04-19 20:46:40.208485 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.70:6806/17398 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x609e000 con 0x3eda580
-324> 2013-04-19 20:46:40.208510 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (3651642919 0 0) 0x3eb0000 con 0x3edab00
-323> 2013-04-19 20:46:40.208735 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6800/5691 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d81c00 con 0x3edab00
-322> 2013-04-19 20:46:40.208756 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (3132364415 0 0) 0x5ec8000 con 0x3fa71e0
-321> 2013-04-19 20:46:40.208905 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6806/5892 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3eb0000 con 0x3fa71e0
-320> 2013-04-19 20:46:40.208924 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (4195045578 0 0) 0x5ec9600 con 0x3fa74a0
-319> 2013-04-19 20:46:40.209068 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.136:6803/16299 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8000 con 0x3fa74a0
-318> 2013-04-19 20:46:40.209086 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (3529293791 0 0) 0x5ec8400 con 0x3fa7b80
-317> 2013-04-19 20:46:40.209240 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.73:6803/20825 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9600 con 0x3fa7b80
-316> 2013-04-19 20:46:40.209259 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (6776368 0 0) 0x5ec9200 con 0x3fa7ce0
-315> 2013-04-19 20:46:40.209458 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.137:6803/5123 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8400 con 0x3fa7ce0
-314> 2013-04-19 20:46:40.209479 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (2398714386 0 0) 0x5ec9000 con 0x3fa6b00
-313> 2013-04-19 20:46:40.209692 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.6:6800/30645 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9200 con 0x3fa6b00
-312> 2013-04-19 20:46:40.209711 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (2027200610 0 0) 0x5ec8a00 con 0x3fa6f20
-311> 2013-04-19 20:46:40.209855 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.5:6803/14788 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9000 con 0x3fa6f20
-310> 2013-04-19 20:46:40.209874 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5808 2 ==== auth(proto 2 32 bytes epoch 0) v1
==== 62+0+0 (1151847595 0 0) 0x3e20600 con 0x3fa7080
-309> 2013-04-19 20:46:40.209990 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8a00 con 0x3fa7080
-308> 2013-04-19 20:46:40.210010 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (652860118 0 0) 0x5d1ba00 con 0x3fa6420
-307> 2013-04-19 20:46:40.210158 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6800/5606 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e20600 con 0x3fa6420
-306> 2013-04-19 20:46:40.210178 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (360523710 0 0) 0x3e20200 con 0x3fa66e0
-305> 2013-04-19 20:46:40.210328 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6800/1216 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1ba00 con 0x3fa66e0
-304> 2013-04-19 20:46:40.210350 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (1701575060 0 0) 0x5897000 con 0x3fa7600
-303> 2013-04-19 20:46:40.210500 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6806/1386 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e20200 con 0x3fa7600
-302> 2013-04-19 20:46:40.210523 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (3159492289 0 0) 0x5d1ac00 con 0x3fa7a20
-301> 2013-04-19 20:46:40.210674 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.3:6806/3242 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897000 con 0x3fa7a20
-300> 2013-04-19 20:46:40.210692 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (221685237 0 0) 0x5d1b000 con 0x3fa7340
-299> 2013-04-19 20:46:40.210836 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6806/5787 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1ac00 con 0x3fa7340
-298> 2013-04-19 20:46:40.210855 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (4015086919 0 0) 0x5897400 con 0x3fa7760
-297> 2013-04-19 20:46:40.210999 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.68:6803/30491 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1b000 con 0x3fa7760
-296> 2013-04-19 20:46:40.211019 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (3701228685 0 0) 0x5897600 con 0x3edadc0
-295> 2013-04-19 20:46:40.211194 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.70:6800/17218 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897400 con 0x3edadc0
-294> 2013-04-19 20:46:40.211214 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5798 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (1256713350 0 0) 0x5897e00 con 0x3edb080
-293> 2013-04-19 20:46:40.211381 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5798 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897600 con 0x3edb080
-292> 2013-04-19 20:46:40.211400 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (207928548 0 0) 0x5d81e00 con 0x3eda580
-291> 2013-04-19 20:46:40.211605 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.70:6806/17398 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5897e00 con 0x3eda580
-290> 2013-04-19 20:46:40.211626 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (1454381923 0 0) 0x5d81a00 con 0x3edab00
-289> 2013-04-19 20:46:40.211800 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6800/5691 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d81e00 con 0x3edab00
-288> 2013-04-19 20:46:40.211820 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (3459110416 0 0) 0x5ec9800 con 0x3fa71e0
-287> 2013-04-19 20:46:40.212023 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6806/5892 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d81a00 con 0x3fa71e0
-286> 2013-04-19 20:46:40.212042 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (2627412008 0 0) 0x5ec8800 con 0x3fa74a0
-285> 2013-04-19 20:46:40.212212 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.136:6803/16299 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9800 con 0x3fa74a0
-284> 2013-04-19 20:46:40.212232 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (3925232558 0 0) 0x5ec8200 con 0x3fa7b80
-283> 2013-04-19 20:46:40.212415 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.73:6803/20825 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8800 con 0x3fa7b80
-282> 2013-04-19 20:46:40.212435 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (2756848682 0 0) 0x5ec9400 con 0x3fa7ce0
-281> 2013-04-19 20:46:40.212613 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.137:6803/5123 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8200 con 0x3fa7ce0
-280> 2013-04-19 20:46:40.212633 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (2119542918 0 0) 0x5ec8e00 con 0x3fa6b00
-279> 2013-04-19 20:46:40.212879 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.6:6800/30645 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9400 con 0x3fa6b00
-278> 2013-04-19 20:46:40.212899 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (737010031 0 0) 0x5ec8c00 con 0x3fa6f20
-277> 2013-04-19 20:46:40.213080 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.5:6803/14788 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8e00 con 0x3fa6f20
-276> 2013-04-19 20:46:40.213226 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5808 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (1315457472 0 0) 0x3e20400 con 0x3fa7080
-275> 2013-04-19 20:46:40.213425 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8c00 con 0x3fa7080
-274> 2013-04-19 20:46:40.213446 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (3836105199 0 0) 0x3e28800 con 0x3fa6420
-273> 2013-04-19 20:46:40.213651 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6800/5606 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e20400 con 0x3fa6420
-272> 2013-04-19 20:46:40.213672 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (2934577959 0 0) 0x5ec8600 con 0x3fa66e0
-271> 2013-04-19 20:46:40.213848 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6800/1216 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e28800 con 0x3fa66e0
-270> 2013-04-19 20:46:40.213867 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (1400839318 0 0) 0x5ec9a00 con 0x3fa7600
-269> 2013-04-19 20:46:40.214046 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6806/1386 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec8600 con 0x3fa7600
-268> 2013-04-19 20:46:40.214066 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (2052111959 0 0) 0x5d1b600 con 0x3fa7a20
-267> 2013-04-19 20:46:40.214244 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.3:6806/3242 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5ec9a00 con 0x3fa7a20
-266> 2013-04-19 20:46:40.214264 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (2933205665 0 0) 0x5d1ae00 con 0x3fa7340
-265> 2013-04-19 20:46:40.214434 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6806/5787 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1b600 con 0x3fa7340
-264> 2013-04-19 20:46:40.214454 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (87761700 0 0) 0x5897200 con 0x3fa7760
-263> 2013-04-19 20:46:40.214624 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.68:6803/30491 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1ae00 con 0x3fa7760
-262> 2013-04-19 20:46:40.214644 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de3340 con 0x3edadc0
-261> 2013-04-19 20:46:40.214814 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x5897200
-260> 2013-04-19 20:46:40.214853 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2000
-259> 2013-04-19 20:46:40.214864 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5798 4 ==== mon_subscribe({monmap=0+}) v2 ====
23+0+0 (1620593354 0 0) 0x5de3880 con 0x3edb080
-258> 2013-04-19 20:46:40.214878 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5798 -- mon_map v1 -- ?+0 0x505cb40 con 0x3edb080
-257> 2013-04-19 20:46:40.214911 7fa90880b700 1 -- 10.1.0.3:6789/0
--> client.? 10.1.0.67:0/5798 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac21a0
-256> 2013-04-19 20:46:40.214927 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x5de3a40 con 0x3eda580
-255> 2013-04-19 20:46:40.215038 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x3e1ac00
-254> 2013-04-19 20:46:40.215055 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2340
-253> 2013-04-19 20:46:40.215066 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de3c00 con 0x3edab00
-252> 2013-04-19 20:46:40.215161 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1aa00
-251> 2013-04-19 20:46:40.215184 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac24e0
-250> 2013-04-19 20:46:40.215200 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x3c78000 con 0x3fa71e0
-249> 2013-04-19 20:46:40.215311 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x3e1a800
-248> 2013-04-19 20:46:40.215333 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2680
-247> 2013-04-19 20:46:40.215362 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c781c0 con 0x3fa74a0
-246> 2013-04-19 20:46:40.215423 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- osd_map(869..869 src has 368..869)
v3 -- ?+0 0x3e1a600
-245> 2013-04-19 20:46:40.215437 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2820
-244> 2013-04-19 20:46:40.215446 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78a80 con 0x3fa7b80
-243> 2013-04-19 20:46:40.215498 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1a400
-242> 2013-04-19 20:46:40.215511 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac29c0
-241> 2013-04-19 20:46:40.215520 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78fc0 con 0x3fa7ce0
-240> 2013-04-19 20:46:40.215577 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.64 10.1.0.137:6803/5123 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1a200
-239> 2013-04-19 20:46:40.215591 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.64 10.1.0.137:6803/5123 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2b60
-238> 2013-04-19 20:46:40.215600 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x3c79500 con 0x3fa6b00
-237> 2013-04-19 20:46:40.215689 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.10 10.1.0.6:6800/30645 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x3e1a000
-236> 2013-04-19 20:46:40.215703 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.10 10.1.0.6:6800/30645 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2d00
-235> 2013-04-19 20:46:40.215713 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c79dc0 con 0x3fa6f20
-234> 2013-04-19 20:46:40.215774 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.7 10.1.0.5:6803/14788 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1de00
-233> 2013-04-19 20:46:40.215788 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.7 10.1.0.5:6803/14788 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac2ea0
-232> 2013-04-19 20:46:40.215797 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5808 4 ==== mon_subscribe({monmap=0+}) v2 ====
23+0+0 (1620593354 0 0) 0x63cc380 con 0x3fa7080
-231> 2013-04-19 20:46:40.215809 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 -- mon_map v1 -- ?+0 0x505d2c0 con 0x3fa7080
-230> 2013-04-19 20:46:40.215819 7fa90880b700 1 -- 10.1.0.3:6789/0
--> client.? 10.1.0.67:0/5808 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac3040
-229> 2013-04-19 20:46:40.215843 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63cc540 con 0x3fa6420
-228> 2013-04-19 20:46:40.215909 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.36 10.1.0.72:6800/5606 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1dc00
-227> 2013-04-19 20:46:40.215923 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.36 10.1.0.72:6800/5606 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7fba0
-226> 2013-04-19 20:46:40.215933 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x63cc700 con 0x3fa66e0
-225> 2013-04-19 20:46:40.215999 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.45 10.1.0.131:6800/1216 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x3e1da00
-224> 2013-04-19 20:46:40.216012 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.45 10.1.0.131:6800/1216 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f380
-223> 2013-04-19 20:46:40.216019 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x63cc8c0 con 0x3fa7600
-222> 2013-04-19 20:46:40.216083 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.47 10.1.0.131:6806/1386 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x3e1d800
-221> 2013-04-19 20:46:40.216098 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.47 10.1.0.131:6806/1386 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f1e0
-220> 2013-04-19 20:46:40.216107 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63cce00 con 0x3fa7a20
-219> 2013-04-19 20:46:40.216158 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.2 10.1.0.3:6806/3242 -- osd_map(869..869 src has 368..869) v3 --
?+0 0x3e1d600
-218> 2013-04-19 20:46:40.216171 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.2 10.1.0.3:6806/3242 -- mon_subscribe_ack(300s) v1 -- ?+0 0x2f7eea0
-217> 2013-04-19 20:46:40.216179 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63cd340 con 0x3fa7340
-216> 2013-04-19 20:46:40.216229 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.38 10.1.0.72:6806/5787 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x3e1d400
-215> 2013-04-19 20:46:40.216241 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.38 10.1.0.72:6806/5787 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7fa00
-214> 2013-04-19 20:46:40.216250 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x5de28c0 con 0x3fa7760
-213> 2013-04-19 20:46:40.216325 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.25 10.1.0.68:6803/30491 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x3e1d200
-212> 2013-04-19 20:46:40.216338 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.25 10.1.0.68:6803/30491 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f860
-211> 2013-04-19 20:46:40.216347 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 5 ==== pg_stats(10 pgs tid 897 v 869) v1
==== 4086+0+0 (3284441714 0 0) 0x6516b40 con 0x3edadc0
-210> 2013-04-19 20:46:40.216496 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5798 5 ==== mon_command(auth get-or-create
client.admin mon allow * osd allow * mds allow v 0) v1 ==== 131+0+0
(2360349124 0 0) 0x505cf00 con 0x3edb080
-209> 2013-04-19 20:46:40.216514 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-208> 2013-04-19 20:46:40.216588 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5798 --
mon_command_ack([auth,get-or-create,client.admin,mon,allow *,osd,allow
*,mds,allow]=-13 access denied v34578) v1 -- ?+0 0x505dc20 con 0x3edb080
-207> 2013-04-19 20:46:40.216606 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 5 ==== pg_stats(17 pgs tid 884 v 869) v1
==== 6753+0+0 (652440185 0 0) 0x2f54fc0 con 0x3eda580
-206> 2013-04-19 20:46:40.216718 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 5 ==== pg_stats(18 pgs tid 946 v 869) v1
==== 7122+0+0 (2483024403 0 0) 0x2f55b00 con 0x3edab00
-205> 2013-04-19 20:46:40.216828 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 5 ==== pg_stats(15 pgs tid 847 v 869) v1
==== 5979+0+0 (2684676607 0 0) 0x505bd40 con 0x3fa71e0
-204> 2013-04-19 20:46:40.216924 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 5 ==== pg_stats(16 pgs tid 880 v 869)
v1 ==== 6360+0+0 (88892783 0 0) 0x505ad80 con 0x3fa74a0
-203> 2013-04-19 20:46:40.217024 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 5 ==== pg_stats(27 pgs tid 873 v 869) v1
==== 10563+0+0 (191424931 0 0) 0x505a900 con 0x3fa7b80
-202> 2013-04-19 20:46:40.217143 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 5 ==== pg_stats(9 pgs tid 814 v 869) v1
==== 3693+0+0 (1438016683 0 0) 0x505afc0 con 0x3fa7ce0
-201> 2013-04-19 20:46:40.217234 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 5 ==== pg_stats(13 pgs tid 688 v 869) v1
==== 5229+0+0 (2323473417 0 0) 0x505b8c0 con 0x3fa6b00
-200> 2013-04-19 20:46:40.217333 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 5 ==== pg_stats(21 pgs tid 625 v 869) v1
==== 8277+0+0 (175938291 0 0) 0x5ec6000 con 0x3fa6f20
-199> 2013-04-19 20:46:40.217335 7fa9041d3700 2 -- 10.1.0.3:6789/0
>> 10.1.0.67:0/5798 pipe(0x5bf6780 sd=59 :6789 s=2 pgs=276 cs=1
l=1).reader couldn't read tag, Success
-198> 2013-04-19 20:46:40.217354 7fa9041d3700 2 -- 10.1.0.3:6789/0
>> 10.1.0.67:0/5798 pipe(0x5bf6780 sd=59 :6789 s=2 pgs=276 cs=1
l=1).fault 0: Success
-197> 2013-04-19 20:46:40.217449 7fa90880b700 1 -- 10.1.0.3:6789/0
<== client.? 10.1.0.67:0/5808 5 ==== mon_command(auth get-or-create
client.admin mon allow * osd allow * mds allow v 0) v1 ==== 131+0+0
(2360349124 0 0) 0x505d0e0 con 0x3fa7080
-196> 2013-04-19 20:46:40.217463 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-195> 2013-04-19 20:46:40.217526 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.67:0/5808 --
mon_command_ack([auth,get-or-create,client.admin,mon,allow *,osd,allow
*,mds,allow]=-13 access denied v34578) v1 -- ?+0 0x505cf00 con 0x3fa7080
-194> 2013-04-19 20:46:40.217545 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 5 ==== pg_stats(15 pgs tid 776 v 869) v1
==== 5991+0+0 (1078691406 0 0) 0x5ec66c0 con 0x3fa6420
-193> 2013-04-19 20:46:40.217674 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 5 ==== pg_stats(6 pgs tid 635 v 869) v1
==== 2550+0+0 (1020579911 0 0) 0x5ec6900 con 0x3fa66e0
-192> 2013-04-19 20:46:40.217760 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 5 ==== pg_stats(22 pgs tid 637 v 869) v1
==== 8646+0+0 (1833308224 0 0) 0x5ec6d80 con 0x3fa7600
-191> 2013-04-19 20:46:40.217875 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 5 ==== pg_stats(12 pgs tid 684 v 869) v1
==== 4848+0+0 (571346949 0 0) 0x5ec6b40 con 0x3fa7a20
-190> 2013-04-19 20:46:40.217970 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 5 ==== pg_stats(20 pgs tid 830 v 869) v1
==== 7896+0+0 (929921554 0 0) 0x5ec6fc0 con 0x3fa7340
-189> 2013-04-19 20:46:40.218076 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 5 ==== pg_stats(21 pgs tid 821 v 869) v1
==== 8277+0+0 (1885790742 0 0) 0x5ec7200 con 0x3fa7760
-188> 2013-04-19 20:46:40.218079 7fa9061f3700 2 -- 10.1.0.3:6789/0
>> 10.1.0.67:0/5808 pipe(0x7d16c80 sd=32 :6789 s=2 pgs=276 cs=1
l=1).reader couldn't read tag, Success
-187> 2013-04-19 20:46:40.218098 7fa9061f3700 2 -- 10.1.0.3:6789/0
>> 10.1.0.67:0/5808 pipe(0x7d16c80 sd=32 :6789 s=2 pgs=276 cs=1
l=1).fault 0: Success
-186> 2013-04-19 20:46:40.218249 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de36c0 con 0x3edadc0
-185> 2013-04-19 20:46:40.218311 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x545ae00
-184> 2013-04-19 20:46:40.218395 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f6c0
-183> 2013-04-19 20:46:40.218409 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x5de3dc0 con 0x3eda580
-182> 2013-04-19 20:46:40.218497 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x5a68800
-181> 2013-04-19 20:46:40.218512 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f520
-180> 2013-04-19 20:46:40.218521 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78540 con 0x3edab00
-179> 2013-04-19 20:46:40.218573 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x5a68600
-178> 2013-04-19 20:46:40.218586 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7fd40
-177> 2013-04-19 20:46:40.218595 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x3c788c0 con 0x3fa71e0
-176> 2013-04-19 20:46:40.218662 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x5a68400
-175> 2013-04-19 20:46:40.218676 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3ba0
-174> 2013-04-19 20:46:40.218685 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78e00 con 0x3fa74a0
-173> 2013-04-19 20:46:40.218736 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- osd_map(869..869 src has 368..869)
v3 -- ?+0 0x5a68200
-172> 2013-04-19 20:46:40.218758 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3a00
-171> 2013-04-19 20:46:40.218767 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c79880 con 0x3fa7b80
-170> 2013-04-19 20:46:40.218817 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x5a68000
-169> 2013-04-19 20:46:40.218830 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3860
-168> 2013-04-19 20:46:40.218839 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c79340 con 0x3fa7ce0
-167> 2013-04-19 20:46:40.218897 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.64 10.1.0.137:6803/5123 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x7dc9e00
-166> 2013-04-19 20:46:40.218917 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.64 10.1.0.137:6803/5123 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac21a0
-165> 2013-04-19 20:46:40.218942 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.10 10.1.0.6:6800/30645 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x3c79c00 con 0x3fa6b00
-164> 2013-04-19 20:46:40.219028 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.10 10.1.0.6:6800/30645 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x7dc9c00
-163> 2013-04-19 20:46:40.219042 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.10 10.1.0.6:6800/30645 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3040
-162> 2013-04-19 20:46:40.219051 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63cc1c0 con 0x3fa6f20
-161> 2013-04-19 20:46:40.219101 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.7 10.1.0.5:6803/14788 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x7dc9a00
-160> 2013-04-19 20:46:40.219115 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.7 10.1.0.5:6803/14788 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac2000
-159> 2013-04-19 20:46:40.219123 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.36 10.1.0.72:6800/5606 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63ccc40 con 0x3fa6420
-158> 2013-04-19 20:46:40.219172 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.36 10.1.0.72:6800/5606 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x7dc9800
-157> 2013-04-19 20:46:40.219185 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.36 10.1.0.72:6800/5606 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac24e0
-156> 2013-04-19 20:46:40.219194 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.45 10.1.0.131:6800/1216 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x63cd180 con 0x3fa66e0
-155> 2013-04-19 20:46:40.219264 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.45 10.1.0.131:6800/1216 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x7dc9600
-154> 2013-04-19 20:46:40.219281 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.45 10.1.0.131:6800/1216 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2340
-153> 2013-04-19 20:46:40.219290 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.47 10.1.0.131:6806/1386 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x63cda40 con 0x3fa7600
-152> 2013-04-19 20:46:40.219354 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.47 10.1.0.131:6806/1386 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x7dc9400
-151> 2013-04-19 20:46:40.219367 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.47 10.1.0.131:6806/1386 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3ba0
-150> 2013-04-19 20:46:40.219376 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.2 10.1.0.3:6806/3242 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x63cd6c0 con 0x3fa7a20
-149> 2013-04-19 20:46:40.219426 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.2 10.1.0.3:6806/3242 -- osd_map(869..869 src has 368..869) v3 --
?+0 0x7dc9200
-148> 2013-04-19 20:46:40.219439 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.2 10.1.0.3:6806/3242 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac2680
-147> 2013-04-19 20:46:40.219449 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.38 10.1.0.72:6806/5787 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de2e00 con 0x3fa7340
-146> 2013-04-19 20:46:40.219499 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.38 10.1.0.72:6806/5787 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x7dc9000
-145> 2013-04-19 20:46:40.219511 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.38 10.1.0.72:6806/5787 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3a00
-144> 2013-04-19 20:46:40.219520 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.25 10.1.0.68:6803/30491 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x5de3880 con 0x3fa7760
-143> 2013-04-19 20:46:40.219595 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.25 10.1.0.68:6803/30491 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x7dc8e00
-142> 2013-04-19 20:46:40.219607 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.25 10.1.0.68:6803/30491 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2820
-141> 2013-04-19 20:46:40.219616 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 7 ==== pg_stats(10 pgs tid 898 v 869) v1
==== 4086+0+0 (1252635494 0 0) 0x2f55d40 con 0x3edadc0
-140> 2013-04-19 20:46:40.219701 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 7 ==== pg_stats(17 pgs tid 885 v 869) v1
==== 6753+0+0 (1167113881 0 0) 0x2f54d80 con 0x3eda580
-139> 2013-04-19 20:46:40.219809 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 7 ==== pg_stats(27 pgs tid 874 v 869) v1
==== 10563+0+0 (3735343305 0 0) 0x505ab40 con 0x3fa7b80
-138> 2013-04-19 20:46:40.219907 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 7 ==== pg_stats(9 pgs tid 815 v 869) v1
==== 3693+0+0 (4215792884 0 0) 0x505b680 con 0x3fa7ce0
-137> 2013-04-19 20:46:40.219986 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.7 10.1.0.5:6803/14788 7 ==== pg_stats(21 pgs tid 626 v 869) v1
==== 8277+0+0 (1830736769 0 0) 0x5ec6480 con 0x3fa6f20
-136> 2013-04-19 20:46:40.225900 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.30 10.1.0.70:6800/17218 8 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de3500 con 0x3edadc0
-135> 2013-04-19 20:46:40.225984 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x7dc8c00
-134> 2013-04-19 20:46:40.226003 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.30 10.1.0.70:6800/17218 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f6c0
-133> 2013-04-19 20:46:40.308412 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879385 ==== paxos(accept lc 349957 fc 0
pn 26300 opn 0) v3 ==== 80+0+0 (4132069560 0 0) 0x3eaa780 con 0x2f67600
-132> 2013-04-19 20:46:40.319736 7fa90880b700 1 -- 10.1.0.3:6789/0
--> mon.2 10.1.0.130:6789/0 -- paxos(commit lc 349958 fc 0 pn 26300 opn
0) v3 -- ?+0 0x3eaa500
-131> 2013-04-19 20:46:40.319772 7fa90880b700 1 -- 10.1.0.3:6789/0
--> mon.2 10.1.0.130:6789/0 -- paxos(lease lc 349958 fc 349939 pn 0 opn
0) v3 -- ?+0 0x6098a00
-130> 2013-04-19 20:46:40.320027 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.3:6789/0 -- log(last 1) v1 -- ?+0 0x2f7fd40 con 0x2f666e0
-129> 2013-04-19 20:46:40.320047 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.3:6789/0 -- log(last 2) v1 -- ?+0 0x2f7f520 con 0x2f666e0
-128> 2013-04-19 20:46:40.320427 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.0 10.1.0.3:6789/0 0 ==== log(last 1) v1 ==== 0+0+0 (0 0 0)
0x2f7fd40 con 0x2f666e0
-127> 2013-04-19 20:46:40.320440 7fa90880b700 10 handle_log_ack
log(last 1) v1
-126> 2013-04-19 20:46:40.320445 7fa90880b700 10 logged 2013-04-19
20:46:27.144414 mon.0 10.1.0.3:6789/0 1 : [INF] mon.a calling new
monitor election
-125> 2013-04-19 20:46:40.320453 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.0 10.1.0.3:6789/0 0 ==== log(last 2) v1 ==== 0+0+0 (0 0 0)
0x2f7f520 con 0x2f666e0
-124> 2013-04-19 20:46:40.320458 7fa90880b700 10 handle_log_ack
log(last 2) v1
-123> 2013-04-19 20:46:40.320460 7fa90880b700 10 logged 2013-04-19
20:46:31.117474 mon.0 10.1.0.3:6789/0 2 : [INF] mon.a calling new
monitor election
-122> 2013-04-19 20:46:40.320465 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.32 10.1.0.70:6806/17398 8 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=867}) v2 ==== 69+0+0
(832282147 0 0) 0x3c78540 con 0x3eda580
-121> 2013-04-19 20:46:40.320554 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- osd_map(867..869 src has 368..869) v3
-- ?+0 0x3eac800
-120> 2013-04-19 20:46:40.320569 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.32 10.1.0.70:6806/17398 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7f520
-119> 2013-04-19 20:46:40.337399 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.42 10.1.0.130:6800/5691 7 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78380 con 0x3edab00
-118> 2013-04-19 20:46:40.337482 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x4535400
-117> 2013-04-19 20:46:40.337498 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.42 10.1.0.130:6800/5691 -- mon_subscribe_ack(300s) v1 -- ?+0
0x2f7fd40
-116> 2013-04-19 20:46:40.422368 7fa9061f3700 1 -- 10.1.0.3:6789/0
>> :/0 pipe(0x6099680 sd=32 :6789 s=0 pgs=0 cs=0 l=0).accept sd=32
10.1.0.130:47463/0
-115> 2013-04-19 20:46:40.423007 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (2563296573 0 0) 0x5d1ae00 con 0x3fa62c0
-114> 2013-04-19 20:46:40.423196 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6803/5771 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x7dc8800 con 0x3fa62c0
-113> 2013-04-19 20:46:40.423981 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (2883588472 0 0) 0x3e28800 con 0x3fa62c0
-112> 2013-04-19 20:46:40.424249 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6803/5771 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x5d1ae00 con 0x3fa62c0
-111> 2013-04-19 20:46:40.424937 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (4292545613 0 0) 0x5d1b600 con 0x3fa62c0
-110> 2013-04-19 20:46:40.425215 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6803/5771 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e28800 con 0x3fa62c0
-109> 2013-04-19 20:46:40.425876 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x5de2c40 con 0x3fa62c0
-108> 2013-04-19 20:46:40.426008 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.43 10.1.0.130:6803/5771 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x5d1b600
-107> 2013-04-19 20:46:40.426030 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.43 10.1.0.130:6803/5771 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3040
-106> 2013-04-19 20:46:40.426049 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 5 ==== pg_stats(15 pgs tid 955 v 869) v1
==== 5975+0+0 (1516527665 0 0) 0x6516900 con 0x3fa62c0
-105> 2013-04-19 20:46:40.426206 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.43 10.1.0.130:6803/5771 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78e00 con 0x3fa62c0
-104> 2013-04-19 20:46:40.426273 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.43 10.1.0.130:6803/5771 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x42a0800
-103> 2013-04-19 20:46:40.426288 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.43 10.1.0.130:6803/5771 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2d00
-102> 2013-04-19 20:46:40.431429 7fa9041d3700 1 -- 10.1.0.3:6789/0
>> :/0 pipe(0x6099180 sd=36 :6789 s=0 pgs=0 cs=0 l=0).accept sd=36
10.1.0.72:49650/0
-101> 2013-04-19 20:46:40.432182 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (114193530 0 0) 0x3e28800 con 0x3fa6c60
-100> 2013-04-19 20:46:40.432330 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6803/5685 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x42a0400 con 0x3fa6c60
-99> 2013-04-19 20:46:40.433186 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (2296283565 0 0) 0x42a0800 con 0x3fa6c60
-98> 2013-04-19 20:46:40.433416 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6803/5685 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x3e28800 con 0x3fa6c60
-97> 2013-04-19 20:46:40.434229 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (491144608 0 0) 0x5d1b600 con 0x3fa6c60
-96> 2013-04-19 20:46:40.434484 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.72:6803/5685 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x42a0800 con 0x3fa6c60
-95> 2013-04-19 20:46:40.435213 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=864}) v2 ==== 69+0+0
(674533898 0 0) 0x3c79880 con 0x3fa6c60
-94> 2013-04-19 20:46:40.435421 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.37 10.1.0.72:6803/5685 -- osd_map(864..869 src has 368..869) v3
-- ?+0 0x5d1b600
-93> 2013-04-19 20:46:40.435439 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.37 10.1.0.72:6803/5685 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2000
-92> 2013-04-19 20:46:40.435694 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 5 ==== pg_stats(29 pgs tid 794 v 869) v1
==== 11325+0+0 (1913177260 0 0) 0x5ec7440 con 0x3fa6c60
-91> 2013-04-19 20:46:40.435887 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.37 10.1.0.72:6803/5685 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=864}) v2 ==== 69+0+0
(674533898 0 0) 0x3c79c00 con 0x3fa6c60
-90> 2013-04-19 20:46:40.436011 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.37 10.1.0.72:6803/5685 -- osd_map(864..869 src has 368..869) v3
-- ?+0 0x5ca8600
-89> 2013-04-19 20:46:40.436027 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.37 10.1.0.72:6803/5685 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2ea0
-88> 2013-04-19 20:46:40.446792 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879386 ==== paxos(lease_ack lc 349958 fc
349939 pn 0 opn 0) v3 ==== 80+0+0 (3515811072 0 0) 0x7d17400 con 0x2f67600
-87> 2013-04-19 20:46:40.452774 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.44 10.1.0.130:6806/5892 7 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x3c78700 con 0x3fa71e0
-86> 2013-04-19 20:46:40.452919 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x5ca8200
-85> 2013-04-19 20:46:40.452936 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.44 10.1.0.130:6806/5892 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac24e0
-84> 2013-04-19 20:46:40.464126 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.61 10.1.0.136:6803/16299 7 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c78c40 con 0x3fa74a0
-83> 2013-04-19 20:46:40.464258 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- osd_map(869..869 src has 368..869)
v3 -- ?+0 0x5ca8000
-82> 2013-04-19 20:46:40.464286 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.61 10.1.0.136:6803/16299 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3ba0
-81> 2013-04-19 20:46:40.470262 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.40 10.1.0.73:6803/20825 8 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c796c0 con 0x3fa7b80
-80> 2013-04-19 20:46:40.470375 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- osd_map(869..869 src has 368..869) v3
-- ?+0 0x5cabe00
-79> 2013-04-19 20:46:40.470396 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.40 10.1.0.73:6803/20825 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2340
-78> 2013-04-19 20:46:40.505969 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879387 ==== forward(pg_stats(19 pgs tid
867 v 869) v1) to leader v1 ==== 7775+0+0 (728647759 0 0) 0x2f79600 con
0x2f67600
-77> 2013-04-19 20:46:40.506191 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879388 ==== forward(pg_stats(14 pgs tid
614 v 869) v1) to leader v1 ==== 5870+0+0 (4091012803 0 0) 0x2f79080 con
0x2f67600
-76> 2013-04-19 20:46:40.506303 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879389 ==== forward(pg_stats(18 pgs tid
890 v 869) v1) to leader v1 ==== 7406+0+0 (3407156785 0 0) 0x3de0dc0 con
0x2f67600
-75> 2013-04-19 20:46:40.506419 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879390 ==== forward(pg_stats(18 pgs tid
676 v 869) v1) to leader v1 ==== 7406+0+0 (3342077306 0 0) 0x3de1080 con
0x2f67600
-74> 2013-04-19 20:46:40.506703 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879391 ==== forward(pg_stats(17 pgs tid
700 v 869) v1) to leader v1 ==== 7025+0+0 (4122683813 0 0) 0x3de1340 con
0x2f67600
-73> 2013-04-19 20:46:40.506897 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879392 ==== forward(pg_stats(14 pgs tid
657 v 869) v1) to leader v1 ==== 5882+0+0 (4273081116 0 0) 0x3de1600 con
0x2f67600
-72> 2013-04-19 20:46:40.507003 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879393 ==== forward(pg_stats(18 pgs tid
861 v 869) v1) to leader v1 ==== 7394+0+0 (1432642447 0 0) 0x3de18c0 con
0x2f67600
-71> 2013-04-19 20:46:40.507117 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879394 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (3114508070 0 0) 0x3cbadc0 con 0x2f67600
-70> 2013-04-19 20:46:40.507136 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-69> 2013-04-19 20:46:40.507194 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1245) v2 --
?+0 0x505b200 con 0x2f67600
-68> 2013-04-19 20:46:40.507212 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879395 ==== forward(pg_stats(23 pgs tid
671 v 869) v1) to leader v1 ==== 9299+0+0 (700762101 0 0) 0x3d64580 con
0x2f67600
-67> 2013-04-19 20:46:40.507337 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879396 ==== forward(pg_stats(21 pgs tid
825 v 869) v1) to leader v1 ==== 8537+0+0 (1915302581 0 0) 0x3d642c0 con
0x2f67600
-66> 2013-04-19 20:46:40.507454 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879397 ==== forward(pg_stats(18 pgs tid
623 v 869) v1) to leader v1 ==== 7394+0+0 (1485925630 0 0) 0x3d64000 con
0x2f67600
-65> 2013-04-19 20:46:40.507567 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879398 ==== forward(pg_stats(25 pgs tid
658 v 869) v1) to leader v1 ==== 10073+0+0 (3890648552 0 0) 0x2f79b80
con 0x2f67600
-64> 2013-04-19 20:46:40.507694 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879399 ==== forward(pg_stats(13 pgs tid
657 v 869) v1) to leader v1 ==== 5501+0+0 (51453908 0 0) 0x3de0b00 con
0x2f67600
-63> 2013-04-19 20:46:40.507798 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879400 ==== forward(pg_stats(18 pgs tid
884 v 869) v1) to leader v1 ==== 7394+0+0 (2104113953 0 0) 0x3cbb8c0 con
0x2f67600
-62> 2013-04-19 20:46:40.508145 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879401 ==== forward(pg_stats(22 pgs tid
878 v 869) v1) to leader v1 ==== 8926+0+0 (2949352605 0 0) 0x3cbbb80 con
0x2f67600
-61> 2013-04-19 20:46:40.508345 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879402 ==== forward(pg_stats(16 pgs tid
855 v 869) v1) to leader v1 ==== 6644+0+0 (1853649727 0 0) 0x3cba000 con
0x2f67600
-60> 2013-04-19 20:46:40.508495 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879403 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (607791097 0 0) 0x3cba2c0 con 0x2f67600
-59> 2013-04-19 20:46:40.508512 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-58> 2013-04-19 20:46:40.508567 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1254) v2 --
?+0 0x6516fc0 con 0x2f67600
-57> 2013-04-19 20:46:40.508584 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879404 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (602189977 0 0) 0x3cba580 con 0x2f67600
-56> 2013-04-19 20:46:40.508647 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-55> 2013-04-19 20:46:40.508726 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1255) v2 --
?+0 0x6517200 con 0x2f67600
-54> 2013-04-19 20:46:40.508798 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879405 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (989012989 0 0) 0x3cba840 con 0x2f67600
-53> 2013-04-19 20:46:40.508813 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-52> 2013-04-19 20:46:40.508873 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1256) v2 --
?+0 0x6517440 con 0x2f67600
-51> 2013-04-19 20:46:40.508889 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879406 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (2254911044 0 0) 0x3cbab00 con 0x2f67600
-50> 2013-04-19 20:46:40.508929 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-49> 2013-04-19 20:46:40.508927 7fa9043d5700 1 -- 10.1.0.3:6789/0
>> :/0 pipe(0x3dab400 sd=29 :6789 s=0 pgs=0 cs=0 l=0).accept sd=29
10.1.0.131:54319/0
-48> 2013-04-19 20:46:40.509013 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1257) v2 --
?+0 0x6517680 con 0x2f67600
-47> 2013-04-19 20:46:40.509195 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879407 ==== forward(pg_stats(20 pgs tid
646 v 869) v1) to leader v1 ==== 8156+0+0 (1238169845 0 0) 0x3de0000 con
0x2f67600
-46> 2013-04-19 20:46:40.509341 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879408 ==== forward(pg_stats(23 pgs tid
866 v 869) v1) to leader v1 ==== 9299+0+0 (3632557150 0 0) 0x3de02c0 con
0x2f67600
-45> 2013-04-19 20:46:40.509508 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879409 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (448352222 0 0) 0x3de0580 con 0x2f67600
-44> 2013-04-19 20:46:40.509534 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-43> 2013-04-19 20:46:40.509593 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1260) v2 --
?+0 0x65178c0 con 0x2f67600
-42> 2013-04-19 20:46:40.509609 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879410 ==== forward(pg_stats(9 pgs tid
643 v 869) v1) to leader v1 ==== 3977+0+0 (2965359266 0 0) 0x3de0840 con
0x2f67600
-41> 2013-04-19 20:46:40.509715 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 1 ==== auth(proto 0 27 bytes epoch 1) v1
==== 57+0+0 (3763761031 0 0) 0x7e82000 con 0x3f2a420
-40> 2013-04-19 20:46:40.509784 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6803/1302 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x545f600 con 0x3f2a420
-39> 2013-04-19 20:46:40.509802 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879411 ==== forward(pg_stats(25 pgs tid
893 v 869) v1) to leader v1 ==== 10057+0+0 (1169603745 0 0) 0x2f79b80
con 0x2f67600
-38> 2013-04-19 20:46:40.509964 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879412 ==== forward(pg_stats(15 pgs tid
609 v 869) v1) to leader v1 ==== 6263+0+0 (624775136 0 0) 0x2f79080 con
0x2f67600
-37> 2013-04-19 20:46:40.510105 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879413 ==== forward(mon_command(auth
get-or-create client.admin mon allow * osd allow * mds allow v 0) v1) to
leader v1 ==== 394+0+0 (1455554774 0 0) 0x2f79600 con 0x2f67600
-36> 2013-04-19 20:46:40.510127 7fa90880b700 0 mon.a@0(leader) e1
handle_command mon_command(auth get-or-create client.admin mon allow *
osd allow * mds allow v 0) v1
-35> 2013-04-19 20:46:40.510193 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.130:6789/0 --
route(mon_command_ack([auth,get-or-create,client.admin,mon,allow
*,osd,allow *,mds,allow]=-13 access denied v34578) v1 tid 1264) v2 --
?+0 0x7e85d40 con 0x2f67600
-34> 2013-04-19 20:46:40.510211 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879414 ==== forward(pg_stats(16 pgs tid
911 v 869) v1) to leader v1 ==== 6644+0+0 (1068991193 0 0) 0x2f79340 con
0x2f67600
-33> 2013-04-19 20:46:40.510389 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879415 ==== forward(pg_stats(28 pgs tid
627 v 869) v1) to leader v1 ==== 11216+0+0 (4031370868 0 0) 0x3cbb340
con 0x2f67600
-32> 2013-04-19 20:46:40.510571 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879416 ==== forward(pg_stats(18 pgs tid
626 v 869) v1) to leader v1 ==== 7406+0+0 (3651646549 0 0) 0x3cbadc0 con
0x2f67600
-31> 2013-04-19 20:46:40.510721 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 2 ==== auth(proto 2 128 bytes epoch 0)
v1 ==== 158+0+0 (3124271367 0 0) 0x7e82400 con 0x3f2a420
-30> 2013-04-19 20:46:40.510892 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6803/1302 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x4fea000 con 0x3f2a420
-29> 2013-04-19 20:46:40.511669 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 3 ==== auth(proto 2 165 bytes epoch 0)
v1 ==== 195+0+0 (3165417802 0 0) 0x7e82200 con 0x3f2a420
-28> 2013-04-19 20:46:40.511944 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.131:6803/1302 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x7e82400 con 0x3f2a420
-27> 2013-04-19 20:46:40.512561 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x3d65500 con 0x3f2a420
-26> 2013-04-19 20:46:40.512674 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.46 10.1.0.131:6803/1302 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x7e82200
-25> 2013-04-19 20:46:40.512692 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.46 10.1.0.131:6803/1302 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac3860
-24> 2013-04-19 20:46:40.512953 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 5 ==== pg_stats(21 pgs tid 708 v 869) v1
==== 8265+0+0 (639420979 0 0) 0x7e86000 con 0x3f2a420
-23> 2013-04-19 20:46:40.513117 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.46 10.1.0.131:6803/1302 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x3d65880 con 0x3f2a420
-22> 2013-04-19 20:46:40.513192 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.46 10.1.0.131:6803/1302 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x4fe7400
-21> 2013-04-19 20:46:40.513206 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.46 10.1.0.131:6803/1302 -- mon_subscribe_ack(300s) v1 -- ?+0
0x7ac2d00
-20> 2013-04-19 20:46:40.515395 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879417 ==== forward(pg_stats(18 pgs tid
862 v 869) v1) to leader v1 ==== 7394+0+0 (766536100 0 0) 0x3cbb600 con
0x2f67600
-19> 2013-04-19 20:46:40.515570 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879418 ==== forward(pg_stats(18 pgs tid
885 v 869) v1) to leader v1 ==== 7394+0+0 (111214024 0 0) 0x3de0b00 con
0x2f67600
-18> 2013-04-19 20:46:40.515690 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879419 ==== forward(pg_stats(22 pgs tid
879 v 869) v1) to leader v1 ==== 8926+0+0 (3804612260 0 0) 0x3de18c0 con
0x2f67600
-17> 2013-04-19 20:46:40.515798 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879420 ==== forward(pg_stats(16 pgs tid
912 v 869) v1) to leader v1 ==== 6644+0+0 (826149121 0 0) 0x3de1600 con
0x2f67600
-16> 2013-04-19 20:46:40.623622 7fa90880b700 1 -- 10.1.0.3:6789/0
<== mon.2 10.1.0.130:6789/0 1293879421 ==== forward(pg_stats(16 pgs tid
856 v 869) v1) to leader v1 ==== 6644+0+0 (2519858941 0 0) 0x3de1340 con
0x2f67600
-15> 2013-04-19 20:46:40.680702 7fa90c122700 1 -- 10.1.0.3:6789/0
>> :/0 pipe(0x3dabb80 sd=37 :6789 s=0 pgs=0 cs=0 l=0).accept sd=37
10.1.0.4:37961/0
-14> 2013-04-19 20:46:40.681369 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 1 ==== auth(proto 0 26 bytes epoch 1) v1
==== 56+0+0 (2899718969 0 0) 0x7e82400 con 0x3f2b1e0
-13> 2013-04-19 20:46:40.681514 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.4:6806/29925 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x4fe6600 con 0x3f2b1e0
-12> 2013-04-19 20:46:40.682332 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 2 ==== auth(proto 2 128 bytes epoch 0) v1
==== 158+0+0 (1540308490 0 0) 0x4fe7400 con 0x3f2b1e0
-11> 2013-04-19 20:46:40.682549 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.4:6806/29925 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x7e82400 con 0x3f2b1e0
-10> 2013-04-19 20:46:40.683231 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 3 ==== auth(proto 2 165 bytes epoch 0) v1
==== 195+0+0 (940588970 0 0) 0x7e82200 con 0x3f2b1e0
-9> 2013-04-19 20:46:40.683422 7fa90880b700 1 -- 10.1.0.3:6789/0
--> 10.1.0.4:6806/29925 -- auth_reply(proto 2 0 Success) v1 -- ?+0
0x4fe7400 con 0x3f2b1e0
-8> 2013-04-19 20:46:40.684091 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 4 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x4214000 con 0x3f2b1e0
-7> 2013-04-19 20:46:40.684194 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.5 10.1.0.4:6806/29925 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x7e82200
-6> 2013-04-19 20:46:40.684211 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.5 10.1.0.4:6806/29925 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac2b60
-5> 2013-04-19 20:46:40.685780 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 5 ==== pg_stats(119 pgs tid 910 v 869) v1
==== 45615+0+0 (1758882131 0 0) 0x7e87680 con 0x3f2b1e0
-4> 2013-04-19 20:46:40.686189 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 6 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=868}) v2 ==== 69+0+0
(4143665209 0 0) 0x4214380 con 0x3f2b1e0
-3> 2013-04-19 20:46:40.686262 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.5 10.1.0.4:6806/29925 -- osd_map(868..869 src has 368..869) v3
-- ?+0 0x5028200
-2> 2013-04-19 20:46:40.686275 7fa90880b700 1 -- 10.1.0.3:6789/0
--> osd.5 10.1.0.4:6806/29925 -- mon_subscribe_ack(300s) v1 -- ?+0 0x7ac21a0
-1> 2013-04-19 20:46:40.686287 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.5 10.1.0.4:6806/29925 7 ==== log(107 entries) v1 ==== 20097+0+0
(1892829298 0 0) 0x6060600 con 0x3f2b1e0
0> 2013-04-19 20:46:40.703848 7fa90900c700 -1 mon/Monitor.cc: In
function 'void Monitor::sync_timeout(entity_inst_t&)' thread
7fa90900c700 time 2013-04-19 20:46:40.703222
mon/Monitor.cc: 1105: FAILED assert(0 == "We should never reach this")
ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
1: (Monitor::sync_timeout(entity_inst_t&)+0x535) [0x4b5ed5]
2: (Context::complete(int)+0xa) [0x4c297a]
3: (SafeTimer::timer_thread()+0x1b2) [0x62e3c2]
4: (SafeTimerThread::entry()+0xd) [0x62fa4d]
5: (()+0x7f8e) [0x7fa90f63af8e]
6: (clone()+0x6d) [0x7fa90e280e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 1 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.a.log
--- end dump of recent events ---
2013-04-19 20:46:40.733976 7fa90900c700 -1 *** Caught signal (Aborted) **
in thread 7fa90900c700
ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
1: /usr/bin/ceph-mon() [0x5869b0]
2: (()+0xfbd0) [0x7fa90f642bd0]
3: (gsignal()+0x37) [0x7fa90e1be037]
4: (abort()+0x148) [0x7fa90e1c1698]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa90eacae8d]
6: (()+0x5ef76) [0x7fa90eac8f76]
7: (()+0x5efa3) [0x7fa90eac8fa3]
8: (()+0x5f1de) [0x7fa90eac91de]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x43d) [0x6378cd]
10: (Monitor::sync_timeout(entity_inst_t&)+0x535) [0x4b5ed5]
11: (Context::complete(int)+0xa) [0x4c297a]
12: (SafeTimer::timer_thread()+0x1b2) [0x62e3c2]
13: (SafeTimerThread::entry()+0xd) [0x62fa4d]
14: (()+0x7f8e) [0x7fa90f63af8e]
15: (clone()+0x6d) [0x7fa90e280e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- begin dump of recent events ---
-2> 2013-04-19 20:46:40.721635 7fa90880b700 1 -- 10.1.0.3:6789/0
<== osd.64 10.1.0.137:6803/5123 8 ====
mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=869}) v2 ==== 69+0+0
(46499441 0 0) 0x3c79180 con 0x3fa7ce0
-1> 2013-04-19 20:46:40.730843 7fa9033c5700 1 -- 10.1.0.3:6789/0
>> :/0 pipe(0x3daac80 sd=38 :6789 s=0 pgs=0 cs=0 l=0).accept sd=38
10.1.0.135:47591/0
0> 2013-04-19 20:46:40.733976 7fa90900c700 -1 *** Caught signal
(Aborted) **
in thread 7fa90900c700
ceph version 0.60 (f26f7a39021dbf440c28d6375222e21c94fe8e5c)
1: /usr/bin/ceph-mon() [0x5869b0]
2: (()+0xfbd0) [0x7fa90f642bd0]
3: (gsignal()+0x37) [0x7fa90e1be037]
4: (abort()+0x148) [0x7fa90e1c1698]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa90eacae8d]
6: (()+0x5ef76) [0x7fa90eac8f76]
7: (()+0x5efa3) [0x7fa90eac8fa3]
8: (()+0x5f1de) [0x7fa90eac91de]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x43d) [0x6378cd]
10: (Monitor::sync_timeout(entity_inst_t&)+0x535) [0x4b5ed5]
11: (Context::complete(int)+0xa) [0x4c297a]
12: (SafeTimer::timer_thread()+0x1b2) [0x62e3c2]
13: (SafeTimerThread::entry()+0xd) [0x62fa4d]
14: (()+0x7f8e) [0x7fa90f63af8e]
15: (clone()+0x6d) [0x7fa90e280e1d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 1 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mon.a.log
--- end dump of recent events ---
root@xxxx:~#
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
