[Re-adding users list.] I notice that each of your ceph -s runs has output a different result, and in your latest one you've got an OSD that is down (osdmap e523639: 3 osds: 2 up, 3 in). Given that (and what appears to be constant change) it's not surprising your cluster won't go all the way healthy. Is your cluster just too overloaded and thrashing for some reason? If you want to reset you can shut down the cluster and go through ceph-deploy (with the cluster zapping stuff), or just delete all your existing pools and re-create them — you will maintain your existing cluster history but it will get rid of all the data and recreate all the PGs. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Wed, Sep 25, 2013 at 4:50 AM, Markus Goldberg <goldberg@xxxxxxxxxxxxxxxxx> wrote: > Hi Gregory, > sorry for the delay, i was ill and out of office a few days. > I think i will give up. > I followed the trobleshooting steps but nothing helped. > Because i only had very little data on this cluster, i will reset it to > zero. > How should i do this ? purgedata ? reinstall ? > Of course there will stay a little bit fear, that this will happen again. I > don't know exactly what happened. Perhaps during kernel-upgrade and reboot > the cluster was still mounted. Is there a roadmap, how to handle > kernel-upgrades ? It is possible, to stop the cluster, because it is (will > be) a backup-system. > > Attached is some output. Perhaps you can give me a last tip what to do ? > > Thank you, > Markus > > root@bd-a:~# > root@bd-a:~# ceph -s > cluster e0dbf70d-af59-42a5-b834-7ad739a7f89b > health HEALTH_ERR 2 pgs inconsistent; 265 pgs peering; 272 pgs stale; 2 > pgs stuck inactive; 265 pgs stuck unclean; 1 requests are blocked > 32 sec; > 2 scrub errors; mds cluster is degraded; 1/3 in osds are down > monmap e1: 3 mons at > {bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}, > election epoch 796, quorum 0,1,2 bd-0,bd-1,bd-2 > mdsmap e451479: 1/1/1 up {0=bd-2=up:replay}, 2 up:standby > osdmap e523639: 3 osds: 2 up, 3 in > pgmap v1553072: 792 pgs, 9 pools, 15145 MB data, 4986 objects > 32759 MB used, 61363 GB / 61408 GB avail > 387 active+clean > 41 stale+active > 99 stale+active+clean > 132 peering > 131 stale+peering > 1 inconsistent+peering > 1 stale+inconsistent+peering > > root@bd-a:~# ceph pg dump_stuck stale > ok > pg_stat objects mip degr unf bytes log disklog state > state_stamp v reported up acting last_scrub > scrub_stamp last_deep_scrub deep_scrub_stamp > > root@bd-a:~# ceph pg dump_stuck inactive > ok > pg_stat objects mip degr unf bytes log disklog state > state_stamp v reported up acting last_scrub > scrub_stamp last_deep_scrub deep_scrub_stamp > 1.3f 3 0 0 0 4196848 3002 3002 peering > 2013-09-25 13:38:25.686898 521351'7629 523649:526332 [0,1] > [0,1] 82674'7628 2013-09-23 11:02:00.822456 82674'7628 > 2013-09-23 11:02:00.822456 > 0.20 92 0 0 0 281018368 118 118 > remapped+peering 2013-09-25 13:38:21.466380 20704'118 > 523649:28344 [0,1] [1,2] 20704'118 2013-09-23 11:05:28.448391 > 20704'1182013-09-23 11:05:28.448391 > 4.24 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.467020 0'0 > 523649:493589 [0,1] [1,2] 0'0 2013-09-23 11:04:12.800652 0'0 > 2013-09-23 11:04:12.800652 > 6.19 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.437294 0'0 > 523649:493352 [0,1] [1,2] 0'0 2013-09-23 11:02:36.778942 0'0 > 2013-09-23 11:02:36.778942 > 5.1a 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.135371 0'0 > 523649:493493 [0,1] [1,2] 0'0 2013-09-23 11:05:11.826873 0'0 > 2013-09-23 11:05:11.826873 > 8.16 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.438289 0'0 > 523649:493362 [0,1] [1,2] 0'0 2013-09-23 11:06:38.270626 0'0 > 2013-09-23 11:06:38.270626 > 2.1c 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.440052 0'0 > 523649:510483 [0,1] [1,2] 0'0 2013-09-23 11:02:20.774699 0'0 > 2013-09-23 11:02:20.774699 > 1.1f 0 0 0 0 0 3002 3002 > remapped+peering 2013-09-25 13:38:21.440943 20696'15250 > 523649:54608 [0,1] [1,2] 20696'15250 2013-09-23 11:01:46.767961 > 20696'15250 2013-09-23 11:01:46.767961 > 0.1e 82 0 0 0 293601280 90 90 > remapped+peering 2013-09-25 13:38:21.441844 20704'90 > 523649:30479 [0,1] [1,2] 20704'90 2013-09-23 11:01:10.965007 > 20704'90 2013-09-23 11:01:10.965007 > 3.1d 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.439173 0'0 > 523649:493338 [0,1] [1,2] 0'0 2013-09-23 11:03:20.789125 0'0 > 2013-09-23 11:03:20.789125 > 9.14 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.450200 0'0 > 523649:493189 [0,1] [1,2] 0'0 2013-09-23 11:05:52.260195 0'0 > 2013-09-23 11:05:52.260195 > 5.18 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.462825 0'0 > 523649:493199 [0,1] [1,2] 0'0 2013-09-23 11:04:58.811264 0'0 > 2013-09-23 11:04:58.811264 > 4.19 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.462143 0'0 > 523649:493194 [0,1] [1,2] 0'0 2013-09-23 11:04:11.800180 0'0 > 2013-09-23 11:04:11.800180 > 6.1a 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.478589 0'0 > 523649:493131 [0,1] [1,2] 0'0 2013-09-23 11:02:37.779451 0'0 > 2013-09-23 11:02:37.779451 > 2.1e 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.466891 0'0 > 523649:510334 [0,1] [1,2] 0'0 2013-09-23 11:02:22.775290 0'0 > 2013-09-23 11:02:22.775290 > 9.12 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.479132 0'0 > 523649:493169 [0,1] [1,2] 0'0 2013-09-23 11:05:50.259627 0'0 > 2013-09-23 11:05:50.259627 > 8.10 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.500467 0'0 > 523649:492861 [0,1] [1,2] 0'0 2013-09-23 11:06:36.270107 0'0 > 2013-09-23 11:06:36.270107 > 6.1e 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.514829 0'0 > 523649:492722 [0,1] [1,2] 0'0 2013-09-23 11:02:38.779745 0'0 > 2013-09-23 11:02:38.779745 > 2.15 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.515533 0'0 > 523649:510099 [0,1] [1,2] 0'0 2013-09-23 11:02:19.774466 0'0 > 2013-09-23 11:02:19.774466 > 1.16 1 0 0 0 4194304 3001 3001 > remapped+peering 2013-09-25 13:38:21.851844 20704'12853 > 523649:50380 [0,1] [1,2] 20704'12853 2013-09-23 11:01:42.819355 > 20704'12853 2013-09-23 11:01:42.819355 > 0.17 92 0 0 0 285212672 112 112 > remapped+peering 2013-09-25 13:38:21.853177 20704'112 > 523649:30556 [0,1] [1,2] 20704'112 2013-09-23 11:01:06.331793 > 20704'1122013-09-23 11:01:06.331793 > 5.13 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.491272 0'0 > 523649:492706 [0,1] [1,2] 0'0 2013-09-23 11:04:47.808565 0'0 > 2013-09-23 11:04:47.808565 > 1.14 2 0 0 0 516 3005 3005 > remapped+peering 2013-09-25 13:38:21.501703 521354'157507 > 523649:743202 [0,1] [1,2] 82678'157503 2013-09-23 11:01:40.771856 > 82678'157503 2013-09-23 11:01:40.771856 > 0.15 65 0 0 0 193210615 74 74 > remapped+peering 2013-09-25 13:38:21.502315 20696'74 > 523649:28339 [0,1] [1,2] 20696'74 2013-09-23 11:01:03.458628 > 20696'74 2013-09-23 11:01:03.458628 > 3.16 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.492626 0'0 > 523649:492675 [0,1] [1,2] 0'0 2013-09-23 11:03:18.789056 0'0 > 2013-09-23 11:03:18.789056 > 6.12 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.515489 0'0 > 523649:492434 [0,1] [1,2] 0'0 2013-09-23 11:02:33.778079 0'0 > 2013-09-23 11:02:33.778079 > 3.10 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:38:21.851142 0'0 > 523649:492399 [0,1] [1,2] 0'0 2013-09-23 11:03:14.788151 0'0 > 2013-09-23 11:03:14.788151 > > root@bd-a:~# > root@bd-a:~# > root@bd-a:~# ceph pg dump_stuck unclean > ok > pg_stat objects mip degr unf bytes log disklog state > state_stamp v reported up acting last_scrub > scrub_stamp last_deep_scrub deep_scrub_stamp > 6.a 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.977757 0'0 523661:414525 [1,0] [1,0] 0'0 > 2013-09-23 11:02:02.072190 0'0 2013-09-23 11:02:02.072190 > 9.2 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.858140 0'0 523657:498099 [0,1] [0,1] 0'0 > 2013-09-23 11:05:42.257861 0'0 2013-09-23 11:05:42.257861 > 3.9 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.858611 0'0 523657:498186 [0,1] [0,1] 0'0 > 2013-09-23 11:03:09.787056 0'0 2013-09-23 11:03:09.787056 > 9.0 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.977840 0'0 523661:414925 [1,0] [1,0] 0'0 > 2013-09-23 11:06:19.052949 0'0 2013-09-23 11:06:19.052949 > 6.f 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.938707 0'0 523661:414959 [1,0] [1,0] 0'0 > 2013-09-23 11:02:03.072590 0'0 2013-09-23 11:02:03.072590 > 5.c 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.977973 0'0 523661:414901 [1,0] [1,0] 0'0 > 2013-09-23 11:05:07.103854 0'0 2013-09-23 11:05:07.103854 > 2.b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.899553 0'0 523657:515318 [0,1] [0,1] 0'0 > 2013-09-23 11:02:07.771996 0'0 2013-09-23 11:02:07.771996 > 8.0 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.857475 0'0 523657:497805 [0,1] [0,1] 0'0 > 2013-09-23 11:06:35.269705 0'0 2013-09-23 11:06:35.269705 > 6.e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.884464 0'0 523657:497598 [0,1] [0,1] 0'0 > 2013-09-23 11:02:28.777026 0'0 2013-09-23 11:02:28.777026 > 0.8 74 0 0 0 239075328 93 93 > peering 2013-09-25 13:39:22.883027 20696'93 523657:515718 > [0,1] [0,1] 20696'93 2013-09-23 11:00:58.804215 20696'93 > 2013-09-23 11:00:58.804215 > 4.c 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.938896 0'0 523661:415030 [1,0] [1,0] 0'0 > 2013-09-23 11:04:22.095585 0'0 2013-09-23 11:04:22.095585 > 5.2 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.886274 0'0 523657:497414 [0,1] [0,1] 0'0 > 2013-09-23 11:04:45.808294 0'0 2013-09-23 11:04:45.808294 > 9.e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.884920 0'0 523657:497629 [0,1] [0,1] 0'0 > 2013-09-23 11:05:49.259338 0'0 2013-09-23 11:05:49.259338 > 1.7 3 0 0 0 1909 3002 3002 peering > 2013-09-25 13:39:22.903286 20696'6617 523657:526565 [0,1] > [0,1] 20696'6617 2013-09-23 11:01:37.773530 20696'6617 > 2013-09-23 11:01:37.773530 > 5.0 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.935919 0'0 523657:497620 [0,1] [0,1] 0'0 > 2013-09-23 11:04:37.806500 0'0 2013-09-23 11:04:37.806500 > 9.c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.903991 0'0 523657:497446 [0,1] [0,1] 0'0 > 2013-09-23 11:05:48.259264 0'0 2013-09-23 11:05:48.259264 > 2.6 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.983182 0'0 523657:514702 [0,1] [0,1] 0'0 > 2013-09-23 11:02:04.771383 0'0 2013-09-23 11:02:04.771383 > 4.7 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.036854 0'0 523661:415288 [1,0] [1,0] 0'0 > 2013-09-23 11:04:16.094538 0'0 2013-09-23 11:04:16.094538 > 6.4 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.036121 0'0 523661:415248 [1,0] [1,0] 0'0 > 2013-09-23 11:02:01.072246 0'0 2013-09-23 11:02:01.072246 > 3.1 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.037042 0'0 523661:415301 [1,0] [1,0] 0'0 > 2013-09-23 11:03:16.084725 0'0 2013-09-23 11:03:16.084725 > 4.6 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.986952 0'0 523657:497295 [0,1] [0,1] 0'0 > 2013-09-23 11:04:03.798751 0'0 2013-09-23 11:04:03.798751 > 9.b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.985129 0'0 523657:497231 [0,1] [0,1] 0'0 > 2013-09-23 11:05:45.258465 0'0 2013-09-23 11:05:45.258465 > 6.7 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.983168 0'0 523657:497042 [0,1] [0,1] 0'0 > 2013-09-23 11:02:27.776764 0'0 2013-09-23 11:02:27.776764 > 1.0 1 0 0 0 4194304 3001 3001 peering > 2013-09-25 13:39:22.989352 20704'10568 523657:533044 [0,1] > [0,1] 20704'10568 2013-09-23 11:01:36.829418 20704'10568 > 2013-09-23 11:01:36.829418 > 0.1 61 0 0 0 209715200 67 67 > peering 2013-09-25 13:39:22.989906 20675'67 523657:515008 > [0,1] [0,1] 20675'67 2013-09-23 11:00:55.671116 20675'67 > 2013-09-23 11:00:55.671116 > 9.9 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.102512 0'0 523661:415299 [1,0] [1,0] 0'0 > 2013-09-25 11:00:49.850789 0'0 2013-09-22 17:02:07.603374 > 9.61 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.052423 0'0 523657:497759 [0,1] [0,1] 0'0 > 2013-09-23 11:06:34.269771 0'0 2013-09-23 11:06:34.269771 > 6.61 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.052946 0'0 523657:497930 [0,1] [0,1] 0'0 > 2013-09-23 11:03:08.786404 0'0 2013-09-23 11:03:08.786404 > 4.62 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.102563 0'0 523661:416948 [1,0] [1,0] 0'0 > 2013-09-23 11:05:05.103451 0'0 2013-09-23 11:05:05.103451 > 5.60 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.078051 0'0 523657:497731 [0,1] [0,1] 0'0 > 2013-09-23 11:05:39.257883 0'0 2013-09-23 11:05:39.257883 > 4.60 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.078613 0'0 523657:497627 [0,1] [0,1] 0'0 > 2013-09-23 11:04:33.805173 0'0 2013-09-23 11:04:33.805173 > 5.61 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.027326 0'0 523657:497686 [0,1] [0,1] 0'0 > 2013-09-23 11:05:41.258064 0'0 2013-09-23 11:05:41.258064 > 3.62 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.094456 0'0 523661:416785 [1,0] [1,0] 0'0 > 2013-09-23 11:04:12.093747 0'0 2013-09-23 11:04:12.093747 > 3.63 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.094552 0'0 523661:416930 [1,0] [1,0] 0'0 > 2013-09-23 11:04:13.093818 0'0 2013-09-23 11:04:13.093818 > 3.5c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.030271 0'0 523657:497455 [0,1] [0,1] 0'0 > 2013-09-23 11:04:01.797997 0'0 2013-09-23 11:04:01.797997 > 6.59 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.094680 0'0 523661:416703 [1,0] [1,0] 0'0 > 2013-09-23 11:03:05.082666 0'0 2013-09-23 11:03:05.082666 > 5.5b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.046735 0'0 523657:497421 [0,1] [0,1] 0'0 > 2013-09-23 11:05:38.257607 0'0 2013-09-23 11:05:38.257607 > 8.56 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.094798 0'0 523661:416789 [1,0] [1,0] 0'0 > 2013-09-23 11:00:12.922722 0'0 2013-09-23 11:00:12.922722 > 4.59 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.094900 0'0 523661:416920 [1,0] [1,0] 0'0 > 2013-09-23 11:05:03.102909 0'0 2013-09-23 11:05:03.102909 > 4.58 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.095112 0'0 523661:416784 [1,0] [1,0] 0'0 > 2013-09-23 11:05:01.102483 0'0 2013-09-23 11:05:01.102483 > 3.5f 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.095012 0'0 523661:416824 [1,0] [1,0] 0'0 > 2013-09-23 11:04:10.093132 0'0 2013-09-23 11:04:10.093132 > 5.59 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.095212 0'0 523661:416786 [1,0] [1,0] 0'0 > 2013-09-23 11:06:17.052573 0'0 2013-09-23 11:06:17.052573 > 6.5a 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.095311 0'0 523661:416742 [1,0] [1,0] 0'0 > 2013-09-23 11:03:06.083140 0'0 2013-09-23 11:03:06.083140 > 6.5d 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.188423 0'0 523661:416792 [1,0] [1,0] 0'0 > 2013-09-23 11:03:09.083291 0'0 2013-09-23 11:03:09.083291 > 9.52 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.076579 0'0 523657:497246 [0,1] [0,1] 0'0 > 2013-09-23 11:06:26.267889 0'0 2013-09-23 11:06:26.267889 > 3.59 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.191222 0'0 523661:416862 [1,0] [1,0] 0'0 > 2013-09-23 11:04:07.093133 0'0 2013-09-23 11:04:07.093133 > 6.5c 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.189913 0'0 523661:416773 [1,0] [1,0] 0'0 > 2013-09-23 11:03:08.083038 0'0 2013-09-23 11:03:08.083038 > 8.52 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.188552 0'0 523661:416763 [1,0] [1,0] 0'0 > 2013-09-23 11:00:06.921972 0'0 2013-09-23 11:00:06.921972 > 3.5a 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.191481 0'0 523661:416856 [1,0] [1,0] 0'0 > 2013-09-23 11:04:08.093445 0'0 2013-09-23 11:04:08.093445 > 6.5f 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.191348 0'0 523661:416708 [1,0] [1,0] 0'0 > 2013-09-23 11:03:12.084129 0'0 2013-09-23 11:03:12.084129 > 9.51 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.126771 0'0 523657:497124 [0,1] [0,1] 0'0 > 2013-09-23 11:06:25.267753 0'0 2013-09-23 11:06:25.267753 > 9.5e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.159678 0'0 523657:497023 [0,1] [0,1] 0'0 > 2013-09-23 11:06:27.268212 0'0 2013-09-23 11:06:27.268212 > 5.52 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.189774 0'0 523661:416654 [1,0] [1,0] 0'0 > 2013-09-23 11:06:09.051161 0'0 2013-09-23 11:06:09.051161 > 9.5f 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.214791 0'0 523661:416649 [1,0] [1,0] 0'0 > 2013-09-23 11:06:51.060212 0'0 2013-09-23 11:06:51.060212 > 4.52 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.215029 0'0 523661:416599 [1,0] [1,0] 0'0 > 2013-09-23 11:05:00.102397 0'0 2013-09-23 11:05:00.102397 > 8.5e 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.214873 0'0 523661:416766 [1,0] [1,0] 0'0 > 2013-09-23 11:00:19.923544 0'0 2013-09-23 11:00:19.923544 > 9.5c 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.215156 0'0 523661:416722 [1,0] [1,0] 0'0 > 2013-09-23 11:06:49.059465 0'0 2013-09-23 11:06:49.059465 > 6.53 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.173924 0'0 523657:496987 [0,1] [0,1] 0'0 > 2013-09-23 11:03:07.786443 0'0 2013-09-23 11:03:07.786443 > 3.57 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.215325 0'0 523661:416738 [1,0] [1,0] 0'0 > 2013-09-23 11:04:05.092345 0'0 2013-09-23 11:04:05.092345 > 6.52 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.177026 0'0 523657:496908 [0,1] [0,1] 0'0 > 2013-09-23 11:03:04.786194 0'0 2013-09-23 11:03:04.786194 > 8.5b 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.215449 0'0 523661:416968 [1,0] [1,0] 0'0 > 2013-09-23 11:00:17.923487 0'0 2013-09-23 11:00:17.923487 > 8.5a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.187872 0'0 523657:496631 [0,1] [0,1] 0'0 > 2013-09-23 11:00:51.017728 0'0 2013-09-23 11:00:51.017728 > 4.55 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.212327 0'0 523657:496866 [0,1] [0,1] 0'0 > 2013-09-23 11:04:32.804942 0'0 2013-09-23 11:04:32.804942 > 5.54 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.308359 0'0 523661:416866 [1,0] [1,0] 0'0 > 2013-09-23 11:06:14.051869 0'0 2013-09-23 11:06:14.051869 > 6.57 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.308462 0'0 523661:416880 [1,0] [1,0] 0'0 > 2013-09-23 11:02:58.082078 0'0 2013-09-23 11:02:58.082078 > 8.59 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.211898 0'0 523657:496609 [0,1] [0,1] 0'0 > 2013-09-23 11:00:43.367142 0'0 2013-09-23 11:00:43.367142 > 4.54 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.230544 0'0 523657:496780 [0,1] [0,1] 0'0 > 2013-09-23 11:04:29.804240 0'0 2013-09-23 11:04:29.804240 > 6.49 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.308597 0'0 523661:417071 [1,0] [1,0] 0'0 > 2013-09-23 11:02:51.080646 0'0 2013-09-23 11:02:51.080646 > 4.4a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.281997 0'0 523657:496776 [0,1] [0,1] 0'0 > 2013-09-23 11:04:26.803618 0'0 2013-09-23 11:04:26.803618 > 3.4e 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.292526 0'0 523661:417020 [1,0] [1,0] 0'0 > 2013-09-23 11:04:04.092295 0'0 2013-09-23 11:04:04.092295 > 9.44 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.308723 0'0 523661:417080 [1,0] [1,0] 0'0 > 2013-09-22 17:02:09.603462 0'0 2013-09-22 17:02:09.603462 > 3.4f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.226677 0'0 523657:496456 [0,1] [0,1] 0'0 > 2013-09-23 11:04:00.797771 0'0 2013-09-23 11:04:00.797771 > 5.49 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.227325 0'0 523657:496557 [0,1] [0,1] 0'0 > 2013-09-23 11:05:31.255579 0'0 2013-09-23 11:05:31.255579 > 8.44 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.226083 0'0 523657:496537 [0,1] [0,1] 0'0 > 2013-09-23 11:00:19.010664 0'0 2013-09-23 11:00:19.010664 > 3.48 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.244738 0'0 523657:496376 [0,1] [0,1] 0'0 > 2013-09-23 11:03:59.797547 0'0 2013-09-23 11:03:59.797547 > 5.4e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.245338 0'0 523657:496488 [0,1] [0,1] 0'0 > 2013-09-23 11:05:33.256040 0'0 2013-09-23 11:05:33.256040 > 6.4d 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.292649 0'0 523661:417001 [1,0] [1,0] 0'0 > 2013-09-23 11:02:56.081633 0'0 2013-09-23 11:02:56.081633 > 8.43 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.244128 0'0 523657:496435 [0,1] [0,1] 0'0 > 2013-09-23 11:00:12.009515 0'0 2013-09-23 11:00:12.009515 > 3.49 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.292788 0'0 523661:417150 [1,0] [1,0] 0'0 > 2013-09-23 11:04:02.091954 0'0 2013-09-23 11:04:02.091954 > 6.4c 0 0 0 0 0 0 0 active > 2013-09-25 13:39:23.292907 0'0 523661:417124 [1,0] [1,0] 0'0 > 2013-09-23 11:02:52.080872 0'0 2013-09-23 11:02:52.080872 > 8.41 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.268081 0'0 523657:496264 [0,1] [0,1] 0'0 > 2013-09-23 11:00:10.009162 0'0 2013-09-23 11:00:10.009162 > 9.40 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.267698 0'0 523657:496216 [0,1] [0,1] 0'0 > 2013-09-23 11:06:21.266673 0'0 2013-09-23 11:06:21.266673 > 8.40 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.305287 0'0 523657:496267 [0,1] [0,1] 0'0 > 2013-09-23 11:00:08.008441 0'0 2013-09-23 11:00:08.008441 > 4.43 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.500772 0'0 523661:417239 [1,0] [1,0] 0'0 > 2013-09-22 17:02:02.602658 0'0 2013-09-22 17:02:02.602658 > 5.42 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.500929 0'0 523661:417128 [1,0] [1,0] 0'0 > 2013-09-23 11:05:51.048826 0'0 2013-09-23 11:05:51.048826 > 6.41 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.322520 0'0 523657:496270 [0,1] [0,1] 0'0 > 2013-09-23 11:02:52.782890 0'0 2013-09-23 11:02:52.782890 > 9.4e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.305750 0'0 523657:496203 [0,1] [0,1] 0'0 > 2013-09-23 11:06:22.266860 0'0 2013-09-23 11:06:22.266860 > 5.43 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.344469 0'0 523657:496266 [0,1] [0,1] 0'0 > 2013-09-23 11:05:29.255168 0'0 2013-09-23 11:05:29.255168 > 9.4f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.501262 0'0 523661:417056 [1,0] [1,0] 0'0 > 2013-09-23 11:06:48.059410 0'0 2013-09-23 11:06:48.059410 > 8.4d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.501664 0'0 523661:417042 [1,0] [1,0] 0'0 > 2013-09-23 11:00:03.921642 0'0 2013-09-23 11:00:03.921642 > 3.47 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502090 0'0 523661:417092 [1,0] [1,0] 0'0 > 2013-09-23 11:04:00.091926 0'0 2013-09-23 11:04:00.091926 > 5.41 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502254 0'0 523661:417104 [1,0] [1,0] 0'0 > 2013-09-23 11:05:50.048781 0'0 2013-09-23 11:05:50.048781 > 6.42 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:26.864026 0'0 523657:496187 [0,1] [0,1] 0'0 > 2013-09-23 11:02:58.784472 0'0 2013-09-23 11:02:58.784472 > 4.47 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502261 0'0 523661:417099 [1,0] [1,0] 0'0 > 2013-09-23 11:04:49.234571 0'0 2013-09-23 11:04:49.234571 > 6.45 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.306548 0'0 523657:496110 [0,1] [0,1] 0'0 > 2013-09-23 11:02:59.784474 0'0 2013-09-23 11:02:59.784474 > 8.4b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:23.305305 0'0 523657:496161 [0,1] [0,1] 0'0 > 2013-09-23 11:00:36.366386 0'0 2013-09-23 11:00:36.366386 > 5.47 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502762 0'0 523661:417144 [1,0] [1,0] 0'0 > 2013-09-23 11:06:05.050907 0'0 2013-09-23 11:06:05.050907 > 4.45 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502933 0'0 523661:416959 [1,0] [1,0] 0'0 > 2013-09-23 11:04:48.158121 0'0 2013-09-23 11:04:48.158121 > 3.43 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:26.849619 0'0 523657:496010 [0,1] [0,1] 0'0 > 2013-09-23 11:03:53.796260 0'0 2013-09-23 11:03:53.796260 > 5.3a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502797 0'0 523661:417004 [1,0] [1,0] 0'0 > 2013-09-23 11:05:44.047866 0'0 2013-09-23 11:05:44.047866 > 6.39 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.502963 0'0 523661:417043 [1,0] [1,0] 0'0 > 2013-09-23 11:02:27.076629 0'0 2013-09-23 11:02:27.076629 > 8.37 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.503282 0'0 523661:416970 [1,0] [1,0] 0'0 > 2013-09-23 10:59:59.921038 0'0 2013-09-23 10:59:59.921038 > 1.3f 3 0 0 0 4196848 3002 3002 > remapped+peering 2013-09-25 13:39:19.000068 521351'7629 > 523661:525501 [0,1] [1,2] 82674'7628 2013-09-23 11:02:00.822456 > 82674'7628 2013-09-23 11:02:00.822456 > 0.3d 77 0 0 0 226492416 98 98 > remapped+peering 2013-09-25 13:39:19.000828 20680'98 > 523661:28846 [0,1] [1,2] 20680'98 2013-09-23 11:01:34.861473 > 20680'98 2013-09-23 11:01:34.861473 > 1.3c 1 0 0 0 230 3001 3001 peering > 2013-09-25 13:39:27.214978 20696'9700 523657:530140 [0,1] > [0,1] 20696'9700 2013-09-23 11:01:54.772265 20696'9700 > 2013-09-23 11:01:54.772265 > 2.3f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.504441 0'0 523661:434372 [1,0] [1,0] 0'0 > 2013-09-23 11:01:54.071111 0'0 2013-09-23 11:01:54.071111 > 5.38 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.233806 0'0 523657:495864 [0,1] [0,1] 0'0 > 2013-09-23 11:05:25.254162 0'0 2013-09-23 11:05:25.254162 > 9.34 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.200318 0'0 523657:495733 [0,1] [0,1] 0'0 > 2013-09-23 11:06:09.264408 0'0 2013-09-23 11:06:09.264408 > 3.3f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.252698 0'0 523657:495741 [0,1] [0,1] 0'0 > 2013-09-23 11:03:52.796105 0'0 2013-09-23 11:03:52.796105 > 2.3e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.262559 0'0 523657:513057 [0,1] [0,1] 0'0 > 2013-09-23 11:02:26.776579 0'0 2013-09-23 11:02:26.776579 > 6.3a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.506730 0'0 523661:417269 [1,0] [1,0] 0'0 > 2013-09-23 11:02:36.078019 0'0 2013-09-23 11:02:36.078019 > 8.34 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.506186 0'0 523661:417233 [1,0] [1,0] 0'0 > 2013-09-23 10:59:57.920292 0'0 2013-09-23 10:59:57.920292 > 3.38 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.507099 0'0 523661:417242 [1,0] [1,0] 0'0 > 2013-09-23 11:03:49.090028 0'0 2013-09-23 11:03:49.090028 > 9.32 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.191455 0'0 523657:495576 [0,1] [0,1] 0'0 > 2013-09-23 11:06:05.263157 0'0 2013-09-23 11:06:05.263157 > 3.39 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.503826 0'0 523661:417379 [1,0] [1,0] 0'0 > 2013-09-23 11:03:51.090570 0'0 2013-09-23 11:03:51.090570 > 6.3c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.503988 0'0 523661:417238 [1,0] [1,0] 0'0 > 2013-09-23 11:02:39.078386 0'0 2013-09-23 11:02:39.078386 > 9.33 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:27.257185 0'0 523657:495562 [0,1] [0,1] 0'0 > 2013-09-23 11:06:07.263854 0'0 2013-09-23 11:06:07.263854 > 3.3a 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.000739 0'0 > 523661:495164 [0,1] [1,2] 0'0 2013-09-23 11:03:48.795123 0'0 > 2013-09-23 11:03:48.795123 > 2.3b 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.001255 0'0 > 523661:512301 [0,1] [1,2] 0'0 2013-09-23 11:02:25.775992 0'0 > 2013-09-23 11:02:25.775992 > 5.3c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.506293 0'0 523661:417196 [1,0] [1,0] 0'0 > 2013-09-23 11:05:49.048351 0'0 2013-09-23 11:05:49.048351 > 6.3f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.506450 0'0 523661:417129 [1,0] [1,0] 0'0 > 2013-09-23 11:02:50.080368 0'0 2013-09-23 11:02:50.080368 > 8.31 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.504165 0'0 523661:417146 [1,0] [1,0] 0'0 > 2013-09-23 11:07:32.069099 0'0 2013-09-23 11:07:32.069099 > 3.3b 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.002283 0'0 > 523661:495090 [0,1] [1,2] 0'0 2013-09-23 11:03:50.795702 0'0 > 2013-09-23 11:03:50.795702 > 0.38 80 0 0 0 251658240 91 91 > peering 2013-09-25 13:39:15.510073 20696'91 523661:436664 > [1,0] [1,0] 20696'91 2013-09-23 11:00:49.934208 20696'91 > 2013-09-23 11:00:49.934208 > 6.3e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.510269 0'0 523661:417122 [1,0] [1,0] 0'0 > 2013-09-23 11:02:42.079166 0'0 2013-09-23 11:02:42.079166 > 8.30 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.508362 0'0 523661:417029 [1,0] [1,0] 0'0 > 2013-09-23 11:07:26.067595 0'0 2013-09-23 11:07:26.067595 > 9.31 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.001567 0'0 > 523661:494846 [0,1] [1,2] 0'0 2013-09-23 11:06:02.262718 0'0 > 2013-09-23 11:06:02.262718 > 4.33 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.004029 0'0 > 523661:494912 [0,1] [1,2] 0'0 2013-09-23 11:04:24.803149 0'0 > 2013-09-23 11:04:24.803149 > 3.34 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.003485 0'0 > 523661:495006 [0,1] [1,2] 0'0 2013-09-23 11:03:42.793780 0'0 > 2013-09-23 11:03:42.793780 > 5.32 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.510403 0'0 523661:417206 [1,0] [1,0] 0'0 > 2013-09-23 11:05:25.338321 0'0 2013-09-23 11:05:25.338321 > 9.3e 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.002962 0'0 > 523661:494834 [0,1] [1,2] 0'0 2013-09-23 11:06:20.266547 0'0 > 2013-09-23 11:06:20.266547 > 4.32 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.002775 0'0 > 523661:494875 [0,1] [1,2] 0'0 2013-09-23 11:04:14.801043 0'0 > 2013-09-23 11:04:14.801043 > 3.35 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.002210 0'0 > 523661:494941 [0,1] [1,2] 0'0 2013-09-23 11:03:43.794079 0'0 > 2013-09-23 11:03:43.794079 > 1.37 0 0 0 0 0 3002 3002 peering > 2013-09-25 13:39:15.513761 20696'6705 523661:446171 [1,0] > [1,0] 20696'6705 2013-09-23 11:01:28.459127 20696'6705 > 2013-09-23 11:01:28.459127 > 6.30 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.003302 0'0 > 523661:494842 [0,1] [1,2] 0'0 2013-09-23 11:02:43.780806 0'0 > 2013-09-23 11:02:43.780806 > 9.3f 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.510566 0'0 523661:416947 [1,0] [1,0] 0'0 > 2013-09-23 11:06:45.058811 0'0 2013-09-23 11:06:45.058811 > 3.36 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.004326 0'0 > 523661:494942 [0,1] [1,2] 0'0 2013-09-23 11:03:45.794306 0'0 > 2013-09-23 11:03:45.794306 > 9.3c 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.003815 0'0 > 523661:494737 [0,1] [1,2] 0'0 2013-09-23 11:06:18.266225 0'0 > 2013-09-23 11:06:18.266225 > 3.37 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.987716 0'0 > 523661:494932 [0,1] [1,2] 0'0 2013-09-23 11:03:47.794937 0'0 > 2013-09-23 11:03:47.794937 > 2.36 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.514935 0'0 523661:434157 [1,0] [1,0] 0'0 > 2013-09-23 11:01:53.070940 0'0 2013-09-23 11:01:53.070940 > 5.31 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.988523 0'0 > 523661:494928 [0,1] [1,2] 0'0 2013-09-23 11:05:23.253913 0'0 > 2013-09-23 11:05:23.253913 > 6.32 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.989116 0'0 > 523661:494768 [0,1] [1,2] 0'0 2013-09-23 11:02:47.782046 0'0 > 2013-09-23 11:02:47.782046 > 9.3d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:15.513824 0'0 523661:417029 [1,0] [1,0] 0'0 > 2013-09-23 11:06:41.057822 0'0 2013-09-23 11:06:41.057822 > 4.37 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.982111 0'0 523661:417133 [1,0] [1,0] 0'0 > 2013-09-23 11:04:47.100073 0'0 2013-09-23 11:04:47.100073 > 5.36 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.982375 0'0 523661:417226 [1,0] [1,0] 0'0 > 2013-09-23 11:05:40.047401 0'0 2013-09-23 11:05:40.047401 > 8.3b 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.989701 0'0 > 523661:494665 [0,1] [1,2] 0'0 2013-09-23 11:00:02.007440 0'0 > 2013-09-23 11:00:02.007440 > 5.37 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.985122 0'0 523661:417377 [1,0] [1,0] 0'0 > 2013-09-23 11:05:42.047366 0'0 2013-09-23 11:05:42.047366 > 9.3b 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.990551 0'0 > 523661:494544 [0,1] [1,2] 0'0 2013-09-23 11:06:10.264419 0'0 > 2013-09-23 11:06:10.264419 > 3.32 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.168580 0'0 523661:417260 [1,0] [1,0] 0'0 > 2013-09-23 11:03:48.090062 0'0 2013-09-23 11:03:48.090062 > 0.30 88 0 0 0 276824064 104 104 > remapped+peering 2013-09-25 13:39:19.004903 20704'104 > 523661:28269 [0,1] [1,2] 20704'104 2013-09-23 11:01:31.395540 > 20704'1042013-09-23 11:01:31.395540 > 9.39 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.171877 0'0 523661:417293 [1,0] [1,0] 0'0 > 2013-09-23 11:06:39.057180 0'0 2013-09-23 11:06:39.057180 > 8.27 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.173181 0'0 523661:417279 [1,0] [1,0] 0'0 > 2013-09-23 11:07:09.063650 0'0 2013-09-23 11:07:09.063650 > 4.2a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.175513 0'0 523661:416866 [1,0] [1,0] 0'0 > 2013-09-23 11:04:40.098740 0'0 2013-09-23 11:04:40.098740 > 3.2d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.174027 0'0 523661:417002 [1,0] [1,0] 0'0 > 2013-09-23 11:03:37.088129 0'0 2013-09-23 11:03:37.088129 > 1.2f 2 0 0 0 4194534 3001 3001 > remapped+peering 2013-09-25 13:39:19.005054 20704'8831 > 523661:43413 [0,1] [1,2] 20704'8831 2013-09-23 11:01:53.822530 > 20704'8831 2013-09-23 11:01:53.822530 > 5.2b 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.005601 0'0 > 523661:494542 [0,1] [1,2] 0'0 2013-09-23 11:05:22.253235 0'0 > 2013-09-23 11:05:22.253235 > 6.28 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.177131 0'0 523661:416724 [1,0] [1,0] 0'0 > 2013-09-23 11:02:25.076176 0'0 2013-09-23 11:02:25.076176 > 8.26 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.173760 0'0 523661:417137 [1,0] [1,0] 0'0 > 2013-09-23 11:07:06.063173 0'0 2013-09-23 11:07:06.063173 > 9.27 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.173463 0'0 523661:417297 [1,0] [1,0] 0'0 > 2013-09-23 11:06:38.057213 0'0 2013-09-23 11:06:38.057213 > 3.2e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.178609 0'0 523661:416782 [1,0] [1,0] 0'0 > 2013-09-23 11:03:41.089176 0'0 2013-09-23 11:03:41.089176 > 0.2d 73 0 0 0 222298112 80 80 > peering 2013-09-25 13:39:18.181358 20675'80 523661:434380 > [1,0] [1,0] 20675'80 2013-09-23 11:00:46.481856 20675'80 > 2013-09-23 11:00:46.481856 > 1.2c 0 0 0 0 0 3001 3001 peering > 2013-09-25 13:39:18.178894 20704'7560 523661:447713 [1,0] > [1,0] 20704'7560 2013-09-23 11:01:27.459028 20704'7560 > 2013-09-23 11:01:27.459028 > 8.25 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.006162 0'0 > 523661:494245 [0,1] [1,2] 0'0 2013-09-23 11:06:44.272241 0'0 > 2013-09-23 11:06:44.272241 > 9.24 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.177416 0'0 523661:416862 [1,0] [1,0] 0'0 > 2013-09-23 11:06:32.055617 0'0 2013-09-23 11:06:32.055617 > 2.2e 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.006818 0'0 > 523661:511464 [0,1] [1,2] 0'0 2013-09-23 11:02:24.775846 0'0 > 2013-09-23 11:02:24.775846 > 9.25 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.181647 0'0 523661:416939 [1,0] [1,0] 0'0 > 2013-09-23 11:06:35.056025 0'0 2013-09-23 11:06:35.056025 > 3.28 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.184103 0'0 523661:416904 [1,0] [1,0] 0'0 > 2013-09-23 11:03:33.087643 0'0 2013-09-23 11:03:33.087643 > 8.23 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.007438 0'0 > 523661:494119 [0,1] [1,2] 0'0 2013-09-23 11:06:41.271171 0'0 > 2013-09-23 11:06:41.271171 > 8.22 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.006143 0'0 > 523661:494302 [0,1] [1,2] 0'0 2013-09-23 11:06:40.271294 0'0 > 2013-09-23 11:06:40.271294 > 9.23 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.005595 0'0 > 523661:494091 [0,1] [1,2] 0'0 2013-09-23 11:05:56.261064 0'0 > 2013-09-23 11:05:56.261064 > 3.2a 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.006729 0'0 > 523661:494204 [0,1] [1,2] 0'0 2013-09-23 11:03:36.792528 0'0 > 2013-09-23 11:03:36.792528 > 2.2b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.188734 0'0 523661:433860 [1,0] [1,0] 0'0 > 2013-09-23 11:01:49.070293 0'0 2013-09-23 11:01:49.070293 > 5.2c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.189022 0'0 523661:416751 [1,0] [1,0] 0'0 > 2013-09-23 11:05:24.338686 0'0 2013-09-23 11:05:24.338686 > 9.20 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.186734 0'0 523661:416758 [1,0] [1,0] 0'0 > 2013-09-23 11:06:29.054508 0'0 2013-09-23 11:06:29.054508 > 9.21 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.189293 0'0 523661:416595 [1,0] [1,0] 0'0 > 2013-09-23 11:06:30.054767 0'0 2013-09-23 11:06:30.054767 > 3.25 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.007442 0'0 > 523661:493901 [0,1] [1,2] 0'0 2013-09-23 11:03:26.790699 0'0 > 2013-09-23 11:03:26.790699 > 0.26 103 0 0 0 318767104 119 119 > remapped+peering 2013-09-25 13:39:19.008486 20704'119 > 523661:28409 [0,1] [1,2] 20704'119 2013-09-23 11:01:22.428529 > 20704'1192013-09-23 11:01:22.428529 > 2.24 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.007960 0'0 > 523661:510988 [0,1] [1,2] 0'0 2013-09-23 11:02:23.775487 0'0 > 2013-09-23 11:02:23.775487 > 5.23 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.008991 0'0 > 523661:493854 [0,1] [1,2] 0'0 2013-09-23 11:05:12.814052 0'0 > 2013-09-23 11:05:12.814052 > 6.23 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.727405 0'0 523657:494488 [0,1] [0,1] 0'0 > 2013-09-23 11:02:39.779694 0'0 2013-09-23 11:02:39.779694 > 8.2d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.197283 0'0 523661:416054 [1,0] [1,0] 0'0 > 2013-09-23 11:07:17.065762 0'0 2013-09-23 11:07:17.065762 > 9.2c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.659356 0'0 523657:494369 [0,1] [0,1] 0'0 > 2013-09-23 11:06:00.261947 0'0 2013-09-23 11:06:00.261947 > 3.27 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.197931 0'0 523661:414416 [1,0] [1,0] 0'0 > 2013-09-23 11:03:32.087522 0'0 2013-09-23 11:03:32.087522 > 1.25 1 0 0 0 620 3001 3001 peering > 2013-09-25 13:39:28.781608 20704'10630 523657:529820 [0,1] > [0,1] 20704'10630 2013-09-23 11:01:47.770633 20704'10630 > 2013-09-23 11:01:47.770633 > 6.22 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.199454 0'0 523661:414054 [1,0] [1,0] 0'0 > 2013-09-23 11:02:23.075913 0'0 2013-09-23 11:02:23.075913 > 0.23 68 0 0 0 209715200 84 84 > peering 2013-09-25 13:39:18.201190 20704'84 523661:444486 > [1,0] [1,0] 20704'84 2013-09-23 11:00:42.568776 20704'84 > 2013-09-23 11:00:42.568776 > 1.22 3 0 0 0 8390372 3001 3001 > inconsistent+peering 2013-09-25 13:39:18.200899 20704'11554 > 523661:454321 [1,0] [1,0] 20704'11554 2013-09-23 11:01:23.567303 > 20704'11554 2013-09-23 11:01:23.567303 > 2.21 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.200155 0'0 523661:431303 [1,0] [1,0] 0'0 > 2013-09-23 11:01:48.070081 0'0 2013-09-23 11:01:48.070081 > 5.26 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.201452 0'0 523661:410312 [1,0] [1,0] 0'0 > 2013-09-23 11:05:19.337891 0'0 2013-09-23 11:05:19.337891 > 4.26 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.203006 0'0 523661:409949 [1,0] [1,0] 0'0 > 2013-09-23 11:04:36.098327 0'0 2013-09-23 11:04:36.098327 > 0.22 75 0 0 0 247463936 83 83 > peering 2013-09-25 13:39:18.202030 20686'83 523661:426193 > [1,0] [1,0] 20686'83 2013-09-23 11:00:34.391753 20686'83 > 2013-09-23 11:00:34.391753 > 2.20 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.201757 0'0 523661:427300 [1,0] [1,0] 0'0 > 2013-09-23 11:01:45.069492 0'0 2013-09-23 11:01:45.069492 > 5.27 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.203267 0'0 523661:408749 [1,0] [1,0] 0'0 > 2013-09-23 11:05:21.338149 0'0 2013-09-23 11:05:21.338149 > 3.22 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.203543 0'0 523661:409272 [1,0] [1,0] 0'0 > 2013-09-23 11:03:28.086761 0'0 2013-09-23 11:03:28.086761 > 4.24 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.010074 0'0 > 523661:493595 [0,1] [1,2] 0'0 2013-09-23 11:04:12.800652 0'0 > 2013-09-23 11:04:12.800652 > 3.23 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:19.008489 0'0 > 523661:493570 [0,1] [1,2] 0'0 2013-09-23 11:03:21.789296 0'0 > 2013-09-23 11:03:21.789296 > 0.20 92 0 0 0 281018368 118 118 > remapped+peering 2013-09-25 13:39:19.009295 20704'118 > 523661:28350 [0,1] [1,2] 20704'118 2013-09-23 11:05:28.448391 > 20704'1182013-09-23 11:05:28.448391 > 1.21 1 0 0 0 1744 3002 3002 peering > 2013-09-25 13:39:18.205029 521351'7114 523661:437573 [1,0] > [1,0] 82674'7113 2013-09-23 11:01:18.222367 82674'7113 > 2013-09-23 11:01:18.222367 > 8.28 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.203833 0'0 523661:408655 [1,0] [1,0] 0'0 > 2013-09-23 11:07:14.064598 0'0 2013-09-23 11:07:14.064598 > 3.1c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.207890 0'0 523661:407418 [1,0] [1,0] 0'0 > 2013-09-23 11:03:24.086184 0'0 2013-09-23 11:03:24.086184 > 0.1f 51 0 0 0 142606336 65 65 > peering 2013-09-25 13:39:18.208698 20675'65 523661:426104 > [1,0] [1,0] 20675'65 2013-09-23 11:00:30.745891 20675'65 > 2013-09-23 11:00:30.745891 > 1.1e 1 0 0 0 4194304 3001 3001 peering > 2013-09-25 13:39:18.208427 20704'9407 523661:441375 [1,0] > [1,0] 20704'9407 2013-09-23 11:01:13.264472 20704'9407 > 2013-09-23 11:01:13.264472 > 2.1d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.208152 0'0 523661:424500 [1,0] [1,0] 0'0 > 2013-09-23 11:01:43.069238 0'0 2013-09-23 11:01:43.069238 > 5.1a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.847352 0'0 523657:493531 [0,1] [0,1] 0'0 > 2013-09-23 11:05:11.826873 0'0 2013-09-23 11:05:11.826873 > 6.19 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.847814 0'0 523657:493391 [0,1] [0,1] 0'0 > 2013-09-23 11:02:36.778942 0'0 2013-09-23 11:02:36.778942 > 3.1d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.862143 0'0 523657:493363 [0,1] [0,1] 0'0 > 2013-09-23 11:03:20.789125 0'0 2013-09-23 11:03:20.789125 > 0.1e 82 0 0 0 293601280 90 90 > peering 2013-09-25 13:39:28.863584 20704'90 523657:513334 > [0,1] [0,1] 20704'90 2013-09-23 11:01:10.965007 20704'90 > 2013-09-23 11:01:10.965007 > 1.1f 0 0 0 0 0 3002 3002 peering > 2013-09-25 13:39:28.863096 20696'15250 523657:537057 [0,1] > [0,1] 20696'15250 2013-09-23 11:01:46.767961 20696'15250 > 2013-09-23 11:01:46.767961 > 2.1c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.862609 0'0 523657:510507 [0,1] [0,1] 0'0 > 2013-09-23 11:02:20.774699 0'0 2013-09-23 11:02:20.774699 > 6.18 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.217000 0'0 523661:405320 [1,0] [1,0] 0'0 > 2013-09-23 11:02:19.075507 0'0 2013-09-23 11:02:19.075507 > 8.16 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.861633 0'0 523657:493404 [0,1] [0,1] 0'0 > 2013-09-23 11:06:38.270626 0'0 2013-09-23 11:06:38.270626 > 4.19 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.902109 0'0 523657:493189 [0,1] [0,1] 0'0 > 2013-09-23 11:04:11.800180 0'0 2013-09-23 11:04:11.800180 > 3.1e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.218396 0'0 523661:403632 [1,0] [1,0] 0'0 > 2013-09-23 11:03:27.086460 0'0 2013-09-23 11:03:27.086460 > 0.1d 75 0 0 0 201326592 123 123 > peering 2013-09-25 13:39:18.218919 521354'123 523661:420291 > [1,0] [1,0] 82674'122 2013-09-23 11:00:27.533536 82674'122 > 2013-09-23 11:00:27.533536 > 1.1c 0 0 0 0 0 3001 3001 peering > 2013-09-25 13:39:18.218669 20704'12137 523661:442487 [1,0] > [1,0] 20704'12137 2013-09-23 11:01:08.217183 20704'12137 > 2013-09-23 11:01:08.217183 > 5.18 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.902584 0'0 523657:493194 [0,1] [0,1] 0'0 > 2013-09-23 11:04:58.811264 0'0 2013-09-23 11:04:58.811264 > 8.15 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.218139 0'0 523661:403755 [1,0] [1,0] 0'0 > 2013-09-23 11:07:01.062383 0'0 2013-09-23 11:07:01.062383 > 9.14 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.902162 0'0 523657:493188 [0,1] [0,1] 0'0 > 2013-09-23 11:05:52.260195 0'0 2013-09-23 11:05:52.260195 > 6.1a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.945248 0'0 523657:493109 [0,1] [0,1] 0'0 > 2013-09-23 11:02:37.779451 0'0 2013-09-23 11:02:37.779451 > 2.1e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.926865 0'0 523657:510349 [0,1] [0,1] 0'0 > 2013-09-23 11:02:22.775290 0'0 2013-09-23 11:02:22.775290 > 1.1d 3 0 0 0 4197551 3001 3001 > inconsistent+peering 2013-09-25 13:39:28.927349 20704'12370 > 523657:532853 [0,1] [0,1] 20704'12370 2013-09-23 11:01:45.819556 > 20704'12370 2013-09-23 11:01:45.819556 > 0.1c 71 0 0 0 239075328 82 82 > peering 2013-09-25 13:39:18.228888 20680'82 523661:425827 > [1,0] [1,0] 20680'82 2013-09-23 11:00:24.131491 20680'82 > 2013-09-23 11:00:24.131491 > 9.12 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.952285 0'0 523657:493113 [0,1] [0,1] 0'0 > 2013-09-23 11:05:50.259627 0'0 2013-09-23 11:05:50.259627 > 8.13 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.231364 0'0 523661:402303 [1,0] [1,0] 0'0 > 2013-09-23 11:06:52.060561 0'0 2013-09-23 11:06:52.060561 > 5.1e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.231868 0'0 523661:402159 [1,0] [1,0] 0'0 > 2013-09-23 11:05:14.105718 0'0 2013-09-23 11:05:14.105718 > 3.18 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.231616 0'0 523661:403117 [1,0] [1,0] 0'0 > 2013-09-23 11:03:19.084858 0'0 2013-09-23 11:03:19.084858 > 1.1b 1 0 0 0 4194304 3001 3001 peering > 2013-09-25 13:39:18.232407 20704'6512 523661:430836 [1,0] > [1,0] 20704'6512 2013-09-23 11:00:56.968811 20704'6512 > 2013-09-23 11:00:56.968811 > 3.19 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.232135 0'0 523661:402921 [1,0] [1,0] 0'0 > 2013-09-23 11:03:20.085256 0'0 2013-09-23 11:03:20.085256 > 2.1b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.232621 0'0 523661:419815 [1,0] [1,0] 0'0 > 2013-09-23 11:01:41.068473 0'0 2013-09-23 11:01:41.068473 > 8.10 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.971830 0'0 523657:492785 [0,1] [0,1] 0'0 > 2013-09-23 11:06:36.270107 0'0 2013-09-23 11:06:36.270107 > 6.1e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:28.974051 0'0 523657:492646 [0,1] [0,1] 0'0 > 2013-09-23 11:02:38.779745 0'0 2013-09-23 11:02:38.779745 > 2.1a 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.235576 0'0 523661:419134 [1,0] [1,0] 0'0 > 2013-09-23 11:01:39.068202 0'0 2013-09-23 11:01:39.068202 > 2.15 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.236782 0'0 > 523661:510109 [0,1] [1,2] 0'0 2013-09-23 11:02:19.774466 0'0 > 2013-09-23 11:02:19.774466 > 1.16 1 0 0 0 4194304 3001 3001 > remapped+peering 2013-09-25 13:39:18.237308 20704'12853 > 523661:50390 [0,1] [1,2] 20704'12853 2013-09-23 11:01:42.819355 > 20704'12853 2013-09-23 11:01:42.819355 > 0.17 92 0 0 0 285212672 112 112 > remapped+peering 2013-09-25 13:39:18.237887 20704'112 > 523661:30566 [0,1] [1,2] 20704'112 2013-09-23 11:01:06.331793 > 20704'1122013-09-23 11:01:06.331793 > 8.1e 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.239217 0'0 523661:400647 [1,0] [1,0] 0'0 > 2013-09-23 11:07:04.062578 0'0 2013-09-23 11:07:04.062578 > 6.10 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.240020 0'0 523661:401228 [1,0] [1,0] 0'0 > 2013-09-23 11:02:11.074196 0'0 2013-09-23 11:02:11.074196 > 5.13 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.239501 0'0 > 523661:492716 [0,1] [1,2] 0'0 2013-09-23 11:04:47.808565 0'0 > 2013-09-23 11:04:47.808565 > 9.1c 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.241151 0'0 523661:401512 [1,0] [1,0] 0'0 > 2013-09-23 11:06:25.054055 0'0 2013-09-23 11:06:25.054055 > 8.1d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.241365 0'0 523661:401431 [1,0] [1,0] 0'0 > 2013-09-23 11:07:03.062588 0'0 2013-09-23 11:07:03.062588 > 6.13 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.244079 0'0 523661:401155 [1,0] [1,0] 0'0 > 2013-09-23 11:02:15.074715 0'0 2013-09-23 11:02:15.074715 > 1.14 2 0 0 0 516 3005 3005 > remapped+peering 2013-09-25 13:39:18.242206 521354'157507 > 523661:743212 [0,1] [1,2] 82678'157503 2013-09-23 11:01:40.771856 > 82678'157503 2013-09-23 11:01:40.771856 > 0.15 65 0 0 0 193210615 74 74 > remapped+peering 2013-09-25 13:39:18.242757 20696'74 > 523661:28349 [0,1] [1,2] 20696'74 2013-09-23 11:01:03.458628 > 20696'74 2013-09-23 11:01:03.458628 > 3.16 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.241645 0'0 > 523661:492685 [0,1] [1,2] 0'0 2013-09-23 11:03:18.789056 0'0 > 2013-09-23 11:03:18.789056 > 9.1d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.244293 0'0 523661:401439 [1,0] [1,0] 0'0 > 2013-09-23 11:06:27.054274 0'0 2013-09-23 11:06:27.054274 > 6.12 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.244610 0'0 > 523661:492444 [0,1] [1,2] 0'0 2013-09-23 11:02:33.778079 0'0 > 2013-09-23 11:02:33.778079 > 3.10 0 0 0 0 0 0 0 > remapped+peering 2013-09-25 13:39:18.245148 0'0 > 523661:492409 [0,1] [1,2] 0'0 2013-09-23 11:03:14.788151 0'0 > 2013-09-23 11:03:14.788151 > 4.17 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:18.245636 0'0 523661:401050 [1,0] [1,0] 0'0 > 2013-09-23 11:04:30.096755 0'0 2013-09-23 11:04:30.096755 > 9.1b 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.793038 0'0 523658:499439 [0,1] [0,1] 0'0 > 2013-09-23 11:05:55.260896 0'0 2013-09-23 11:05:55.260896 > 4.16 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.793778 0'0 523657:499315 [0,1] [0,1] 0'0 > 2013-09-23 11:04:09.800006 0'0 2013-09-23 11:04:09.800006 > 9.18 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.793012 0'0 523657:499310 [0,1] [0,1] 0'0 > 2013-09-23 11:05:53.260280 0'0 2013-09-23 11:05:53.260280 > 6.17 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.806847 0'0 523657:498952 [0,1] [0,1] 0'0 > 2013-09-23 10:59:57.339506 0'0 2013-09-23 10:59:57.339506 > 5.14 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.806301 0'0 523657:499082 [0,1] [0,1] 0'0 > 2013-09-23 11:04:53.814919 0'0 2013-09-23 11:04:53.814919 > 2.13 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.793518 0'0 523657:516538 [0,1] [0,1] 0'0 > 2013-09-23 11:02:11.772562 0'0 2013-09-23 11:02:11.772562 > 6.16 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.809335 0'0 523657:498793 [0,1] [0,1] 0'0 > 2013-09-23 11:02:35.778353 0'0 2013-09-23 11:02:35.778353 > 5.15 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.808804 0'0 523657:498813 [0,1] [0,1] 0'0 > 2013-09-23 11:04:55.810434 0'0 2013-09-23 11:04:55.810434 > 4.14 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.873797 0'0 523661:414546 [1,0] [1,0] 0'0 > 2013-09-23 11:04:23.095588 0'0 2013-09-23 11:04:23.095588 > 2.d 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.874038 0'0 523661:431525 [1,0] [1,0] 0'0 > 2013-09-23 11:01:37.067912 0'0 2013-09-23 11:01:37.067912 > 1.e 1 0 0 0 230 3001 3001 active > 2013-09-25 13:39:22.874378 20704'9889 523661:449295 [1,0] > [1,0] 20704'9889 2013-09-23 11:00:51.904924 20704'9889 > 2013-09-23 11:00:51.904924 > 0.f 90 0 0 0 310378496 101 101 > active+replay 2013-09-25 13:39:22.874635 20686'101 > 523661:432154 [1,0] [1,0] 20686'101 2013-09-23 11:05:18.673393 > 20686'101 2013-09-23 11:05:18.673393 > 3.d 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.825279 0'0 523657:498357 [0,1] [0,1] 0'0 > 2013-09-23 11:03:12.787394 0'0 2013-09-23 11:03:12.787394 > 9.4 0 0 0 0 0 0 0 active > 2013-09-25 13:39:22.874828 0'0 523661:414316 [1,0] [1,0] 0'0 > 2013-09-23 11:06:23.053251 0'0 2013-09-23 11:06:23.053251 > 1.c 1 0 0 0 4194304 3001 3001 peering > 2013-09-25 13:39:22.815112 20704'5377 523657:525162 [0,1] > [0,1] 20704'5377 2013-09-23 11:01:38.826408 20704'5377 > 2013-09-23 11:01:38.826408 > 0.d 74 0 0 0 239075328 85 85 > peering 2013-09-25 13:39:22.815604 20675'85 523657:516099 > [0,1] [0,1] 20675'85 2013-09-23 11:01:01.187212 20675'85 > 2013-09-23 11:01:01.187212 > 4.9 0 0 0 0 0 0 0 peering > 2013-09-25 13:39:22.816067 0'0 523657:498272 [0,1] [0,1] 0'0 > 2013-09-23 11:04:04.798651 0'0 2013-09-23 11:04:04.798651 > > root@bd-a:~# > > > Am 16.09.2013 18:47, schrieb Gregory Farnum: > > Obviously your OSDs aren't getting all the PGs up and running. Have > you followed the troubleshooting steps? > (http://ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/) > -Greg > Software Engineer #42 @ http://inktank.com | http://ceph.com > > > On Mon, Sep 16, 2013 at 6:35 AM, Markus Goldberg > <goldberg@xxxxxxxxxxxxxxxxx> wrote: > > Hi, > i must ask once again. Is there really no help to this problem? > The errors are still remaining. I can't mount the cluster anymore. All my > data is gone. > The error-messages are still changing every few seconds. > > What can i do ? > > Please help, > Markus > > Am 11.09.2013 08:39, schrieb Markus Goldberg: > > Does noone have an idea ? > I can't mount the cluster anymore. > > Thank you, > Markus > > Am 10.09.2013 09:43, schrieb Markus Goldberg: > > Hi, > i made a 'stop ceph-all' on my ceph-admin-host and then a kernel-upgrade > from 3.9 to 3.11 on all of my 3 nodes. > Ubuntu 13.04, ceph 0,68 > The kernel-upgrade required a reboot. > Now after rebooting i get the following errors: > > root@bd-a:~# ceph -s > cluster e0dbf70d-af59-42a5-b834-7ad739a7f89b > health HEALTH_WARN 133 pgs peering; 272 pgs stale; 265 pgs stuck > unclean; 2 requests are blocked > 32 sec; mds cluster is degraded > monmap e1: 3 mons at > {bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}, > election epoch 782, quorum 0,1,2 bd-0,bd-1,bd-2 > mdsmap e451467: 1/1/1 up {0=bd-0=up:replay}, 2 up:standby > osdmap e464358: 3 osds: 3 up, 3 in > pgmap v1343477: 792 pgs, 9 pools, 15145 MB data, 4986 objects > 30927 MB used, 61372 GB / 61408 GB avail > 387 active+clean > 122 stale+active > 140 stale+active+clean > 133 peering > 10 stale+active+replay > > root@bd-a:~# ceph -s > cluster e0dbf70d-af59-42a5-b834-7ad739a7f89b > health HEALTH_WARN 6 pgs down; 377 pgs peering; 296 pgs stuck unclean; > mds cluster is degraded > monmap e1: 3 mons at > {bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}, > election epoch 782, quorum 0,1,2 bd-0,bd-1,bd-2 > mdsmap e451467: 1/1/1 up {0=bd-0=up:replay}, 2 up:standby > osdmap e464400: 3 osds: 3 up, 3 in > pgmap v1343586: 792 pgs, 9 pools, 15145 MB data, 4986 objects > 31046 MB used, 61372 GB / 61408 GB avail > 142 active > 270 active+clean > 3 active+replay > 371 peering > 6 down+peering > > root@bd-a:~# ceph -s > cluster e0dbf70d-af59-42a5-b834-7ad739a7f89b > health HEALTH_WARN 257 pgs peering; 359 pgs stuck unclean; 1 requests > are blocked > 32 sec; mds cluster is degraded > monmap e1: 3 mons at > {bd-0=xxx.xxx.xxx.20:6789/0,bd-1=xxx.xxx.xxx.21:6789/0,bd-2=xxx.xxx.xxx.22:6789/0}, > election epoch 782, quorum 0,1,2 bd-0,bd-1,bd-2 > mdsmap e451467: 1/1/1 up {0=bd-0=up:replay}, 2 up:standby > osdmap e464403: 3 osds: 3 up, 3 in > pgmap v1343594: 792 pgs, 9 pools, 15145 MB data, 4986 objects > 31103 MB used, 61372 GB / 61408 GB avail > 373 active > 157 active+clean > 5 active+replay > 257 peering > > root@bd-a:~# > > As you can see above, the errors are changing, perhaps any selfrepair is on > the run in the background. But this is since 12 hours. > What should i do ? > > Thank you, > Markus > Am 09.09.2013 13:52, schrieb Yan, Zheng: > > The bug has been fixed in 3.11 kernel by commit ccca4e37b1 (libceph: fix > truncate size calculation). We don't backport cephfs bug fixes to old > kernel. please update the kernel or use ceph-fuse. Regards Yan, Zheng > > Best regards, > Tobi > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > MfG, > Markus Goldberg > > ------------------------------------------------------------------------ > Markus Goldberg | Universität Hildesheim > | Rechenzentrum > Tel +49 5121 883212 | Marienburger Platz 22, D-31141 Hildesheim, Germany > Fax +49 5121 883205 | email goldberg@xxxxxxxxxxxxxxxxx > ------------------------------------------------------------------------ > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > MfG, > Markus Goldberg > > ------------------------------------------------------------------------ > Markus Goldberg | Universität Hildesheim > | Rechenzentrum > Tel +49 5121 883212 | Marienburger Platz 22, D-31141 Hildesheim, Germany > Fax +49 5121 883205 | email goldberg@xxxxxxxxxxxxxxxxx > ------------------------------------------------------------------------ > > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > MfG, > Markus Goldberg > > ------------------------------------------------------------------------ > Markus Goldberg | Universität Hildesheim > | Rechenzentrum > Tel +49 5121 883212 | Marienburger Platz 22, D-31141 Hildesheim, Germany > Fax +49 5121 883205 | email goldberg@xxxxxxxxxxxxxxxxx > ------------------------------------------------------------------------ > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > MfG, > Markus Goldberg > > ------------------------------------------------------------------------ > Markus Goldberg | Universität Hildesheim > | Rechenzentrum > Tel +49 5121 883212 | Marienburger Platz 22, D-31141 Hildesheim, Germany > Fax +49 5121 883205 | email goldberg@xxxxxxxxxxxxxxxxx > ------------------------------------------------------------------------ _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |