08.08.2016 13:51, Wido den Hollander пишет:
>
>> Op 8 augustus 2016 om 12:49 schreef John Spray <jspray@xxxxxxxxxx>:
>>
>>
>> On Mon, Aug 8, 2016 at 9:26 AM, Dmitriy Lysenko <tavx@xxxxxxxxxx> wrote:
>>> Good day.
>>>
>>> My CephFS switched to read only
>>> This problem was previously on Hammer, but i recreated cephfs, upgraded to Jewel and problem was solved, but appeared after some time.
>>>
>>> ceph.log
>>> 2016-08-07 18:11:31.226960 mon.0 192.168.13.100:6789/0 148601 : cluster [INF] HEALTH_WARN; mds0: MDS in read-only mode
>>>
>>> ceph-mds.log:
>>> 2016-08-07 18:10:58.699731 7f9fa2ba6700 1 mds.0.cache.dir(10000000afe) commit error -22 v 1
>>> 2016-08-07 18:10:58.699755 7f9fa2ba6700 -1 log_channel(cluster) log [ERR] : failed to commit dir 10000000afe object, errno -22
>>> 2016-08-07 18:10:58.699763 7f9fa2ba6700 -1 mds.0.2271 unhandled write error (22) Invalid argument, force readonly...
>>> 2016-08-07 18:10:58.699773 7f9fa2ba6700 1 mds.0.cache force file system read-only
>>> 2016-08-07 18:10:58.699777 7f9fa2ba6700 0 log_channel(cluster) log [WRN] : force file system read-only
>>
>> The MDS is going read only because it received an error (22, aka
>> EINVAL) from an OSD when trying to write a metadata object. You need
>> to investigate why the error occurred. Are your OSDs using the same
>> Ceph version as your MDS? Look in the OSD logs for the time at which
>> the error happened to see if there is more detail about why.
>>
All OSDs using same version:
# ceph tell osd.* version
osd.0: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.1: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.2: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.3: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.4: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.5: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.6: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.7: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.8: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.9: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.10: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.11: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.12: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
osd.14: {
"version": "ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)"
}
I did not find any errors in osd logs from 18:00 to 19:00 (included in this message)
>
> You might want to add this to the mds config:
>
> debug_rados = 20
>
> That should show you which RADOS operations it is performing and you can also figure out which one failed.
>
> Like John said, might be a issue with a specific OSD.
>
> Wido
I added debug_rados into [mds] section in ceph.conf
But i've already fixed error by http://docs.ceph.com/docs/jewel/cephfs/disaster-recovery/:
cephfs-journal-tool event recover_dentries summary
cephfs-table-tool all reset session
cephfs-journal-tool journal reset
cephfs-data-scan init
cephfs-data-scan scan_extents data
cephfs-data-scan scan_inodes data
>
>> The readonly flag will clear if you restart your MDS (but it will get
>> set again if it keeps encountering errors writing to OSDs)
>>
>> John
>>
>>> I founded this object:
>>> $ rados --pool metadata ls | grep 10000000afe
>>> 10000000afe.00000000
>>>
>>> and successfully got it:
>>> $ rados --pool metadata get 10000000afe.00000000 obj
>>> $ echo $?
>>> 0
>>>
>>> How to switchout MDS from readonly mode?
>>> Are there any tools to test the CephFS system for errors?
>>>
>>> $ ceph -v
>>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>>>
>>> $ ceph fs ls
>>> name: cephfs, metadata pool: metadata, data pools: [data ]
>>>
>>> $ ceph mds stat
>>> e2283: 1/1/1 up {0=drop-03=up:active}, 3 up:standby
>>>
>>> $ ceph osd lspools
>>> 0 data,1 metadata,6 one,
>>>
>>> $ ceph osd dump | grep 'replicated size'
>>> pool 0 'data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 45647 crash_replay_interval 45 min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0
>>> pool 1 'metadata' replicated size 2 min_size 1 crush_ruleset 1 object_hash rjenkins pg_num 256 pgp_num 256 last_change 45649 min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0
>>> pool 6 'one' replicated size 3 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 512 pgp_num 512 last_change 53462 flags hashpspool min_read_recency_for_promote 1 min_write_recency_for_promote 1 stripe_width 0
>>>
>>>
>>> Thank you for help.
>>>
>>> --
>>> Dmitry Lysenko
>>> ISP Sovtest, Kursk, Russia
>>> jabber: tavx@xxxxxxxxxxxxxxxxx
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Dmitry Lysenko
ISP Sovtest, Kursk, Russia
jabber: tavx@xxxxxxxxxxxxxxxxx
2016-08-07 18:06:56.279208 7f2009e19700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0df10000 sd=38 :40248 s=2 pgs=378 cs=173 l=0 c=0x564aefeb1c80).fault with nothing to send, going to standby
2016-08-07 18:10:58.496071 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=0 pgs=0 cs=0 l=0 c=0x564ae5e18880).accept connect_seq 173 vs existing 173 state standby
2016-08-07 18:10:58.499683 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=0 pgs=0 cs=0 l=0 c=0x564ae5e18880).accept connect_seq 174 vs existing 173 state standby
2016-08-07 18:11:39.316784 7f201851e700 0 log_channel(cluster) log [INF] : 6.dc scrub starts
2016-08-07 18:11:42.391837 7f201851e700 0 log_channel(cluster) log [INF] : 6.dc scrub ok
2016-08-07 18:13:25.331038 7f2015518700 0 log_channel(cluster) log [INF] : 6.150 scrub starts
2016-08-07 18:13:27.084678 7f2017d1d700 0 log_channel(cluster) log [INF] : 6.150 scrub ok
2016-08-07 18:20:28.030515 7f2005f06700 1 leveldb: Level-0 table #748689: started
2016-08-07 18:20:28.097967 7f2005f06700 1 leveldb: Level-0 table #748689: 1343239 bytes OK
2016-08-07 18:20:28.107135 7f2005f06700 1 leveldb: Delete type=0 #748686
2016-08-07 18:25:58.563282 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=2 pgs=379 cs=175 l=0 c=0x564aefeb1c80).fault with nothing to send, going to standby
2016-08-07 18:32:46.510590 7f2013d15700 0 log_channel(cluster) log [INF] : 6.15d scrub starts
2016-08-07 18:32:50.719043 7f2013d15700 0 log_channel(cluster) log [INF] : 6.15d scrub ok
2016-08-07 18:34:02.521334 7f201851e700 0 log_channel(cluster) log [INF] : 6.19f scrub starts
2016-08-07 18:34:03.086709 7f201851e700 0 log_channel(cluster) log [INF] : 6.19f scrub ok
2016-08-07 18:45:28.624007 7f2013d15700 0 log_channel(cluster) log [INF] : 6.194 scrub starts
2016-08-07 18:45:33.382791 7f2013d15700 0 log_channel(cluster) log [INF] : 6.194 scrub ok
2016-08-07 18:47:39.642168 7f2015d19700 0 log_channel(cluster) log [INF] : 6.163 deep-scrub starts
2016-08-07 18:49:14.530911 7f2015d19700 0 log_channel(cluster) log [INF] : 6.163 deep-scrub ok
2016-08-07 18:58:04.316762 7f2005f06700 1 leveldb: Level-0 table #748691: started
2016-08-07 18:58:04.371443 7f2005f06700 1 leveldb: Level-0 table #748691: 1333000 bytes OK
2016-08-07 18:58:04.377450 7f2005f06700 1 leveldb: Delete type=0 #748688
2016-08-07 18:25:58.609976 7f56009fa700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6809/3528 pipe(0x55ab15255400 sd=227 :51698 s=2 pgs=257 cs=114 l=0 c=0x55ab19c4bd80).fault with nothing to send, going to standby
2016-08-07 18:29:01.138954 7f5618db9700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab0c48c000 sd=62 :6802 s=2 pgs=139 cs=95 l=0 c=0x55ab172f7780).fault with nothing to send, going to standby
2016-08-07 18:30:17.445857 7f55fd749700 1 leveldb: Level-0 table #129033: started
2016-08-07 18:30:17.483529 7f55fd749700 1 leveldb: Level-0 table #129033: 1307610 bytes OK
2016-08-07 18:30:17.484932 7f55fd749700 1 leveldb: Delete type=0 #129030
2016-08-07 18:30:17.488879 7f55fd749700 1 leveldb: Compacting 4@0 + 4@1 files
2016-08-07 18:30:17.598413 7f55fd749700 1 leveldb: Generated table #129034: 99575 keys, 2132808 bytes
2016-08-07 18:30:17.720297 7f55fd749700 1 leveldb: Generated table #129035: 111847 keys, 2134201 bytes
2016-08-07 18:30:17.843610 7f55fd749700 1 leveldb: Generated table #129036: 83340 keys, 2135232 bytes
2016-08-07 18:30:17.952287 7f55fd749700 1 leveldb: Generated table #129037: 91542 keys, 1935349 bytes
2016-08-07 18:30:17.952339 7f55fd749700 1 leveldb: Compacted 4@0 + 4@1 files => 8337590 bytes
2016-08-07 18:30:17.952795 7f55fd749700 1 leveldb: compacted to: files[ 0 4 8 0 0 0 0 ]
2016-08-07 18:30:17.953120 7f55fd749700 1 leveldb: Delete type=2 #129023
2016-08-07 18:30:17.953914 7f55fd749700 1 leveldb: Delete type=2 #129029
2016-08-07 18:30:17.954448 7f55fd749700 1 leveldb: Delete type=2 #129033
2016-08-07 18:30:17.955038 7f55fd749700 1 leveldb: Delete type=2 #129024
2016-08-07 18:30:17.955723 7f55fd749700 1 leveldb: Delete type=2 #129027
2016-08-07 18:30:17.956274 7f55fd749700 1 leveldb: Delete type=2 #129022
2016-08-07 18:30:17.956984 7f55fd749700 1 leveldb: Delete type=2 #129025
2016-08-07 18:30:17.957508 7f55fd749700 1 leveldb: Delete type=2 #129031
2016-08-07 18:30:33.707744 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=0 pgs=0 cs=0 l=0 c=0x55ab1ee64600).accept connect_seq 95 vs existing 95 state standby
2016-08-07 18:30:33.707913 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=0 pgs=0 cs=0 l=0 c=0x55ab1ee64600).accept connect_seq 96 vs existing 95 state standby
2016-08-07 18:45:33.812616 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=2 pgs=140 cs=97 l=0 c=0x55ab172f7780).fault with nothing to send, going to standby
2016-08-07 18:00:17.212187 7f626e97d700 0 log_channel(cluster) log [INF] : 6.1d0 scrub starts
2016-08-07 18:00:17.737206 7f626e97d700 0 log_channel(cluster) log [INF] : 6.1d0 scrub ok
2016-08-07 18:10:58.496020 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=0 pgs=0 cs=0 l=0 c=0x561448650880).accept connect_seq 124 vs existing 124 state standby
2016-08-07 18:10:58.499396 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=0 pgs=0 cs=0 l=0 c=0x561448650880).accept connect_seq 125 vs existing 124 state standby
2016-08-07 18:25:58.505631 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=2 pgs=288 cs=126 l=0 c=0x561439278a00).fault with nothing to send, going to standby
2016-08-07 18:29:21.971058 7f625c961700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6809/3528 pipe(0x561436a3a800 sd=217 :6802 s=0 pgs=0 cs=0 l=0 c=0x561428997900).accept connect_seq 151 vs existing 151 state standby
2016-08-07 18:29:21.971431 7f625c961700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6809/3528 pipe(0x561436a3a800 sd=217 :6802 s=0 pgs=0 cs=0 l=0 c=0x561428997900).accept connect_seq 152 vs existing 151 state standby
2016-08-07 18:36:46.920394 7f6266529700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.103:6805/1002280 pipe(0x5614362dc800 sd=25 :40322 s=2 pgs=380 cs=228 l=0 c=0x561436f1dc00).fault with nothing to send, going to standby
2016-08-07 18:48:04.108421 7f625e734700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6812/1003178 pipe(0x5614362f6000 sd=67 :58020 s=2 pgs=110 cs=45 l=0 c=0x561429a40900).fault with nothing to send, going to standby
2016-08-07 18:04:52.941453 7f755dc5d700 1 leveldb: Level-0 table #461987: started
2016-08-07 18:04:53.021400 7f755dc5d700 1 leveldb: Level-0 table #461987: 1344080 bytes OK
2016-08-07 18:04:53.041972 7f755dc5d700 1 leveldb: Delete type=0 #461975
2016-08-07 18:10:21.665031 7f75453e5700 0 log_channel(cluster) log [INF] : 6.84 scrub starts
2016-08-07 18:10:26.016689 7f7547bea700 0 log_channel(cluster) log [INF] : 6.84 scrub ok
2016-08-07 18:10:29.666503 7f75483eb700 0 log_channel(cluster) log [INF] : 6.5b scrub starts
2016-08-07 18:10:33.934689 7f75483eb700 0 log_channel(cluster) log [INF] : 6.5b scrub ok
2016-08-07 18:12:53.685403 7f7548bec700 0 log_channel(cluster) log [INF] : 1.0 scrub starts
2016-08-07 18:12:54.461557 7f7548bec700 0 log_channel(cluster) log [INF] : 1.0 scrub ok
2016-08-07 18:14:06.696063 7f7546be8700 0 log_channel(cluster) log [INF] : 1.4 scrub starts
2016-08-07 18:14:07.612504 7f75443e3700 0 log_channel(cluster) log [INF] : 1.4 scrub ok
2016-08-07 18:14:17.697696 7f7545be6700 0 log_channel(cluster) log [INF] : 0.1 scrub starts
2016-08-07 18:14:18.274362 7f7545be6700 0 log_channel(cluster) log [INF] : 0.1 scrub ok
2016-08-07 18:15:33.708811 7f75453e5700 0 log_channel(cluster) log [INF] : 1.75 scrub starts
2016-08-07 18:15:34.431443 7f7547bea700 0 log_channel(cluster) log [INF] : 1.75 scrub ok
2016-08-07 18:15:36.709260 7f7548bec700 0 log_channel(cluster) log [INF] : 6.96 scrub starts
2016-08-07 18:15:41.289173 7f7548bec700 0 log_channel(cluster) log [INF] : 6.96 scrub ok
2016-08-07 18:16:24.716222 7f7544be4700 0 log_channel(cluster) log [INF] : 0.7b scrub starts
2016-08-07 18:16:25.126539 7f7544be4700 0 log_channel(cluster) log [INF] : 0.7b scrub ok
2016-08-07 18:36:06.883958 7f7547bea700 0 log_channel(cluster) log [INF] : 6.61 scrub starts
2016-08-07 18:36:10.882871 7f7547bea700 0 log_channel(cluster) log [INF] : 6.61 scrub ok
2016-08-07 18:38:54.909297 7f75483eb700 0 log_channel(cluster) log [INF] : 6.ec scrub starts
2016-08-07 18:38:59.552437 7f7545be6700 0 log_channel(cluster) log [INF] : 6.ec scrub ok
2016-08-07 18:50:36.834989 7f755dc5d700 1 leveldb: Level-0 table #461989: started
2016-08-07 18:50:36.884892 7f755dc5d700 1 leveldb: Level-0 table #461989: 1352311 bytes OK
2016-08-07 18:50:36.895575 7f755dc5d700 1 leveldb: Delete type=0 #461986
2016-08-07 18:53:27.035518 7f7547bea700 0 log_channel(cluster) log [INF] : 6.f7 scrub starts
2016-08-07 18:53:31.248614 7f75453e5700 0 log_channel(cluster) log [INF] : 6.f7 scrub ok
2016-08-07 18:59:19.085423 7f7545be6700 0 log_channel(cluster) log [INF] : 6.1f scrub starts
2016-08-07 18:59:23.200839 7f75483eb700 0 log_channel(cluster) log [INF] : 6.1f scrub ok
2016-08-07 18:10:58.493202 7f37d521b700 1 leveldb: Level-0 table #809500: started
2016-08-07 18:10:58.534299 7f37d521b700 1 leveldb: Level-0 table #809500: 1328343 bytes OK
2016-08-07 18:10:58.535319 7f37d521b700 1 leveldb: Delete type=0 #809497
2016-08-07 18:10:58.540852 7f37d521b700 1 leveldb: Compacting 4@0 + 7@1 files
2016-08-07 18:10:58.552414 7f37d521b700 1 leveldb: Generated table #809501: 8171 keys, 245311 bytes
2016-08-07 18:10:58.647256 7f37d521b700 1 leveldb: Generated table #809502: 72147 keys, 2136191 bytes
2016-08-07 18:10:58.653519 7f37d521b700 1 leveldb: Generated table #809503: 4705 keys, 157333 bytes
2016-08-07 18:10:58.735525 7f37d521b700 1 leveldb: Generated table #809504: 76632 keys, 2134484 bytes
2016-08-07 18:10:58.820225 7f37d521b700 1 leveldb: Generated table #809505: 77135 keys, 2135314 bytes
2016-08-07 18:10:58.913617 7f37d521b700 1 leveldb: Generated table #809506: 89864 keys, 2134157 bytes
2016-08-07 18:10:58.970439 7f37d521b700 1 leveldb: Generated table #809507: 52206 keys, 1464019 bytes
2016-08-07 18:10:58.971443 7f37d521b700 1 leveldb: Generated table #809508: 140 keys, 14425 bytes
2016-08-07 18:10:58.971457 7f37d521b700 1 leveldb: Compacted 4@0 + 7@1 files => 10421234 bytes
2016-08-07 18:10:58.971887 7f37d521b700 1 leveldb: compacted to: files[ 0 8 32 18 0 0 0 ]
2016-08-07 18:10:58.972095 7f37d521b700 1 leveldb: Delete type=2 #809496
2016-08-07 18:10:58.972590 7f37d521b700 1 leveldb: Delete type=2 #809500
2016-08-07 18:10:58.973092 7f37d521b700 1 leveldb: Delete type=2 #809494
2016-08-07 18:10:58.973721 7f37d521b700 1 leveldb: Delete type=2 #809478
2016-08-07 18:10:58.974239 7f37d521b700 1 leveldb: Delete type=2 #809498
2016-08-07 18:10:58.974771 7f37d521b700 1 leveldb: Delete type=2 #809479
2016-08-07 18:10:58.975519 7f37d521b700 1 leveldb: Delete type=2 #809480
2016-08-07 18:10:58.976079 7f37d521b700 1 leveldb: Delete type=2 #809476
2016-08-07 18:10:58.976383 7f37d521b700 1 leveldb: Delete type=2 #809481
2016-08-07 18:10:58.976994 7f37d521b700 1 leveldb: Delete type=2 #809482
2016-08-07 18:10:58.977314 7f37d521b700 1 leveldb: Delete type=2 #809483
2016-08-07 18:42:57.941590 7f37d521b700 1 leveldb: Level-0 table #809510: started
2016-08-07 18:42:57.978108 7f37d521b700 1 leveldb: Level-0 table #809510: 1329135 bytes OK
2016-08-07 18:42:57.978928 7f37d521b700 1 leveldb: Delete type=0 #809499
2016-08-07 18:48:04.111052 7f37b382d700 0 -- 192.168.13.104:6812/1003178 >> 192.168.13.102:6802/2031964 pipe(0x5576ad5b4800 sd=212 :6812 s=2 pgs=476 cs=45 l=0 c=0x5576c4094480).fault with nothing to send, going to standby
2016-08-07 18:08:11.527627 7fc324485700 1 leveldb: Level-0 table #1207: started
2016-08-07 18:08:11.561663 7fc324485700 1 leveldb: Level-0 table #1207: 1319962 bytes OK
2016-08-07 18:08:11.562529 7fc324485700 1 leveldb: Delete type=0 #1204
2016-08-07 18:29:01.143978 7fc304a15700 0 -- 192.168.13.104:6806/1002865 >> 192.168.13.101:6802/8506 pipe(0x55b595728000 sd=37 :40026 s=2 pgs=301 cs=95 l=0 c=0x55b59ee9a780).fault with nothing to send, going to standby
2016-08-07 18:45:33.810007 7fc304a15700 0 -- 192.168.13.104:6806/1002865 >> 192.168.13.101:6802/8506 pipe(0x55b595728000 sd=37 :40118 s=2 pgs=303 cs=97 l=0 c=0x55b59ee9a780).fault with nothing to send, going to standby
2016-08-07 18:06:56.280267 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6806/2031883 pipe(0x5639c2164800 sd=79 :6805 s=2 pgs=195 cs=173 l=0 c=0x5639bcd1bb00).fault with nothing to send, going to standby
2016-08-07 18:10:58.490118 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff800).accept connect_seq 226 vs existing 226 state standby
2016-08-07 18:10:58.490494 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff800).accept connect_seq 227 vs existing 226 state standby
2016-08-07 18:10:58.506031 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff500).accept connect_seq 1 vs existing 1 state standby
2016-08-07 18:10:58.506468 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff500).accept connect_seq 2 vs existing 1 state standby
2016-08-07 18:25:58.561878 7f799227a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6806/2031883 pipe(0x5639c2164800 sd=77 :50062 s=2 pgs=196 cs=175 l=0 c=0x5639bcd1bb00).fault with nothing to send, going to standby
2016-08-07 18:25:58.614794 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=2 pgs=36 cs=3 l=0 c=0x5639d2d23a80).fault with nothing to send, going to standby
2016-08-07 18:36:46.920645 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=2 pgs=477 cs=228 l=0 c=0x5639bd19f500).fault with nothing to send, going to standby
2016-08-07 18:54:33.530533 7f7992e86700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639bda12800 sd=212 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2c00e80).accept connect_seq 3 vs existing 3 state standby
2016-08-07 18:54:33.530810 7f7992e86700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639bda12800 sd=212 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2c00e80).accept connect_seq 4 vs existing 3 state standby
2016-08-07 18:00:56.699106 7f537a961700 0 -- 192.168.13.105:6805/1002222 >> 192.168.13.101:6811/1023479 pipe(0x55e45b48d400 sd=42 :32834 s=2 pgs=90 cs=7 l=0 c=0x55e459fbe300).fault with nothing to send, going to standby
2016-08-07 18:16:12.824917 7f537508a700 1 leveldb: Level-0 table #2166: started
2016-08-07 18:16:12.860359 7f537508a700 1 leveldb: Level-0 table #2166: 1319242 bytes OK
2016-08-07 18:16:12.863360 7f537508a700 1 leveldb: Delete type=0 #2163
2016-08-07 18:12:57.686614 7f9720edb700 0 log_channel(cluster) log [INF] : 6.1de scrub starts
2016-08-07 18:12:59.531372 7f971e6d6700 0 log_channel(cluster) log [INF] : 6.1de scrub ok
2016-08-07 18:14:03.695573 7f971f6d8700 0 log_channel(cluster) log [INF] : 6.187 scrub starts
2016-08-07 18:14:05.689589 7f971f6d8700 0 log_channel(cluster) log [INF] : 6.187 scrub ok
2016-08-07 18:14:08.698230 7f97206da700 0 log_channel(cluster) log [INF] : 6.1fd deep-scrub starts
2016-08-07 18:15:11.653405 7f97206da700 0 log_channel(cluster) log [INF] : 6.1fd deep-scrub ok
2016-08-07 18:15:33.706971 7f97216dc700 0 log_channel(cluster) log [INF] : 6.188 scrub starts
2016-08-07 18:15:35.422611 7f97216dc700 0 log_channel(cluster) log [INF] : 6.188 scrub ok
2016-08-07 18:15:35.707872 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.a1 scrub starts
2016-08-07 18:15:36.015697 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.a1 scrub ok
2016-08-07 18:16:25.714768 7f971eed7700 0 log_channel(cluster) log [INF] : 0.a2 deep-scrub starts
2016-08-07 18:16:28.869516 7f971eed7700 0 log_channel(cluster) log [INF] : 0.a2 deep-scrub ok
2016-08-07 18:21:33.759282 7f971eed7700 0 log_channel(cluster) log [INF] : 1.11 scrub starts
2016-08-07 18:21:33.836549 7f971eed7700 0 log_channel(cluster) log [INF] : 1.11 scrub ok
2016-08-07 18:22:31.767734 7f97206da700 0 log_channel(cluster) log [INF] : 6.22 scrub starts
2016-08-07 18:22:33.313274 7f97206da700 0 log_channel(cluster) log [INF] : 6.22 scrub ok
2016-08-07 18:22:58.771600 7f971eed7700 0 log_channel(cluster) log [INF] : 6.c5 scrub starts
2016-08-07 18:23:02.083532 7f97216dc700 0 log_channel(cluster) log [INF] : 6.c5 scrub ok
2016-08-07 18:23:03.772368 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.b0 scrub starts
2016-08-07 18:23:04.173474 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.b0 scrub ok
2016-08-07 18:23:07.772834 7f971e6d6700 0 log_channel(cluster) log [INF] : 1.49 deep-scrub starts
2016-08-07 18:23:10.844818 7f971e6d6700 0 log_channel(cluster) log [INF] : 1.49 deep-scrub ok
2016-08-07 18:23:23.775001 7f971eed7700 0 log_channel(cluster) log [INF] : 1.6b scrub starts
2016-08-07 18:23:24.023631 7f971eed7700 0 log_channel(cluster) log [INF] : 1.6b scrub ok
2016-08-07 18:23:27.775605 7f971f6d8700 0 log_channel(cluster) log [INF] : 0.9c deep-scrub starts
2016-08-07 18:23:36.506531 7f971f6d8700 0 log_channel(cluster) log [INF] : 0.9c deep-scrub ok
2016-08-07 18:25:58.615908 7f97161a5700 0 -- 192.168.13.100:6801/17153 >> 192.168.13.103:6805/1002280 pipe(0x564b9b1d6000 sd=76 :44522 s=2 pgs=381 cs=3 l=0 c=0x564b9b17aa80).fault with nothing to send, going to standby
2016-08-07 18:28:08.668440 7f97393f8700 1 leveldb: Level-0 table #846824: started
2016-08-07 18:28:08.731154 7f97393f8700 1 leveldb: Level-0 table #846824: 1326026 bytes OK
2016-08-07 18:28:08.739001 7f97393f8700 1 leveldb: Delete type=0 #846821
2016-08-07 18:28:30.819476 7f9720edb700 0 log_channel(cluster) log [INF] : 6.198 scrub starts
2016-08-07 18:28:32.394873 7f9720edb700 0 log_channel(cluster) log [INF] : 6.198 scrub ok
2016-08-07 18:28:32.819816 7f9720edb700 0 log_channel(cluster) log [INF] : 6.11b scrub starts
2016-08-07 18:28:35.096006 7f9720edb700 0 log_channel(cluster) log [INF] : 6.11b scrub ok
2016-08-07 18:28:41.820914 7f97226de700 0 log_channel(cluster) log [INF] : 1.64 scrub starts
2016-08-07 18:28:42.735759 7f971fed9700 0 log_channel(cluster) log [INF] : 1.64 scrub ok
2016-08-07 18:35:52.884927 7f97206da700 0 log_channel(cluster) log [INF] : 6.1d5 scrub starts
2016-08-07 18:35:55.023186 7f97206da700 0 log_channel(cluster) log [INF] : 6.1d5 scrub ok
2016-08-07 18:48:08.990171 7f971eed7700 0 log_channel(cluster) log [INF] : 0.b1 deep-scrub starts
2016-08-07 18:48:11.803539 7f971eed7700 0 log_channel(cluster) log [INF] : 0.b1 deep-scrub ok
2016-08-07 18:54:35.045105 7f971e6d6700 0 log_channel(cluster) log [INF] : 6.9e scrub starts
2016-08-07 18:54:38.216295 7f9720edb700 0 log_channel(cluster) log [INF] : 6.9e scrub ok
2016-08-07 18:00:56.701446 7efcaf4ba700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d14bf400 sd=49 :6811 s=2 pgs=208 cs=7 l=0 c=0x55c3ca840280).fault with nothing to send, going to standby
2016-08-07 18:02:29.204759 7efcac188700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d15e3400 sd=233 :6811 s=0 pgs=0 cs=0 l=0 c=0x55c3c27fa000).accept connect_seq 7 vs existing 7 state standby
2016-08-07 18:02:29.204931 7efcac188700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d15e3400 sd=233 :6811 s=0 pgs=0 cs=0 l=0 c=0x55c3c27fa000).accept connect_seq 8 vs existing 7 state standby
2016-08-07 18:03:33.237115 7efcab574700 1 leveldb: Level-0 table #1145712: started
2016-08-07 18:03:33.288520 7efcab574700 1 leveldb: Level-0 table #1145712: 1329529 bytes OK
2016-08-07 18:03:33.296838 7efcab574700 1 leveldb: Delete type=0 #1145709
2016-08-07 18:40:15.608683 7efcab574700 1 leveldb: Level-0 table #1145714: started
2016-08-07 18:40:15.662852 7efcab574700 1 leveldb: Level-0 table #1145714: 1330478 bytes OK
2016-08-07 18:40:15.688457 7efcab574700 1 leveldb: Delete type=0 #1145711
2016-08-07 18:06:18.094825 7f7922b03700 1 leveldb: Level-0 table #697958: started
2016-08-07 18:06:18.150402 7f7922b03700 1 leveldb: Level-0 table #697958: 1316372 bytes OK
2016-08-07 18:06:18.159595 7f7922b03700 1 leveldb: Delete type=0 #697955
2016-08-07 18:10:19.665119 7f793412e700 0 log_channel(cluster) log [INF] : 1.8c scrub starts
2016-08-07 18:10:19.744230 7f793412e700 0 log_channel(cluster) log [INF] : 1.8c scrub ok
2016-08-07 18:12:54.685810 7f793412e700 0 log_channel(cluster) log [INF] : 6.1bd scrub starts
2016-08-07 18:12:56.289243 7f7936933700 0 log_channel(cluster) log [INF] : 6.1bd scrub ok
2016-08-07 18:13:03.687075 7f7935130700 0 log_channel(cluster) log [INF] : 6.19d scrub starts
2016-08-07 18:13:05.800720 7f7935130700 0 log_channel(cluster) log [INF] : 6.19d scrub ok
2016-08-07 18:15:37.708937 7f7935931700 0 log_channel(cluster) log [INF] : 6.133 scrub starts
2016-08-07 18:15:39.191681 7f793312c700 0 log_channel(cluster) log [INF] : 6.133 scrub ok
2016-08-07 18:17:36.727288 7f793212a700 0 log_channel(cluster) log [INF] : 6.18f deep-scrub starts
2016-08-07 18:18:36.941021 7f793212a700 0 log_channel(cluster) log [INF] : 6.18f deep-scrub ok
2016-08-07 18:21:34.760849 7f793392d700 0 log_channel(cluster) log [INF] : 6.6 scrub starts
2016-08-07 18:21:36.227213 7f7936132700 0 log_channel(cluster) log [INF] : 6.6 scrub ok
2016-08-07 18:21:40.761808 7f793392d700 0 log_channel(cluster) log [INF] : 0.6a scrub starts
2016-08-07 18:21:41.026307 7f7936132700 0 log_channel(cluster) log [INF] : 0.6a scrub ok
2016-08-07 18:21:44.762159 7f7936132700 0 log_channel(cluster) log [INF] : 1.74 scrub starts
2016-08-07 18:21:45.112349 7f7936132700 0 log_channel(cluster) log [INF] : 1.74 scrub ok
2016-08-07 18:21:50.763198 7f793212a700 0 log_channel(cluster) log [INF] : 0.72 scrub starts
2016-08-07 18:21:50.920473 7f793492f700 0 log_channel(cluster) log [INF] : 0.72 scrub ok
2016-08-07 18:22:28.767827 7f793492f700 0 log_channel(cluster) log [INF] : 1.d1 scrub starts
2016-08-07 18:22:29.533751 7f793212a700 0 log_channel(cluster) log [INF] : 1.d1 scrub ok
2016-08-07 18:22:30.768151 7f7936132700 0 log_channel(cluster) log [INF] : 0.c4 deep-scrub starts
2016-08-07 18:22:41.843801 7f7936132700 0 log_channel(cluster) log [INF] : 0.c4 deep-scrub ok
2016-08-07 18:28:32.819797 7f793412e700 0 log_channel(cluster) log [INF] : 6.69 scrub starts
2016-08-07 18:28:36.498539 7f7936933700 0 log_channel(cluster) log [INF] : 6.69 scrub ok
2016-08-07 18:42:08.681984 7f7922b03700 1 leveldb: Level-0 table #697960: started
2016-08-07 18:42:08.749477 7f7922b03700 1 leveldb: Level-0 table #697960: 1327032 bytes OK
2016-08-07 18:42:08.760519 7f7922b03700 1 leveldb: Delete type=0 #697957
2016-08-07 18:50:50.012726 7f7935931700 0 log_channel(cluster) log [INF] : 6.d4 scrub starts
2016-08-07 18:50:51.418525 7f793312c700 0 log_channel(cluster) log [INF] : 6.d4 scrub ok
2016-08-07 18:51:59.023257 7f7936132700 0 log_channel(cluster) log [INF] : 6.132 scrub starts
2016-08-07 18:52:00.912383 7f7936132700 0 log_channel(cluster) log [INF] : 6.132 scrub ok
2016-08-07 18:59:40.088618 7f793212a700 0 log_channel(cluster) log [INF] : 0.3b scrub starts
2016-08-07 18:59:40.244603 7f793212a700 0 log_channel(cluster) log [INF] : 0.3b scrub ok
2016-08-07 18:07:33.056055 7fdd3c7e9700 0 log_channel(cluster) log [INF] : 6.cc scrub starts
2016-08-07 18:07:33.410295 7fdd3c7e9700 0 log_channel(cluster) log [INF] : 6.cc scrub ok
2016-08-07 18:21:42.167712 7fdd3d7eb700 0 log_channel(cluster) log [INF] : 0.ac scrub starts
2016-08-07 18:21:42.410908 7fdd3d7eb700 0 log_channel(cluster) log [INF] : 0.ac scrub ok
2016-08-07 18:25:58.503988 7fdd28f79700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.102:6802/2031964 pipe(0x55789fc1e800 sd=73 :51794 s=2 pgs=478 cs=126 l=0 c=0x55789e863000).fault with nothing to send, going to standby
2016-08-07 18:25:58.513461 7fdd30254700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.104:6809/3528 pipe(0x5578a0dac000 sd=22 :47260 s=2 pgs=258 cs=169 l=0 c=0x5578a45fca00).fault with nothing to send, going to standby
2016-08-07 18:46:15.238980 7fdd30254700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.104:6809/3528 pipe(0x5578a0dac000 sd=22 :47284 s=2 pgs=260 cs=171 l=0 c=0x5578a45fca00).fault with nothing to send, going to standby
2016-08-07 18:46:52.801225 7fdd2ceb1700 1 leveldb: Level-0 table #2752: started
2016-08-07 18:46:52.834546 7fdd2ceb1700 1 leveldb: Level-0 table #2752: 1317345 bytes OK
2016-08-07 18:46:52.836676 7fdd2ceb1700 1 leveldb: Delete type=0 #2749
2016-08-07 18:46:52.838322 7fdd2ceb1700 1 leveldb: Compacting 4@0 + 5@1 files
2016-08-07 18:46:52.962256 7fdd2ceb1700 1 leveldb: Generated table #2753: 132257 keys, 2132791 bytes
2016-08-07 18:46:53.072736 7fdd2ceb1700 1 leveldb: Generated table #2754: 116913 keys, 2134008 bytes
2016-08-07 18:46:53.195075 7fdd2ceb1700 1 leveldb: Generated table #2755: 127096 keys, 2133069 bytes
2016-08-07 18:46:53.300826 7fdd2ceb1700 1 leveldb: Generated table #2756: 106521 keys, 2133407 bytes
2016-08-07 18:46:53.430719 7fdd2ceb1700 1 leveldb: Generated table #2757: 164841 keys, 1711532 bytes
2016-08-07 18:46:53.430733 7fdd2ceb1700 1 leveldb: Compacted 4@0 + 5@1 files => 10244807 bytes
2016-08-07 18:46:53.433286 7fdd2ceb1700 1 leveldb: compacted to: files[ 0 5 11 0 0 0 0 ]
2016-08-07 18:46:53.433467 7fdd2ceb1700 1 leveldb: Delete type=2 #2750
2016-08-07 18:46:53.433894 7fdd2ceb1700 1 leveldb: Delete type=2 #2740
2016-08-07 18:46:53.434465 7fdd2ceb1700 1 leveldb: Delete type=2 #2741
2016-08-07 18:46:53.434999 7fdd2ceb1700 1 leveldb: Delete type=2 #2742
2016-08-07 18:46:53.435514 7fdd2ceb1700 1 leveldb: Delete type=2 #2743
2016-08-07 18:46:53.435987 7fdd2ceb1700 1 leveldb: Delete type=2 #2744
2016-08-07 18:46:53.436354 7fdd2ceb1700 1 leveldb: Delete type=2 #2752
2016-08-07 18:46:53.436694 7fdd2ceb1700 1 leveldb: Delete type=2 #2746
2016-08-07 18:46:53.437045 7fdd2ceb1700 1 leveldb: Delete type=2 #2748
2016-08-07 18:01:02.235681 7fa75467e700 1 leveldb: Level-0 table #856024: started
2016-08-07 18:01:02.288072 7fa75467e700 1 leveldb: Level-0 table #856024: 1328433 bytes OK
2016-08-07 18:01:02.296559 7fa75467e700 1 leveldb: Delete type=0 #856021
2016-08-07 18:15:12.352219 7fa76228b700 0 log_channel(cluster) log [INF] : 6.144 scrub starts
2016-08-07 18:15:16.733294 7fa764a90700 0 log_channel(cluster) log [INF] : 6.144 scrub ok
2016-08-07 18:18:37.381090 7fa766a94700 0 log_channel(cluster) log [INF] : 6.1ea scrub starts
2016-08-07 18:18:39.128517 7fa76428f700 0 log_channel(cluster) log [INF] : 6.1ea scrub ok
2016-08-07 18:18:40.381580 7fa76428f700 0 log_channel(cluster) log [INF] : 6.195 scrub starts
2016-08-07 18:18:42.145238 7fa766a94700 0 log_channel(cluster) log [INF] : 6.195 scrub ok
2016-08-07 18:18:42.382031 7fa765a92700 0 log_channel(cluster) log [INF] : 6.160 deep-scrub starts
2016-08-07 18:19:47.432860 7fa76328d700 0 log_channel(cluster) log [INF] : 6.160 deep-scrub ok
2016-08-07 18:19:51.389956 7fa76328d700 0 log_channel(cluster) log [INF] : 1.4d deep-scrub starts
2016-08-07 18:19:55.143503 7fa765a92700 0 log_channel(cluster) log [INF] : 1.4d deep-scrub ok
2016-08-07 18:20:09.392520 7fa765291700 0 log_channel(cluster) log [INF] : 6.c1 scrub starts
2016-08-07 18:20:11.469277 7fa765291700 0 log_channel(cluster) log [INF] : 6.c1 scrub ok
2016-08-07 18:32:35.958425 7fa75467e700 1 leveldb: Level-0 table #856026: started
2016-08-07 18:32:36.007445 7fa75467e700 1 leveldb: Level-0 table #856026: 1344812 bytes OK
2016-08-07 18:32:36.018939 7fa75467e700 1 leveldb: Delete type=0 #856023
2016-08-07 18:33:20.514523 7fa766293700 0 log_channel(cluster) log [INF] : 6.1f5 scrub starts
2016-08-07 18:33:22.418910 7fa766293700 0 log_channel(cluster) log [INF] : 6.1f5 scrub ok
2016-08-07 18:10:58.484575 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c17b22ab80).accept connect_seq 112 vs existing 112 state standby
2016-08-07 18:10:58.484811 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c17b22ab80).accept connect_seq 113 vs existing 112 state standby
2016-08-07 18:10:58.493103 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c177128600).accept connect_seq 167 vs existing 167 state standby
2016-08-07 18:10:58.493322 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c177128600).accept connect_seq 168 vs existing 167 state standby
2016-08-07 18:25:58.513957 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=2 pgs=289 cs=169 l=0 c=0x55c198811500).fault with nothing to send, going to standby
2016-08-07 18:25:58.608660 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=2 pgs=302 cs=114 l=0 c=0x55c198810180).fault with nothing to send, going to standby
2016-08-07 18:31:15.196342 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c19ab8b480).accept connect_seq 169 vs existing 169 state standby
2016-08-07 18:31:15.196783 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c19ab8b480).accept connect_seq 170 vs existing 169 state standby
2016-08-07 18:40:35.312687 7fb278a18700 1 leveldb: Level-0 table #675679: started
2016-08-07 18:40:35.348444 7fb278a18700 1 leveldb: Level-0 table #675679: 1331634 bytes OK
2016-08-07 18:40:35.349305 7fb278a18700 1 leveldb: Delete type=0 #675676
2016-08-07 18:46:15.239328 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=2 pgs=290 cs=171 l=0 c=0x55c198811500).fault with nothing to send, going to standby
osd.2 2016-08-07 18:00:17.212187 7f626e97d700 0 log_channel(cluster) log [INF] : 6.1d0 scrub starts
osd.2 2016-08-07 18:00:17.737206 7f626e97d700 0 log_channel(cluster) log [INF] : 6.1d0 scrub ok
osd.7 2016-08-07 18:00:56.699106 7f537a961700 0 -- 192.168.13.105:6805/1002222 >> 192.168.13.101:6811/1023479 pipe(0x55e45b48d400 sd=42 :32834 s=2 pgs=90 cs=7 l=0 c=0x55e459fbe300).fault with nothing to send, going to standby
osd.9 2016-08-07 18:00:56.701446 7efcaf4ba700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d14bf400 sd=49 :6811 s=2 pgs=208 cs=7 l=0 c=0x55c3ca840280).fault with nothing to send, going to standby
osd.12 2016-08-07 18:01:02.235681 7fa75467e700 1 leveldb: Level-0 table #856024: started
osd.12 2016-08-07 18:01:02.288072 7fa75467e700 1 leveldb: Level-0 table #856024: 1328433 bytes OK
osd.12 2016-08-07 18:01:02.296559 7fa75467e700 1 leveldb: Delete type=0 #856021
osd.9 2016-08-07 18:02:29.204759 7efcac188700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d15e3400 sd=233 :6811 s=0 pgs=0 cs=0 l=0 c=0x55c3c27fa000).accept connect_seq 7 vs existing 7 state standby
osd.9 2016-08-07 18:02:29.204931 7efcac188700 0 -- 192.168.13.101:6811/1023479 >> 192.168.13.105:6805/1002222 pipe(0x55c3d15e3400 sd=233 :6811 s=0 pgs=0 cs=0 l=0 c=0x55c3c27fa000).accept connect_seq 8 vs existing 7 state standby
osd.9 2016-08-07 18:03:33.237115 7efcab574700 1 leveldb: Level-0 table #1145712: started
osd.9 2016-08-07 18:03:33.288520 7efcab574700 1 leveldb: Level-0 table #1145712: 1329529 bytes OK
osd.9 2016-08-07 18:03:33.296838 7efcab574700 1 leveldb: Delete type=0 #1145709
osd.3 2016-08-07 18:04:52.941453 7f755dc5d700 1 leveldb: Level-0 table #461987: started
osd.3 2016-08-07 18:04:53.021400 7f755dc5d700 1 leveldb: Level-0 table #461987: 1344080 bytes OK
osd.3 2016-08-07 18:04:53.041972 7f755dc5d700 1 leveldb: Delete type=0 #461975
osd.10 2016-08-07 18:06:18.094825 7f7922b03700 1 leveldb: Level-0 table #697958: started
osd.10 2016-08-07 18:06:18.150402 7f7922b03700 1 leveldb: Level-0 table #697958: 1316372 bytes OK
osd.10 2016-08-07 18:06:18.159595 7f7922b03700 1 leveldb: Delete type=0 #697955
osd.0 2016-08-07 18:06:56.279208 7f2009e19700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0df10000 sd=38 :40248 s=2 pgs=378 cs=173 l=0 c=0x564aefeb1c80).fault with nothing to send, going to standby
osd.6 2016-08-07 18:06:56.280267 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6806/2031883 pipe(0x5639c2164800 sd=79 :6805 s=2 pgs=195 cs=173 l=0 c=0x5639bcd1bb00).fault with nothing to send, going to standby
osd.11 2016-08-07 18:07:33.056055 7fdd3c7e9700 0 log_channel(cluster) log [INF] : 6.cc scrub starts
osd.11 2016-08-07 18:07:33.410295 7fdd3c7e9700 0 log_channel(cluster) log [INF] : 6.cc scrub ok
osd.5 2016-08-07 18:08:11.527627 7fc324485700 1 leveldb: Level-0 table #1207: started
osd.5 2016-08-07 18:08:11.561663 7fc324485700 1 leveldb: Level-0 table #1207: 1319962 bytes OK
osd.5 2016-08-07 18:08:11.562529 7fc324485700 1 leveldb: Delete type=0 #1204
osd.10 2016-08-07 18:10:19.665119 7f793412e700 0 log_channel(cluster) log [INF] : 1.8c scrub starts
osd.10 2016-08-07 18:10:19.744230 7f793412e700 0 log_channel(cluster) log [INF] : 1.8c scrub ok
osd.3 2016-08-07 18:10:21.665031 7f75453e5700 0 log_channel(cluster) log [INF] : 6.84 scrub starts
osd.3 2016-08-07 18:10:26.016689 7f7547bea700 0 log_channel(cluster) log [INF] : 6.84 scrub ok
osd.3 2016-08-07 18:10:29.666503 7f75483eb700 0 log_channel(cluster) log [INF] : 6.5b scrub starts
osd.3 2016-08-07 18:10:33.934689 7f75483eb700 0 log_channel(cluster) log [INF] : 6.5b scrub ok
osd.14 2016-08-07 18:10:58.484575 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c17b22ab80).accept connect_seq 112 vs existing 112 state standby
osd.14 2016-08-07 18:10:58.484811 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c17b22ab80).accept connect_seq 113 vs existing 112 state standby
osd.6 2016-08-07 18:10:58.490118 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff800).accept connect_seq 226 vs existing 226 state standby
osd.6 2016-08-07 18:10:58.490494 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff800).accept connect_seq 227 vs existing 226 state standby
osd.14 2016-08-07 18:10:58.493103 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c177128600).accept connect_seq 167 vs existing 167 state standby
osd.4 2016-08-07 18:10:58.493202 7f37d521b700 1 leveldb: Level-0 table #809500: started
osd.14 2016-08-07 18:10:58.493322 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c177128600).accept connect_seq 168 vs existing 167 state standby
osd.2 2016-08-07 18:10:58.496020 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=0 pgs=0 cs=0 l=0 c=0x561448650880).accept connect_seq 124 vs existing 124 state standby
osd.0 2016-08-07 18:10:58.496071 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=0 pgs=0 cs=0 l=0 c=0x564ae5e18880).accept connect_seq 173 vs existing 173 state standby
osd.2 2016-08-07 18:10:58.499396 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=0 pgs=0 cs=0 l=0 c=0x561448650880).accept connect_seq 125 vs existing 124 state standby
osd.0 2016-08-07 18:10:58.499683 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=0 pgs=0 cs=0 l=0 c=0x564ae5e18880).accept connect_seq 174 vs existing 173 state standby
osd.6 2016-08-07 18:10:58.506031 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff500).accept connect_seq 1 vs existing 1 state standby
osd.6 2016-08-07 18:10:58.506468 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2bff500).accept connect_seq 2 vs existing 1 state standby
osd.4 2016-08-07 18:10:58.534299 7f37d521b700 1 leveldb: Level-0 table #809500: 1328343 bytes OK
osd.4 2016-08-07 18:10:58.535319 7f37d521b700 1 leveldb: Delete type=0 #809497
osd.4 2016-08-07 18:10:58.540852 7f37d521b700 1 leveldb: Compacting 4@0 + 7@1 files
osd.4 2016-08-07 18:10:58.552414 7f37d521b700 1 leveldb: Generated table #809501: 8171 keys, 245311 bytes
osd.4 2016-08-07 18:10:58.647256 7f37d521b700 1 leveldb: Generated table #809502: 72147 keys, 2136191 bytes
osd.4 2016-08-07 18:10:58.653519 7f37d521b700 1 leveldb: Generated table #809503: 4705 keys, 157333 bytes
osd.4 2016-08-07 18:10:58.735525 7f37d521b700 1 leveldb: Generated table #809504: 76632 keys, 2134484 bytes
osd.4 2016-08-07 18:10:58.820225 7f37d521b700 1 leveldb: Generated table #809505: 77135 keys, 2135314 bytes
osd.4 2016-08-07 18:10:58.913617 7f37d521b700 1 leveldb: Generated table #809506: 89864 keys, 2134157 bytes
osd.4 2016-08-07 18:10:58.970439 7f37d521b700 1 leveldb: Generated table #809507: 52206 keys, 1464019 bytes
osd.4 2016-08-07 18:10:58.971443 7f37d521b700 1 leveldb: Generated table #809508: 140 keys, 14425 bytes
osd.4 2016-08-07 18:10:58.971457 7f37d521b700 1 leveldb: Compacted 4@0 + 7@1 files => 10421234 bytes
osd.4 2016-08-07 18:10:58.971887 7f37d521b700 1 leveldb: compacted to: files[ 0 8 32 18 0 0 0 ]
osd.4 2016-08-07 18:10:58.972095 7f37d521b700 1 leveldb: Delete type=2 #809496
osd.4 2016-08-07 18:10:58.972590 7f37d521b700 1 leveldb: Delete type=2 #809500
osd.4 2016-08-07 18:10:58.973092 7f37d521b700 1 leveldb: Delete type=2 #809494
osd.4 2016-08-07 18:10:58.973721 7f37d521b700 1 leveldb: Delete type=2 #809478
osd.4 2016-08-07 18:10:58.974239 7f37d521b700 1 leveldb: Delete type=2 #809498
osd.4 2016-08-07 18:10:58.974771 7f37d521b700 1 leveldb: Delete type=2 #809479
osd.4 2016-08-07 18:10:58.975519 7f37d521b700 1 leveldb: Delete type=2 #809480
osd.4 2016-08-07 18:10:58.976079 7f37d521b700 1 leveldb: Delete type=2 #809476
osd.4 2016-08-07 18:10:58.976383 7f37d521b700 1 leveldb: Delete type=2 #809481
osd.4 2016-08-07 18:10:58.976994 7f37d521b700 1 leveldb: Delete type=2 #809482
osd.4 2016-08-07 18:10:58.977314 7f37d521b700 1 leveldb: Delete type=2 #809483
osd.0 2016-08-07 18:11:39.316784 7f201851e700 0 log_channel(cluster) log [INF] : 6.dc scrub starts
osd.0 2016-08-07 18:11:42.391837 7f201851e700 0 log_channel(cluster) log [INF] : 6.dc scrub ok
osd.3 2016-08-07 18:12:53.685403 7f7548bec700 0 log_channel(cluster) log [INF] : 1.0 scrub starts
osd.3 2016-08-07 18:12:54.461557 7f7548bec700 0 log_channel(cluster) log [INF] : 1.0 scrub ok
osd.10 2016-08-07 18:12:54.685810 7f793412e700 0 log_channel(cluster) log [INF] : 6.1bd scrub starts
osd.10 2016-08-07 18:12:56.289243 7f7936933700 0 log_channel(cluster) log [INF] : 6.1bd scrub ok
osd.8 2016-08-07 18:12:57.686614 7f9720edb700 0 log_channel(cluster) log [INF] : 6.1de scrub starts
osd.8 2016-08-07 18:12:59.531372 7f971e6d6700 0 log_channel(cluster) log [INF] : 6.1de scrub ok
osd.10 2016-08-07 18:13:03.687075 7f7935130700 0 log_channel(cluster) log [INF] : 6.19d scrub starts
osd.10 2016-08-07 18:13:05.800720 7f7935130700 0 log_channel(cluster) log [INF] : 6.19d scrub ok
osd.0 2016-08-07 18:13:25.331038 7f2015518700 0 log_channel(cluster) log [INF] : 6.150 scrub starts
osd.0 2016-08-07 18:13:27.084678 7f2017d1d700 0 log_channel(cluster) log [INF] : 6.150 scrub ok
osd.8 2016-08-07 18:14:03.695573 7f971f6d8700 0 log_channel(cluster) log [INF] : 6.187 scrub starts
osd.8 2016-08-07 18:14:05.689589 7f971f6d8700 0 log_channel(cluster) log [INF] : 6.187 scrub ok
osd.3 2016-08-07 18:14:06.696063 7f7546be8700 0 log_channel(cluster) log [INF] : 1.4 scrub starts
osd.3 2016-08-07 18:14:07.612504 7f75443e3700 0 log_channel(cluster) log [INF] : 1.4 scrub ok
osd.8 2016-08-07 18:14:08.698230 7f97206da700 0 log_channel(cluster) log [INF] : 6.1fd deep-scrub starts
osd.3 2016-08-07 18:14:17.697696 7f7545be6700 0 log_channel(cluster) log [INF] : 0.1 scrub starts
osd.3 2016-08-07 18:14:18.274362 7f7545be6700 0 log_channel(cluster) log [INF] : 0.1 scrub ok
osd.8 2016-08-07 18:15:11.653405 7f97206da700 0 log_channel(cluster) log [INF] : 6.1fd deep-scrub ok
osd.12 2016-08-07 18:15:12.352219 7fa76228b700 0 log_channel(cluster) log [INF] : 6.144 scrub starts
osd.12 2016-08-07 18:15:16.733294 7fa764a90700 0 log_channel(cluster) log [INF] : 6.144 scrub ok
osd.8 2016-08-07 18:15:33.706971 7f97216dc700 0 log_channel(cluster) log [INF] : 6.188 scrub starts
osd.3 2016-08-07 18:15:33.708811 7f75453e5700 0 log_channel(cluster) log [INF] : 1.75 scrub starts
osd.3 2016-08-07 18:15:34.431443 7f7547bea700 0 log_channel(cluster) log [INF] : 1.75 scrub ok
osd.8 2016-08-07 18:15:35.422611 7f97216dc700 0 log_channel(cluster) log [INF] : 6.188 scrub ok
osd.8 2016-08-07 18:15:35.707872 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.a1 scrub starts
osd.8 2016-08-07 18:15:36.015697 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.a1 scrub ok
osd.3 2016-08-07 18:15:36.709260 7f7548bec700 0 log_channel(cluster) log [INF] : 6.96 scrub starts
osd.10 2016-08-07 18:15:37.708937 7f7935931700 0 log_channel(cluster) log [INF] : 6.133 scrub starts
osd.10 2016-08-07 18:15:39.191681 7f793312c700 0 log_channel(cluster) log [INF] : 6.133 scrub ok
osd.3 2016-08-07 18:15:41.289173 7f7548bec700 0 log_channel(cluster) log [INF] : 6.96 scrub ok
osd.7 2016-08-07 18:16:12.824917 7f537508a700 1 leveldb: Level-0 table #2166: started
osd.7 2016-08-07 18:16:12.860359 7f537508a700 1 leveldb: Level-0 table #2166: 1319242 bytes OK
osd.7 2016-08-07 18:16:12.863360 7f537508a700 1 leveldb: Delete type=0 #2163
osd.3 2016-08-07 18:16:24.716222 7f7544be4700 0 log_channel(cluster) log [INF] : 0.7b scrub starts
osd.3 2016-08-07 18:16:25.126539 7f7544be4700 0 log_channel(cluster) log [INF] : 0.7b scrub ok
osd.8 2016-08-07 18:16:25.714768 7f971eed7700 0 log_channel(cluster) log [INF] : 0.a2 deep-scrub starts
osd.8 2016-08-07 18:16:28.869516 7f971eed7700 0 log_channel(cluster) log [INF] : 0.a2 deep-scrub ok
osd.10 2016-08-07 18:17:36.727288 7f793212a700 0 log_channel(cluster) log [INF] : 6.18f deep-scrub starts
osd.10 2016-08-07 18:18:36.941021 7f793212a700 0 log_channel(cluster) log [INF] : 6.18f deep-scrub ok
osd.12 2016-08-07 18:18:37.381090 7fa766a94700 0 log_channel(cluster) log [INF] : 6.1ea scrub starts
osd.12 2016-08-07 18:18:39.128517 7fa76428f700 0 log_channel(cluster) log [INF] : 6.1ea scrub ok
osd.12 2016-08-07 18:18:40.381580 7fa76428f700 0 log_channel(cluster) log [INF] : 6.195 scrub starts
osd.12 2016-08-07 18:18:42.145238 7fa766a94700 0 log_channel(cluster) log [INF] : 6.195 scrub ok
osd.12 2016-08-07 18:18:42.382031 7fa765a92700 0 log_channel(cluster) log [INF] : 6.160 deep-scrub starts
osd.12 2016-08-07 18:19:47.432860 7fa76328d700 0 log_channel(cluster) log [INF] : 6.160 deep-scrub ok
osd.12 2016-08-07 18:19:51.389956 7fa76328d700 0 log_channel(cluster) log [INF] : 1.4d deep-scrub starts
osd.12 2016-08-07 18:19:55.143503 7fa765a92700 0 log_channel(cluster) log [INF] : 1.4d deep-scrub ok
osd.12 2016-08-07 18:20:09.392520 7fa765291700 0 log_channel(cluster) log [INF] : 6.c1 scrub starts
osd.12 2016-08-07 18:20:11.469277 7fa765291700 0 log_channel(cluster) log [INF] : 6.c1 scrub ok
osd.0 2016-08-07 18:20:28.030515 7f2005f06700 1 leveldb: Level-0 table #748689: started
osd.0 2016-08-07 18:20:28.097967 7f2005f06700 1 leveldb: Level-0 table #748689: 1343239 bytes OK
osd.0 2016-08-07 18:20:28.107135 7f2005f06700 1 leveldb: Delete type=0 #748686
osd.8 2016-08-07 18:21:33.759282 7f971eed7700 0 log_channel(cluster) log [INF] : 1.11 scrub starts
osd.8 2016-08-07 18:21:33.836549 7f971eed7700 0 log_channel(cluster) log [INF] : 1.11 scrub ok
osd.10 2016-08-07 18:21:34.760849 7f793392d700 0 log_channel(cluster) log [INF] : 6.6 scrub starts
osd.10 2016-08-07 18:21:36.227213 7f7936132700 0 log_channel(cluster) log [INF] : 6.6 scrub ok
osd.10 2016-08-07 18:21:40.761808 7f793392d700 0 log_channel(cluster) log [INF] : 0.6a scrub starts
osd.10 2016-08-07 18:21:41.026307 7f7936132700 0 log_channel(cluster) log [INF] : 0.6a scrub ok
osd.11 2016-08-07 18:21:42.167712 7fdd3d7eb700 0 log_channel(cluster) log [INF] : 0.ac scrub starts
osd.11 2016-08-07 18:21:42.410908 7fdd3d7eb700 0 log_channel(cluster) log [INF] : 0.ac scrub ok
osd.10 2016-08-07 18:21:44.762159 7f7936132700 0 log_channel(cluster) log [INF] : 1.74 scrub starts
osd.10 2016-08-07 18:21:45.112349 7f7936132700 0 log_channel(cluster) log [INF] : 1.74 scrub ok
osd.10 2016-08-07 18:21:50.763198 7f793212a700 0 log_channel(cluster) log [INF] : 0.72 scrub starts
osd.10 2016-08-07 18:21:50.920473 7f793492f700 0 log_channel(cluster) log [INF] : 0.72 scrub ok
osd.10 2016-08-07 18:22:28.767827 7f793492f700 0 log_channel(cluster) log [INF] : 1.d1 scrub starts
osd.10 2016-08-07 18:22:29.533751 7f793212a700 0 log_channel(cluster) log [INF] : 1.d1 scrub ok
osd.10 2016-08-07 18:22:30.768151 7f7936132700 0 log_channel(cluster) log [INF] : 0.c4 deep-scrub starts
osd.8 2016-08-07 18:22:31.767734 7f97206da700 0 log_channel(cluster) log [INF] : 6.22 scrub starts
osd.8 2016-08-07 18:22:33.313274 7f97206da700 0 log_channel(cluster) log [INF] : 6.22 scrub ok
osd.10 2016-08-07 18:22:41.843801 7f7936132700 0 log_channel(cluster) log [INF] : 0.c4 deep-scrub ok
osd.8 2016-08-07 18:22:58.771600 7f971eed7700 0 log_channel(cluster) log [INF] : 6.c5 scrub starts
osd.8 2016-08-07 18:23:02.083532 7f97216dc700 0 log_channel(cluster) log [INF] : 6.c5 scrub ok
osd.8 2016-08-07 18:23:03.772368 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.b0 scrub starts
osd.8 2016-08-07 18:23:04.173474 7f971f6d8700 0 log_channel(cluster) log [INF] : 1.b0 scrub ok
osd.8 2016-08-07 18:23:07.772834 7f971e6d6700 0 log_channel(cluster) log [INF] : 1.49 deep-scrub starts
osd.8 2016-08-07 18:23:10.844818 7f971e6d6700 0 log_channel(cluster) log [INF] : 1.49 deep-scrub ok
osd.8 2016-08-07 18:23:23.775001 7f971eed7700 0 log_channel(cluster) log [INF] : 1.6b scrub starts
osd.8 2016-08-07 18:23:24.023631 7f971eed7700 0 log_channel(cluster) log [INF] : 1.6b scrub ok
osd.8 2016-08-07 18:23:27.775605 7f971f6d8700 0 log_channel(cluster) log [INF] : 0.9c deep-scrub starts
osd.8 2016-08-07 18:23:36.506531 7f971f6d8700 0 log_channel(cluster) log [INF] : 0.9c deep-scrub ok
osd.11 2016-08-07 18:25:58.503988 7fdd28f79700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.102:6802/2031964 pipe(0x55789fc1e800 sd=73 :51794 s=2 pgs=478 cs=126 l=0 c=0x55789e863000).fault with nothing to send, going to standby
osd.2 2016-08-07 18:25:58.505631 7f6263aff700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.106:6801/2573 pipe(0x561436260000 sd=137 :6802 s=2 pgs=288 cs=126 l=0 c=0x561439278a00).fault with nothing to send, going to standby
osd.11 2016-08-07 18:25:58.513461 7fdd30254700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.104:6809/3528 pipe(0x5578a0dac000 sd=22 :47260 s=2 pgs=258 cs=169 l=0 c=0x5578a45fca00).fault with nothing to send, going to standby
osd.14 2016-08-07 18:25:58.513957 7fb254dcb700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c19e0df400 sd=45 :6809 s=2 pgs=289 cs=169 l=0 c=0x55c198811500).fault with nothing to send, going to standby
osd.6 2016-08-07 18:25:58.561878 7f799227a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6806/2031883 pipe(0x5639c2164800 sd=77 :50062 s=2 pgs=196 cs=175 l=0 c=0x5639bcd1bb00).fault with nothing to send, going to standby
osd.0 2016-08-07 18:25:58.563282 7f200d57b700 0 -- 192.168.13.102:6806/2031883 >> 192.168.13.103:6805/1002280 pipe(0x564b0ca4c800 sd=140 :6806 s=2 pgs=379 cs=175 l=0 c=0x564aefeb1c80).fault with nothing to send, going to standby
osd.14 2016-08-07 18:25:58.608660 7fb24e6c8700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.101:6802/8506 pipe(0x55c1a450e800 sd=78 :6809 s=2 pgs=302 cs=114 l=0 c=0x55c198810180).fault with nothing to send, going to standby
osd.1 2016-08-07 18:25:58.609976 7f56009fa700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6809/3528 pipe(0x55ab15255400 sd=227 :51698 s=2 pgs=257 cs=114 l=0 c=0x55ab19c4bd80).fault with nothing to send, going to standby
osd.6 2016-08-07 18:25:58.614794 7f799ab6a700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639c0b73400 sd=163 :6805 s=2 pgs=36 cs=3 l=0 c=0x5639d2d23a80).fault with nothing to send, going to standby
osd.8 2016-08-07 18:25:58.615908 7f97161a5700 0 -- 192.168.13.100:6801/17153 >> 192.168.13.103:6805/1002280 pipe(0x564b9b1d6000 sd=76 :44522 s=2 pgs=381 cs=3 l=0 c=0x564b9b17aa80).fault with nothing to send, going to standby
osd.8 2016-08-07 18:28:08.668440 7f97393f8700 1 leveldb: Level-0 table #846824: started
osd.8 2016-08-07 18:28:08.731154 7f97393f8700 1 leveldb: Level-0 table #846824: 1326026 bytes OK
osd.8 2016-08-07 18:28:08.739001 7f97393f8700 1 leveldb: Delete type=0 #846821
osd.8 2016-08-07 18:28:30.819476 7f9720edb700 0 log_channel(cluster) log [INF] : 6.198 scrub starts
osd.8 2016-08-07 18:28:32.394873 7f9720edb700 0 log_channel(cluster) log [INF] : 6.198 scrub ok
osd.10 2016-08-07 18:28:32.819797 7f793412e700 0 log_channel(cluster) log [INF] : 6.69 scrub starts
osd.8 2016-08-07 18:28:32.819816 7f9720edb700 0 log_channel(cluster) log [INF] : 6.11b scrub starts
osd.8 2016-08-07 18:28:35.096006 7f9720edb700 0 log_channel(cluster) log [INF] : 6.11b scrub ok
osd.10 2016-08-07 18:28:36.498539 7f7936933700 0 log_channel(cluster) log [INF] : 6.69 scrub ok
osd.8 2016-08-07 18:28:41.820914 7f97226de700 0 log_channel(cluster) log [INF] : 1.64 scrub starts
osd.8 2016-08-07 18:28:42.735759 7f971fed9700 0 log_channel(cluster) log [INF] : 1.64 scrub ok
osd.1 2016-08-07 18:29:01.138954 7f5618db9700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab0c48c000 sd=62 :6802 s=2 pgs=139 cs=95 l=0 c=0x55ab172f7780).fault with nothing to send, going to standby
osd.5 2016-08-07 18:29:01.143978 7fc304a15700 0 -- 192.168.13.104:6806/1002865 >> 192.168.13.101:6802/8506 pipe(0x55b595728000 sd=37 :40026 s=2 pgs=301 cs=95 l=0 c=0x55b59ee9a780).fault with nothing to send, going to standby
osd.2 2016-08-07 18:29:21.971058 7f625c961700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6809/3528 pipe(0x561436a3a800 sd=217 :6802 s=0 pgs=0 cs=0 l=0 c=0x561428997900).accept connect_seq 151 vs existing 151 state standby
osd.2 2016-08-07 18:29:21.971431 7f625c961700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6809/3528 pipe(0x561436a3a800 sd=217 :6802 s=0 pgs=0 cs=0 l=0 c=0x561428997900).accept connect_seq 152 vs existing 151 state standby
osd.1 2016-08-07 18:30:17.445857 7f55fd749700 1 leveldb: Level-0 table #129033: started
osd.1 2016-08-07 18:30:17.483529 7f55fd749700 1 leveldb: Level-0 table #129033: 1307610 bytes OK
osd.1 2016-08-07 18:30:17.484932 7f55fd749700 1 leveldb: Delete type=0 #129030
osd.1 2016-08-07 18:30:17.488879 7f55fd749700 1 leveldb: Compacting 4@0 + 4@1 files
osd.1 2016-08-07 18:30:17.598413 7f55fd749700 1 leveldb: Generated table #129034: 99575 keys, 2132808 bytes
osd.1 2016-08-07 18:30:17.720297 7f55fd749700 1 leveldb: Generated table #129035: 111847 keys, 2134201 bytes
osd.1 2016-08-07 18:30:17.843610 7f55fd749700 1 leveldb: Generated table #129036: 83340 keys, 2135232 bytes
osd.1 2016-08-07 18:30:17.952287 7f55fd749700 1 leveldb: Generated table #129037: 91542 keys, 1935349 bytes
osd.1 2016-08-07 18:30:17.952339 7f55fd749700 1 leveldb: Compacted 4@0 + 4@1 files => 8337590 bytes
osd.1 2016-08-07 18:30:17.952795 7f55fd749700 1 leveldb: compacted to: files[ 0 4 8 0 0 0 0 ]
osd.1 2016-08-07 18:30:17.953120 7f55fd749700 1 leveldb: Delete type=2 #129023
osd.1 2016-08-07 18:30:17.953914 7f55fd749700 1 leveldb: Delete type=2 #129029
osd.1 2016-08-07 18:30:17.954448 7f55fd749700 1 leveldb: Delete type=2 #129033
osd.1 2016-08-07 18:30:17.955038 7f55fd749700 1 leveldb: Delete type=2 #129024
osd.1 2016-08-07 18:30:17.955723 7f55fd749700 1 leveldb: Delete type=2 #129027
osd.1 2016-08-07 18:30:17.956274 7f55fd749700 1 leveldb: Delete type=2 #129022
osd.1 2016-08-07 18:30:17.956984 7f55fd749700 1 leveldb: Delete type=2 #129025
osd.1 2016-08-07 18:30:17.957508 7f55fd749700 1 leveldb: Delete type=2 #129031
osd.1 2016-08-07 18:30:33.707744 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=0 pgs=0 cs=0 l=0 c=0x55ab1ee64600).accept connect_seq 95 vs existing 95 state standby
osd.1 2016-08-07 18:30:33.707913 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=0 pgs=0 cs=0 l=0 c=0x55ab1ee64600).accept connect_seq 96 vs existing 95 state standby
osd.14 2016-08-07 18:31:15.196342 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c19ab8b480).accept connect_seq 169 vs existing 169 state standby
osd.14 2016-08-07 18:31:15.196783 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=0 pgs=0 cs=0 l=0 c=0x55c19ab8b480).accept connect_seq 170 vs existing 169 state standby
osd.12 2016-08-07 18:32:35.958425 7fa75467e700 1 leveldb: Level-0 table #856026: started
osd.12 2016-08-07 18:32:36.007445 7fa75467e700 1 leveldb: Level-0 table #856026: 1344812 bytes OK
osd.12 2016-08-07 18:32:36.018939 7fa75467e700 1 leveldb: Delete type=0 #856023
osd.0 2016-08-07 18:32:46.510590 7f2013d15700 0 log_channel(cluster) log [INF] : 6.15d scrub starts
osd.0 2016-08-07 18:32:50.719043 7f2013d15700 0 log_channel(cluster) log [INF] : 6.15d scrub ok
osd.12 2016-08-07 18:33:20.514523 7fa766293700 0 log_channel(cluster) log [INF] : 6.1f5 scrub starts
osd.12 2016-08-07 18:33:22.418910 7fa766293700 0 log_channel(cluster) log [INF] : 6.1f5 scrub ok
osd.0 2016-08-07 18:34:02.521334 7f201851e700 0 log_channel(cluster) log [INF] : 6.19f scrub starts
osd.0 2016-08-07 18:34:03.086709 7f201851e700 0 log_channel(cluster) log [INF] : 6.19f scrub ok
osd.8 2016-08-07 18:35:52.884927 7f97206da700 0 log_channel(cluster) log [INF] : 6.1d5 scrub starts
osd.8 2016-08-07 18:35:55.023186 7f97206da700 0 log_channel(cluster) log [INF] : 6.1d5 scrub ok
osd.3 2016-08-07 18:36:06.883958 7f7547bea700 0 log_channel(cluster) log [INF] : 6.61 scrub starts
osd.3 2016-08-07 18:36:10.882871 7f7547bea700 0 log_channel(cluster) log [INF] : 6.61 scrub ok
osd.2 2016-08-07 18:36:46.920394 7f6266529700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.103:6805/1002280 pipe(0x5614362dc800 sd=25 :40322 s=2 pgs=380 cs=228 l=0 c=0x561436f1dc00).fault with nothing to send, going to standby
osd.6 2016-08-07 18:36:46.920645 7f7995643700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.102:6802/2031964 pipe(0x5639bb0dc000 sd=79 :6805 s=2 pgs=477 cs=228 l=0 c=0x5639bd19f500).fault with nothing to send, going to standby
osd.3 2016-08-07 18:38:54.909297 7f75483eb700 0 log_channel(cluster) log [INF] : 6.ec scrub starts
osd.3 2016-08-07 18:38:59.552437 7f7545be6700 0 log_channel(cluster) log [INF] : 6.ec scrub ok
osd.9 2016-08-07 18:40:15.608683 7efcab574700 1 leveldb: Level-0 table #1145714: started
osd.9 2016-08-07 18:40:15.662852 7efcab574700 1 leveldb: Level-0 table #1145714: 1330478 bytes OK
osd.9 2016-08-07 18:40:15.688457 7efcab574700 1 leveldb: Delete type=0 #1145711
osd.14 2016-08-07 18:40:35.312687 7fb278a18700 1 leveldb: Level-0 table #675679: started
osd.14 2016-08-07 18:40:35.348444 7fb278a18700 1 leveldb: Level-0 table #675679: 1331634 bytes OK
osd.14 2016-08-07 18:40:35.349305 7fb278a18700 1 leveldb: Delete type=0 #675676
osd.10 2016-08-07 18:42:08.681984 7f7922b03700 1 leveldb: Level-0 table #697960: started
osd.10 2016-08-07 18:42:08.749477 7f7922b03700 1 leveldb: Level-0 table #697960: 1327032 bytes OK
osd.10 2016-08-07 18:42:08.760519 7f7922b03700 1 leveldb: Delete type=0 #697957
osd.4 2016-08-07 18:42:57.941590 7f37d521b700 1 leveldb: Level-0 table #809510: started
osd.4 2016-08-07 18:42:57.978108 7f37d521b700 1 leveldb: Level-0 table #809510: 1329135 bytes OK
osd.4 2016-08-07 18:42:57.978928 7f37d521b700 1 leveldb: Delete type=0 #809499
osd.0 2016-08-07 18:45:28.624007 7f2013d15700 0 log_channel(cluster) log [INF] : 6.194 scrub starts
osd.0 2016-08-07 18:45:33.382791 7f2013d15700 0 log_channel(cluster) log [INF] : 6.194 scrub ok
osd.5 2016-08-07 18:45:33.810007 7fc304a15700 0 -- 192.168.13.104:6806/1002865 >> 192.168.13.101:6802/8506 pipe(0x55b595728000 sd=37 :40118 s=2 pgs=303 cs=97 l=0 c=0x55b59ee9a780).fault with nothing to send, going to standby
osd.1 2016-08-07 18:45:33.812616 7f55fabce700 0 -- 192.168.13.101:6802/8506 >> 192.168.13.104:6806/1002865 pipe(0x55ab10494800 sd=147 :6802 s=2 pgs=140 cs=97 l=0 c=0x55ab172f7780).fault with nothing to send, going to standby
osd.11 2016-08-07 18:46:15.238980 7fdd30254700 0 -- 192.168.13.106:6801/2573 >> 192.168.13.104:6809/3528 pipe(0x5578a0dac000 sd=22 :47284 s=2 pgs=260 cs=171 l=0 c=0x5578a45fca00).fault with nothing to send, going to standby
osd.14 2016-08-07 18:46:15.239328 7fb251594700 0 -- 192.168.13.104:6809/3528 >> 192.168.13.106:6801/2573 pipe(0x55c198806800 sd=104 :6809 s=2 pgs=290 cs=171 l=0 c=0x55c198811500).fault with nothing to send, going to standby
osd.11 2016-08-07 18:46:52.801225 7fdd2ceb1700 1 leveldb: Level-0 table #2752: started
osd.11 2016-08-07 18:46:52.834546 7fdd2ceb1700 1 leveldb: Level-0 table #2752: 1317345 bytes OK
osd.11 2016-08-07 18:46:52.836676 7fdd2ceb1700 1 leveldb: Delete type=0 #2749
osd.11 2016-08-07 18:46:52.838322 7fdd2ceb1700 1 leveldb: Compacting 4@0 + 5@1 files
osd.11 2016-08-07 18:46:52.962256 7fdd2ceb1700 1 leveldb: Generated table #2753: 132257 keys, 2132791 bytes
osd.11 2016-08-07 18:46:53.072736 7fdd2ceb1700 1 leveldb: Generated table #2754: 116913 keys, 2134008 bytes
osd.11 2016-08-07 18:46:53.195075 7fdd2ceb1700 1 leveldb: Generated table #2755: 127096 keys, 2133069 bytes
osd.11 2016-08-07 18:46:53.300826 7fdd2ceb1700 1 leveldb: Generated table #2756: 106521 keys, 2133407 bytes
osd.11 2016-08-07 18:46:53.430719 7fdd2ceb1700 1 leveldb: Generated table #2757: 164841 keys, 1711532 bytes
osd.11 2016-08-07 18:46:53.430733 7fdd2ceb1700 1 leveldb: Compacted 4@0 + 5@1 files => 10244807 bytes
osd.11 2016-08-07 18:46:53.433286 7fdd2ceb1700 1 leveldb: compacted to: files[ 0 5 11 0 0 0 0 ]
osd.11 2016-08-07 18:46:53.433467 7fdd2ceb1700 1 leveldb: Delete type=2 #2750
osd.11 2016-08-07 18:46:53.433894 7fdd2ceb1700 1 leveldb: Delete type=2 #2740
osd.11 2016-08-07 18:46:53.434465 7fdd2ceb1700 1 leveldb: Delete type=2 #2741
osd.11 2016-08-07 18:46:53.434999 7fdd2ceb1700 1 leveldb: Delete type=2 #2742
osd.11 2016-08-07 18:46:53.435514 7fdd2ceb1700 1 leveldb: Delete type=2 #2743
osd.11 2016-08-07 18:46:53.435987 7fdd2ceb1700 1 leveldb: Delete type=2 #2744
osd.11 2016-08-07 18:46:53.436354 7fdd2ceb1700 1 leveldb: Delete type=2 #2752
osd.11 2016-08-07 18:46:53.436694 7fdd2ceb1700 1 leveldb: Delete type=2 #2746
osd.11 2016-08-07 18:46:53.437045 7fdd2ceb1700 1 leveldb: Delete type=2 #2748
osd.0 2016-08-07 18:47:39.642168 7f2015d19700 0 log_channel(cluster) log [INF] : 6.163 deep-scrub starts
osd.2 2016-08-07 18:48:04.108421 7f625e734700 0 -- 192.168.13.102:6802/2031964 >> 192.168.13.104:6812/1003178 pipe(0x5614362f6000 sd=67 :58020 s=2 pgs=110 cs=45 l=0 c=0x561429a40900).fault with nothing to send, going to standby
osd.4 2016-08-07 18:48:04.111052 7f37b382d700 0 -- 192.168.13.104:6812/1003178 >> 192.168.13.102:6802/2031964 pipe(0x5576ad5b4800 sd=212 :6812 s=2 pgs=476 cs=45 l=0 c=0x5576c4094480).fault with nothing to send, going to standby
osd.8 2016-08-07 18:48:08.990171 7f971eed7700 0 log_channel(cluster) log [INF] : 0.b1 deep-scrub starts
osd.8 2016-08-07 18:48:11.803539 7f971eed7700 0 log_channel(cluster) log [INF] : 0.b1 deep-scrub ok
osd.0 2016-08-07 18:49:14.530911 7f2015d19700 0 log_channel(cluster) log [INF] : 6.163 deep-scrub ok
osd.3 2016-08-07 18:50:36.834989 7f755dc5d700 1 leveldb: Level-0 table #461989: started
osd.3 2016-08-07 18:50:36.884892 7f755dc5d700 1 leveldb: Level-0 table #461989: 1352311 bytes OK
osd.3 2016-08-07 18:50:36.895575 7f755dc5d700 1 leveldb: Delete type=0 #461986
osd.10 2016-08-07 18:50:50.012726 7f7935931700 0 log_channel(cluster) log [INF] : 6.d4 scrub starts
osd.10 2016-08-07 18:50:51.418525 7f793312c700 0 log_channel(cluster) log [INF] : 6.d4 scrub ok
osd.10 2016-08-07 18:51:59.023257 7f7936132700 0 log_channel(cluster) log [INF] : 6.132 scrub starts
osd.10 2016-08-07 18:52:00.912383 7f7936132700 0 log_channel(cluster) log [INF] : 6.132 scrub ok
osd.3 2016-08-07 18:53:27.035518 7f7547bea700 0 log_channel(cluster) log [INF] : 6.f7 scrub starts
osd.3 2016-08-07 18:53:31.248614 7f75453e5700 0 log_channel(cluster) log [INF] : 6.f7 scrub ok
osd.6 2016-08-07 18:54:33.530533 7f7992e86700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639bda12800 sd=212 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2c00e80).accept connect_seq 3 vs existing 3 state standby
osd.6 2016-08-07 18:54:33.530810 7f7992e86700 0 -- 192.168.13.103:6805/1002280 >> 192.168.13.100:6801/17153 pipe(0x5639bda12800 sd=212 :6805 s=0 pgs=0 cs=0 l=0 c=0x5639b2c00e80).accept connect_seq 4 vs existing 3 state standby
osd.8 2016-08-07 18:54:35.045105 7f971e6d6700 0 log_channel(cluster) log [INF] : 6.9e scrub starts
osd.8 2016-08-07 18:54:38.216295 7f9720edb700 0 log_channel(cluster) log [INF] : 6.9e scrub ok
osd.0 2016-08-07 18:58:04.316762 7f2005f06700 1 leveldb: Level-0 table #748691: started
osd.0 2016-08-07 18:58:04.371443 7f2005f06700 1 leveldb: Level-0 table #748691: 1333000 bytes OK
osd.0 2016-08-07 18:58:04.377450 7f2005f06700 1 leveldb: Delete type=0 #748688
osd.3 2016-08-07 18:59:19.085423 7f7545be6700 0 log_channel(cluster) log [INF] : 6.1f scrub starts
osd.3 2016-08-07 18:59:23.200839 7f75483eb700 0 log_channel(cluster) log [INF] : 6.1f scrub ok
osd.10 2016-08-07 18:59:40.088618 7f793212a700 0 log_channel(cluster) log [INF] : 0.3b scrub starts
osd.10 2016-08-07 18:59:40.244603 7f793212a700 0 log_channel(cluster) log [INF] : 0.3b scrub ok
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
