I'm not very sure, it's possible that keyvaluestore will use spare
write which make big difference with ceph space statistic
On Mon, Sep 8, 2014 at 6:35 PM, Kenneth Waegeman
<Kenneth.Waegeman at ugent.be> wrote:
>
> Thank you very much !
>
> Is this problem then related to the weird sizes I see:
> pgmap v55220: 1216 pgs, 3 pools, 3406 GB data, 852 kobjects
> 418 GB used, 88130 GB / 88549 GB avail
>
> a calculation with df shows indeed that there is about 400GB used on disks,
> but the tests I ran should indeed have generated 3,5 TB, as also seen in
> rados df:
>
> pool name category KB objects clones
> degraded unfound rd rd KB wr wr KB
> cache - 59150443 15466 0
> 0 0 1388365 5686734850 3665984 4709621763
> ecdata - 3512807425 857620 0
> 0 0 1109938 312332288 857621 3512807426
>
> I thought it was related to the inconsistency?
> Or can this be a sparse objects thing? (But I don't seem to found anything
> in the docs about that)
>
> Thanks again!
>
> Kenneth
>
>
>
> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
> Date: Sun, 7 Sep 2014 20:34:39 +0800
>
> From: Haomai Wang <haomaiwang at gmail.com>
> Subject: Re: ceph cluster inconsistency keyvaluestore
> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
> Cc: ceph-users at lists.ceph.com
>
>
>> I have found the root cause. It's a bug.
>>
>> When chunky scrub happen, it will iterate the who pg's objects and
>> each iterator only a few objects will be scan.
>>
>> osd/PG.cc:3758
>> ret = get_pgbackend()-> objects_list_partial(
>> start,
>> cct->_conf->osd_scrub_chunk_min,
>> cct->_conf->osd_scrub_chunk_max,
>> 0,
>> &objects,
>> &candidate_end);
>>
>> candidate_end is the end of object set and it's used to indicate the
>> next scrub process's start position. But it will be truncated:
>>
>> osd/PG.cc:3777
>> while (!boundary_found && objects.size() > 1) {
>> hobject_t end = objects.back().get_boundary();
>> objects.pop_back();
>>
>> if (objects.back().get_filestore_key() !=
>> end.get_filestore_key()) {
>> candidate_end = end;
>> boundary_found = true;
>> }
>> }
>> end which only contain "hash" field as hobject_t will be assign to
>> candidate_end. So the next scrub process a hobject_t only contains
>> "hash" field will be passed in to get_pgbackend()->
>> objects_list_partial.
>>
>> It will cause incorrect results for KeyValueStore backend. Because it
>> will use strict key ordering for "collection_list_paritial" method. A
>> hobject_t only contains "hash" field will be:
>>
>> 1%e79s0_head!972F1B5D!!none!!!00000000000000000000!0!0
>>
>> and the actual object is
>> 1%e79s0_head!972F1B5D!!1!!!object-name!head
>>
>> In other word, a object only contain "hash" field can't used by to
>> search a absolute object has the same "hash" field.
>>
>> @sage The simply way is modify obj->key function which will change
>> storage format. Because it's a experiment backend I would like to
>> provide with a external format change program help users do it. Is it
>> OK?
>>
>>
>> On Wed, Sep 3, 2014 at 9:16 PM, Kenneth Waegeman
>> <Kenneth.Waegeman at ugent.be> wrote:
>>>
>>> I also can reproduce it on a new slightly different set up (also EC on KV
>>> and Cache) by running ceph pg scrub on a KV pg: this pg will then get the
>>> 'inconsistent' status
>>>
>>>
>>>
>>> ----- Message from Kenneth Waegeman <Kenneth.Waegeman at UGent.be> ---------
>>> Date: Mon, 01 Sep 2014 16:28:31 +0200
>>> From: Kenneth Waegeman <Kenneth.Waegeman at UGent.be>
>>> Subject: Re: ceph cluster inconsistency keyvaluestore
>>> To: Haomai Wang <haomaiwang at gmail.com>
>>> Cc: ceph-users at lists.ceph.com
>>>
>>>
>>>
>>>> Hi,
>>>>
>>>>
>>>> The cluster got installed with quattor, which uses ceph-deploy for
>>>> installation of daemons, writes the config file and installs the
>>>> crushmap.
>>>> I have 3 hosts, each 12 disks, having a large KV partition (3.6T) for
>>>> the
>>>> ECdata pool and a small cache partition (50G) for the cache
>>>>
>>>> I manually did this:
>>>>
>>>> ceph osd pool create cache 1024 1024
>>>> ceph osd pool set cache size 2
>>>> ceph osd pool set cache min_size 1
>>>> ceph osd erasure-code-profile set profile11 k=8 m=3
>>>> ruleset-failure-domain=osd
>>>> ceph osd pool create ecdata 128 128 erasure profile11
>>>> ceph osd tier add ecdata cache
>>>> ceph osd tier cache-mode cache writeback
>>>> ceph osd tier set-overlay ecdata cache
>>>> ceph osd pool set cache hit_set_type bloom
>>>> ceph osd pool set cache hit_set_count 1
>>>> ceph osd pool set cache hit_set_period 3600
>>>> ceph osd pool set cache target_max_bytes $((280*1024*1024*1024))
>>>>
>>>> (But the previous time I had the problem already without the cache part)
>>>>
>>>>
>>>>
>>>> Cluster live since 2014-08-29 15:34:16
>>>>
>>>> Config file on host ceph001:
>>>>
>>>> [global]
>>>> auth_client_required = cephx
>>>> auth_cluster_required = cephx
>>>> auth_service_required = cephx
>>>> cluster_network = 10.143.8.0/24
>>>> filestore_xattr_use_omap = 1
>>>> fsid = 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
>>>> mon_cluster_log_to_syslog = 1
>>>> mon_host = ceph001.cubone.os, ceph002.cubone.os, ceph003.cubone.os
>>>> mon_initial_members = ceph001, ceph002, ceph003
>>>> osd_crush_update_on_start = 0
>>>> osd_journal_size = 10240
>>>> osd_pool_default_min_size = 2
>>>> osd_pool_default_pg_num = 512
>>>> osd_pool_default_pgp_num = 512
>>>> osd_pool_default_size = 3
>>>> public_network = 10.141.8.0/24
>>>>
>>>> [osd.11]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.13]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.15]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.17]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.19]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.21]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.23]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.25]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.3]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.5]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.7]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>> [osd.9]
>>>> osd_objectstore = keyvaluestore-dev
>>>>
>>>>
>>>> OSDs:
>>>> # id weight type name up/down reweight
>>>> -12 140.6 root default-cache
>>>> -9 46.87 host ceph001-cache
>>>> 2 3.906 osd.2 up 1
>>>> 4 3.906 osd.4 up 1
>>>> 6 3.906 osd.6 up 1
>>>> 8 3.906 osd.8 up 1
>>>> 10 3.906 osd.10 up 1
>>>> 12 3.906 osd.12 up 1
>>>> 14 3.906 osd.14 up 1
>>>> 16 3.906 osd.16 up 1
>>>> 18 3.906 osd.18 up 1
>>>> 20 3.906 osd.20 up 1
>>>> 22 3.906 osd.22 up 1
>>>> 24 3.906 osd.24 up 1
>>>> -10 46.87 host ceph002-cache
>>>> 28 3.906 osd.28 up 1
>>>> 30 3.906 osd.30 up 1
>>>> 32 3.906 osd.32 up 1
>>>> 34 3.906 osd.34 up 1
>>>> 36 3.906 osd.36 up 1
>>>> 38 3.906 osd.38 up 1
>>>> 40 3.906 osd.40 up 1
>>>> 42 3.906 osd.42 up 1
>>>> 44 3.906 osd.44 up 1
>>>> 46 3.906 osd.46 up 1
>>>> 48 3.906 osd.48 up 1
>>>> 50 3.906 osd.50 up 1
>>>> -11 46.87 host ceph003-cache
>>>> 54 3.906 osd.54 up 1
>>>> 56 3.906 osd.56 up 1
>>>> 58 3.906 osd.58 up 1
>>>> 60 3.906 osd.60 up 1
>>>> 62 3.906 osd.62 up 1
>>>> 64 3.906 osd.64 up 1
>>>> 66 3.906 osd.66 up 1
>>>> 68 3.906 osd.68 up 1
>>>> 70 3.906 osd.70 up 1
>>>> 72 3.906 osd.72 up 1
>>>> 74 3.906 osd.74 up 1
>>>> 76 3.906 osd.76 up 1
>>>> -8 140.6 root default-ec
>>>> -5 46.87 host ceph001-ec
>>>> 3 3.906 osd.3 up 1
>>>> 5 3.906 osd.5 up 1
>>>> 7 3.906 osd.7 up 1
>>>> 9 3.906 osd.9 up 1
>>>> 11 3.906 osd.11 up 1
>>>> 13 3.906 osd.13 up 1
>>>> 15 3.906 osd.15 up 1
>>>> 17 3.906 osd.17 up 1
>>>> 19 3.906 osd.19 up 1
>>>> 21 3.906 osd.21 up 1
>>>> 23 3.906 osd.23 up 1
>>>> 25 3.906 osd.25 up 1
>>>> -6 46.87 host ceph002-ec
>>>> 29 3.906 osd.29 up 1
>>>> 31 3.906 osd.31 up 1
>>>> 33 3.906 osd.33 up 1
>>>> 35 3.906 osd.35 up 1
>>>> 37 3.906 osd.37 up 1
>>>> 39 3.906 osd.39 up 1
>>>> 41 3.906 osd.41 up 1
>>>> 43 3.906 osd.43 up 1
>>>> 45 3.906 osd.45 up 1
>>>> 47 3.906 osd.47 up 1
>>>> 49 3.906 osd.49 up 1
>>>> 51 3.906 osd.51 up 1
>>>> -7 46.87 host ceph003-ec
>>>> 55 3.906 osd.55 up 1
>>>> 57 3.906 osd.57 up 1
>>>> 59 3.906 osd.59 up 1
>>>> 61 3.906 osd.61 up 1
>>>> 63 3.906 osd.63 up 1
>>>> 65 3.906 osd.65 up 1
>>>> 67 3.906 osd.67 up 1
>>>> 69 3.906 osd.69 up 1
>>>> 71 3.906 osd.71 up 1
>>>> 73 3.906 osd.73 up 1
>>>> 75 3.906 osd.75 up 1
>>>> 77 3.906 osd.77 up 1
>>>> -4 23.44 root default-ssd
>>>> -1 7.812 host ceph001-ssd
>>>> 0 3.906 osd.0 up 1
>>>> 1 3.906 osd.1 up 1
>>>> -2 7.812 host ceph002-ssd
>>>> 26 3.906 osd.26 up 1
>>>> 27 3.906 osd.27 up 1
>>>> -3 7.812 host ceph003-ssd
>>>> 52 3.906 osd.52 up 1
>>>> 53 3.906 osd.53 up 1
>>>>
>>>> Cache OSDs are each 50G, the EC KV OSDS 3.6T, (ssds not used right now)
>>>>
>>>> Pools:
>>>> pool 0 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash
>>>> rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool
>>>> stripe_width 0
>>>> pool 1 'cache' replicated size 2 min_size 1 crush_ruleset 0 object_hash
>>>> rjenkins pg_num 1024 pgp_num 1024 last_change 174 flags
>>>> hashpspool,incomplete_clones tier_of 2 cache_mode writeback target_bytes
>>>> 300647710720 hit_set bloom{false_positive_probability: 0.05,
>>>> target_size: 0,
>>>> seed: 0} 3600s x1 stripe_width 0
>>>> pool 2 'ecdata' erasure size 11 min_size 8 crush_ruleset 2 object_hash
>>>> rjenkins pg_num 128 pgp_num 128 last_change 170 lfor 170 flags
>>>> hashpspool
>>>> tiers 1 read_tier 1 write_tier 1 stripe_width 4096
>>>>
>>>>
>>>> Crushmap:
>>>> # begin crush map
>>>> tunable choose_local_fallback_tries 0
>>>> tunable choose_local_tries 0
>>>> tunable choose_total_tries 50
>>>> tunable chooseleaf_descend_once 1
>>>>
>>>> # devices
>>>> device 0 osd.0
>>>> device 1 osd.1
>>>> device 2 osd.2
>>>> device 3 osd.3
>>>> device 4 osd.4
>>>> device 5 osd.5
>>>> device 6 osd.6
>>>> device 7 osd.7
>>>> device 8 osd.8
>>>> device 9 osd.9
>>>> device 10 osd.10
>>>> device 11 osd.11
>>>> device 12 osd.12
>>>> device 13 osd.13
>>>> device 14 osd.14
>>>> device 15 osd.15
>>>> device 16 osd.16
>>>> device 17 osd.17
>>>> device 18 osd.18
>>>> device 19 osd.19
>>>> device 20 osd.20
>>>> device 21 osd.21
>>>> device 22 osd.22
>>>> device 23 osd.23
>>>> device 24 osd.24
>>>> device 25 osd.25
>>>> device 26 osd.26
>>>> device 27 osd.27
>>>> device 28 osd.28
>>>> device 29 osd.29
>>>> device 30 osd.30
>>>> device 31 osd.31
>>>> device 32 osd.32
>>>> device 33 osd.33
>>>> device 34 osd.34
>>>> device 35 osd.35
>>>> device 36 osd.36
>>>> device 37 osd.37
>>>> device 38 osd.38
>>>> device 39 osd.39
>>>> device 40 osd.40
>>>> device 41 osd.41
>>>> device 42 osd.42
>>>> device 43 osd.43
>>>> device 44 osd.44
>>>> device 45 osd.45
>>>> device 46 osd.46
>>>> device 47 osd.47
>>>> device 48 osd.48
>>>> device 49 osd.49
>>>> device 50 osd.50
>>>> device 51 osd.51
>>>> device 52 osd.52
>>>> device 53 osd.53
>>>> device 54 osd.54
>>>> device 55 osd.55
>>>> device 56 osd.56
>>>> device 57 osd.57
>>>> device 58 osd.58
>>>> device 59 osd.59
>>>> device 60 osd.60
>>>> device 61 osd.61
>>>> device 62 osd.62
>>>> device 63 osd.63
>>>> device 64 osd.64
>>>> device 65 osd.65
>>>> device 66 osd.66
>>>> device 67 osd.67
>>>> device 68 osd.68
>>>> device 69 osd.69
>>>> device 70 osd.70
>>>> device 71 osd.71
>>>> device 72 osd.72
>>>> device 73 osd.73
>>>> device 74 osd.74
>>>> device 75 osd.75
>>>> device 76 osd.76
>>>> device 77 osd.77
>>>>
>>>> # types
>>>> type 0 osd
>>>> type 1 host
>>>> type 2 root
>>>>
>>>> # buckets
>>>> host ceph001-ssd {
>>>> id -1 # do not change unnecessarily
>>>> # weight 7.812
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.0 weight 3.906
>>>> item osd.1 weight 3.906
>>>> }
>>>> host ceph002-ssd {
>>>> id -2 # do not change unnecessarily
>>>> # weight 7.812
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.26 weight 3.906
>>>> item osd.27 weight 3.906
>>>> }
>>>> host ceph003-ssd {
>>>> id -3 # do not change unnecessarily
>>>> # weight 7.812
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.52 weight 3.906
>>>> item osd.53 weight 3.906
>>>> }
>>>> root default-ssd {
>>>> id -4 # do not change unnecessarily
>>>> # weight 23.436
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item ceph001-ssd weight 7.812
>>>> item ceph002-ssd weight 7.812
>>>> item ceph003-ssd weight 7.812
>>>> }
>>>> host ceph001-ec {
>>>> id -5 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.3 weight 3.906
>>>> item osd.5 weight 3.906
>>>> item osd.7 weight 3.906
>>>> item osd.9 weight 3.906
>>>> item osd.11 weight 3.906
>>>> item osd.13 weight 3.906
>>>> item osd.15 weight 3.906
>>>> item osd.17 weight 3.906
>>>> item osd.19 weight 3.906
>>>> item osd.21 weight 3.906
>>>> item osd.23 weight 3.906
>>>> item osd.25 weight 3.906
>>>> }
>>>> host ceph002-ec {
>>>> id -6 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.29 weight 3.906
>>>> item osd.31 weight 3.906
>>>> item osd.33 weight 3.906
>>>> item osd.35 weight 3.906
>>>> item osd.37 weight 3.906
>>>> item osd.39 weight 3.906
>>>> item osd.41 weight 3.906
>>>> item osd.43 weight 3.906
>>>> item osd.45 weight 3.906
>>>> item osd.47 weight 3.906
>>>> item osd.49 weight 3.906
>>>> item osd.51 weight 3.906
>>>> }
>>>> host ceph003-ec {
>>>> id -7 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.55 weight 3.906
>>>> item osd.57 weight 3.906
>>>> item osd.59 weight 3.906
>>>> item osd.61 weight 3.906
>>>> item osd.63 weight 3.906
>>>> item osd.65 weight 3.906
>>>> item osd.67 weight 3.906
>>>> item osd.69 weight 3.906
>>>> item osd.71 weight 3.906
>>>> item osd.73 weight 3.906
>>>> item osd.75 weight 3.906
>>>> item osd.77 weight 3.906
>>>> }
>>>> root default-ec {
>>>> id -8 # do not change unnecessarily
>>>> # weight 140.616
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item ceph001-ec weight 46.872
>>>> item ceph002-ec weight 46.872
>>>> item ceph003-ec weight 46.872
>>>> }
>>>> host ceph001-cache {
>>>> id -9 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.2 weight 3.906
>>>> item osd.4 weight 3.906
>>>> item osd.6 weight 3.906
>>>> item osd.8 weight 3.906
>>>> item osd.10 weight 3.906
>>>> item osd.12 weight 3.906
>>>> item osd.14 weight 3.906
>>>> item osd.16 weight 3.906
>>>> item osd.18 weight 3.906
>>>> item osd.20 weight 3.906
>>>> item osd.22 weight 3.906
>>>> item osd.24 weight 3.906
>>>> }
>>>> host ceph002-cache {
>>>> id -10 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.28 weight 3.906
>>>> item osd.30 weight 3.906
>>>> item osd.32 weight 3.906
>>>> item osd.34 weight 3.906
>>>> item osd.36 weight 3.906
>>>> item osd.38 weight 3.906
>>>> item osd.40 weight 3.906
>>>> item osd.42 weight 3.906
>>>> item osd.44 weight 3.906
>>>> item osd.46 weight 3.906
>>>> item osd.48 weight 3.906
>>>> item osd.50 weight 3.906
>>>> }
>>>> host ceph003-cache {
>>>> id -11 # do not change unnecessarily
>>>> # weight 46.872
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item osd.54 weight 3.906
>>>> item osd.56 weight 3.906
>>>> item osd.58 weight 3.906
>>>> item osd.60 weight 3.906
>>>> item osd.62 weight 3.906
>>>> item osd.64 weight 3.906
>>>> item osd.66 weight 3.906
>>>> item osd.68 weight 3.906
>>>> item osd.70 weight 3.906
>>>> item osd.72 weight 3.906
>>>> item osd.74 weight 3.906
>>>> item osd.76 weight 3.906
>>>> }
>>>> root default-cache {
>>>> id -12 # do not change unnecessarily
>>>> # weight 140.616
>>>> alg straw
>>>> hash 0 # rjenkins1
>>>> item ceph001-cache weight 46.872
>>>> item ceph002-cache weight 46.872
>>>> item ceph003-cache weight 46.872
>>>> }
>>>>
>>>> # rules
>>>> rule cache {
>>>> ruleset 0
>>>> type replicated
>>>> min_size 1
>>>> max_size 10
>>>> step take default-cache
>>>> step chooseleaf firstn 0 type host
>>>> step emit
>>>> }
>>>> rule metadata {
>>>> ruleset 1
>>>> type replicated
>>>> min_size 1
>>>> max_size 10
>>>> step take default-ssd
>>>> step chooseleaf firstn 0 type host
>>>> step emit
>>>> }
>>>> rule ecdata {
>>>> ruleset 2
>>>> type erasure
>>>> min_size 3
>>>> max_size 20
>>>> step set_chooseleaf_tries 5
>>>> step take default-ec
>>>> step choose indep 0 type osd
>>>> step emit
>>>> }
>>>>
>>>> # end crush map
>>>>
>>>> The benchmarks I then did:
>>>>
>>>> ./benchrw 50000
>>>>
>>>> benchrw:
>>>> /usr/bin/rados -p ecdata bench $1 write --no-cleanup
>>>> /usr/bin/rados -p ecdata bench $1 seq
>>>> /usr/bin/rados -p ecdata bench $1 seq &
>>>> /usr/bin/rados -p ecdata bench $1 write --no-cleanup
>>>>
>>>>
>>>> Srubbing errors started soon after that: 2014-08-31 10:59:14
>>>>
>>>>
>>>> Please let me know if you need more information, and thanks !
>>>>
>>>> Kenneth
>>>>
>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
>>>> Date: Mon, 1 Sep 2014 21:30:16 +0800
>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>> Subject: Re: ceph cluster inconsistency keyvaluestore
>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>> Cc: ceph-users at lists.ceph.com
>>>>
>>>>
>>>>> Hmm, could you please list your instructions including cluster existing
>>>>> time and all relevant ops? I want to reproduce it.
>>>>>
>>>>>
>>>>> On Mon, Sep 1, 2014 at 4:45 PM, Kenneth Waegeman
>>>>> <Kenneth.Waegeman at ugent.be>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I reinstalled the cluster with 0.84, and tried again running rados
>>>>>> bench
>>>>>> on a EC coded pool on keyvaluestore.
>>>>>> Nothing crashed this time, but when I check the status:
>>>>>>
>>>>>> health HEALTH_ERR 128 pgs inconsistent; 128 scrub errors; too few
>>>>>> pgs
>>>>>> per osd (15 < min 20)
>>>>>> monmap e1: 3 mons at {ceph001=10.141.8.180:6789/0,
>>>>>> ceph002=10.141.8.181:6789/0,ceph003=10.141.8.182:6789/0}, election
>>>>>> epoch
>>>>>> 8, quorum 0,1,2 ceph001,ceph002,ceph003
>>>>>> osdmap e174: 78 osds: 78 up, 78 in
>>>>>> pgmap v147680: 1216 pgs, 3 pools, 14758 GB data, 3690 kobjects
>>>>>> 1753 GB used, 129 TB / 131 TB avail
>>>>>> 1088 active+clean
>>>>>> 128 active+clean+inconsistent
>>>>>>
>>>>>> the 128 inconsistent pgs are ALL the pgs of the EC KV store ( the
>>>>>> others
>>>>>> are on Filestore)
>>>>>>
>>>>>> The only thing I can see in the logs is that after the rados tests, it
>>>>>> start scrubbing, and for each KV pg I get something like this:
>>>>>>
>>>>>> 2014-08-31 11:14:09.050747 osd.11 10.141.8.180:6833/61098 4 : [ERR]
>>>>>> 2.3s0
>>>>>> scrub stat mismatch, got 28164/29291 objects, 0/0 clones, 28164/29291
>>>>>> dirty, 0/0 omap, 0/0 hit_set_archive, 0/0 whiteouts,
>>>>>> 118128377856/122855358464 bytes.
>>>>>>
>>>>>> What could here be the problem?
>>>>>> Thanks again!!
>>>>>>
>>>>>> Kenneth
>>>>>>
>>>>>>
>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
>>>>>> Date: Tue, 26 Aug 2014 17:11:43 +0800
>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>>>> Cc: ceph-users at lists.ceph.com
>>>>>>
>>>>>>
>>>>>> Hmm, it looks like you hit this
>>>>>> bug(http://tracker.ceph.com/issues/9223).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Sorry for the late message, I forget that this fix is merged into
>>>>>>> 0.84.
>>>>>>>
>>>>>>> Thanks for your patient :-)
>>>>>>>
>>>>>>> On Tue, Aug 26, 2014 at 4:39 PM, Kenneth Waegeman
>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> In the meantime I already tried with upgrading the cluster to 0.84,
>>>>>>>> to
>>>>>>>> see
>>>>>>>> if that made a difference, and it seems it does.
>>>>>>>> I can't reproduce the crashing osds by doing a 'rados -p ecdata ls'
>>>>>>>> anymore.
>>>>>>>>
>>>>>>>> But now the cluster detect it is inconsistent:
>>>>>>>>
>>>>>>>> cluster 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
>>>>>>>> health HEALTH_ERR 40 pgs inconsistent; 40 scrub errors; too
>>>>>>>> few
>>>>>>>> pgs
>>>>>>>> per osd (4 < min 20); mon.ceph002 low disk space
>>>>>>>> monmap e3: 3 mons at
>>>>>>>> {ceph001=10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,
>>>>>>>> ceph003=10.141.8.182:6789/0},
>>>>>>>> election epoch 30, quorum 0,1,2 ceph001,ceph002,ceph003
>>>>>>>> mdsmap e78951: 1/1/1 up {0=ceph003.cubone.os=up:active}, 3
>>>>>>>> up:standby
>>>>>>>> osdmap e145384: 78 osds: 78 up, 78 in
>>>>>>>> pgmap v247095: 320 pgs, 4 pools, 15366 GB data, 3841 kobjects
>>>>>>>> 1502 GB used, 129 TB / 131 TB avail
>>>>>>>> 279 active+clean
>>>>>>>> 40 active+clean+inconsistent
>>>>>>>> 1 active+clean+scrubbing+deep
>>>>>>>>
>>>>>>>>
>>>>>>>> I tried to do ceph pg repair for all the inconsistent pgs:
>>>>>>>>
>>>>>>>> cluster 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
>>>>>>>> health HEALTH_ERR 40 pgs inconsistent; 1 pgs repair; 40 scrub
>>>>>>>> errors;
>>>>>>>> too few pgs per osd (4 < min 20); mon.ceph002 low disk space
>>>>>>>> monmap e3: 3 mons at
>>>>>>>> {ceph001=10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,
>>>>>>>> ceph003=10.141.8.182:6789/0},
>>>>>>>> election epoch 30, quorum 0,1,2 ceph001,ceph002,ceph003
>>>>>>>> mdsmap e79486: 1/1/1 up {0=ceph003.cubone.os=up:active}, 3
>>>>>>>> up:standby
>>>>>>>> osdmap e146452: 78 osds: 78 up, 78 in
>>>>>>>> pgmap v248520: 320 pgs, 4 pools, 15366 GB data, 3841 kobjects
>>>>>>>> 1503 GB used, 129 TB / 131 TB avail
>>>>>>>> 279 active+clean
>>>>>>>> 39 active+clean+inconsistent
>>>>>>>> 1 active+clean+scrubbing+deep
>>>>>>>> 1
>>>>>>>> active+clean+scrubbing+deep+inconsistent+repair
>>>>>>>>
>>>>>>>> I let it recovering through the night, but this morning the mons
>>>>>>>> were
>>>>>>>> all
>>>>>>>> gone, nothing to see in the log files.. The osds were all still up!
>>>>>>>>
>>>>>>>> cluster 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
>>>>>>>> health HEALTH_ERR 36 pgs inconsistent; 1 pgs repair; 36 scrub
>>>>>>>> errors;
>>>>>>>> too few pgs per osd (4 < min 20)
>>>>>>>> monmap e7: 3 mons at
>>>>>>>> {ceph001=10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,
>>>>>>>> ceph003=10.141.8.182:6789/0},
>>>>>>>> election epoch 44, quorum 0,1,2 ceph001,ceph002,ceph003
>>>>>>>> mdsmap e109481: 1/1/1 up {0=ceph003.cubone.os=up:active}, 3
>>>>>>>> up:standby
>>>>>>>> osdmap e203410: 78 osds: 78 up, 78 in
>>>>>>>> pgmap v331747: 320 pgs, 4 pools, 15251 GB data, 3812 kobjects
>>>>>>>> 1547 GB used, 129 TB / 131 TB avail
>>>>>>>> 1 active+clean+scrubbing+deep+inconsistent+repair
>>>>>>>> 284 active+clean
>>>>>>>> 35 active+clean+inconsistent
>>>>>>>>
>>>>>>>> I restarted the monitors now, I will let you know when I see
>>>>>>>> something
>>>>>>>> more..
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
>>>>>>>> Date: Sun, 24 Aug 2014 12:51:41 +0800
>>>>>>>>
>>>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>,
>>>>>>>> ceph-users at lists.ceph.com
>>>>>>>>
>>>>>>>>
>>>>>>>> It's really strange! I write a test program according the key
>>>>>>>> ordering
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> you provided and parse the corresponding value. It's true!
>>>>>>>>>
>>>>>>>>> I have no idea now. If free, could you add this debug code to
>>>>>>>>> "src/os/GenericObjectMap.cc" and insert *before* "assert(start <=
>>>>>>>>> header.oid);":
>>>>>>>>>
>>>>>>>>> dout(0) << "start: " << start << "header.oid: " << header.oid <<
>>>>>>>>> dendl;
>>>>>>>>>
>>>>>>>>> Then you need to recompile ceph-osd and run it again. The output
>>>>>>>>> log
>>>>>>>>> can help it!
>>>>>>>>>
>>>>>>>>> On Tue, Aug 19, 2014 at 10:19 PM, Haomai Wang
>>>>>>>>> <haomaiwang at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I feel a little embarrassed, 1024 rows still true for me.
>>>>>>>>>>
>>>>>>>>>> I was wondering if you could give your all keys via
>>>>>>>>>> ""ceph-kvstore-tool /var/lib/ceph/osd/ceph-67/current/ list
>>>>>>>>>> _GHOBJTOSEQ_ > keys.log?.
>>>>>>>>>>
>>>>>>>>>> thanks!
>>>>>>>>>>
>>>>>>>>>> On Tue, Aug 19, 2014 at 4:58 PM, Kenneth Waegeman
>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
>>>>>>>>>>> Date: Tue, 19 Aug 2014 12:28:27 +0800
>>>>>>>>>>>
>>>>>>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>>>>>>>>> Cc: Sage Weil <sweil at redhat.com>, ceph-users at lists.ceph.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 18, 2014 at 7:32 PM, Kenneth Waegeman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com> ---------
>>>>>>>>>>>>> Date: Mon, 18 Aug 2014 18:34:11 +0800
>>>>>>>>>>>>>
>>>>>>>>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>>>>>>>>>>> Cc: Sage Weil <sweil at redhat.com>, ceph-users at lists.ceph.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Aug 18, 2014 at 5:38 PM, Kenneth Waegeman
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I tried this after restarting the osd, but I guess that was
>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> aim
>>>>>>>>>>>>>>> (
>>>>>>>>>>>>>>> # ceph-kvstore-tool /var/lib/ceph/osd/ceph-67/current/ list
>>>>>>>>>>>>>>> _GHOBJTOSEQ_|
>>>>>>>>>>>>>>> grep 6adb1100 -A 100
>>>>>>>>>>>>>>> IO error: lock /var/lib/ceph/osd/ceph-67/current//LOCK:
>>>>>>>>>>>>>>> Resource
>>>>>>>>>>>>>>> temporarily
>>>>>>>>>>>>>>> unavailable
>>>>>>>>>>>>>>> tools/ceph_kvstore_tool.cc: In function
>>>>>>>>>>>>>>> 'StoreTool::StoreTool(const
>>>>>>>>>>>>>>> string&)' thread 7f8fecf7d780 time 2014-08-18 11:12:29.551780
>>>>>>>>>>>>>>> tools/ceph_kvstore_tool.cc: 38: FAILED
>>>>>>>>>>>>>>> assert(!db_ptr->open(std::cerr))
>>>>>>>>>>>>>>> ..
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I run it after bringing the osd down, it takes a while,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> has
>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>> output.. (When running it without the grep, I'm getting a
>>>>>>>>>>>>>>> huge
>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oh, sorry for it! I made a mistake, the hash value(6adb1100)
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> reversed into leveldb.
>>>>>>>>>>>>>> So grep "benchmark_data_ceph001.cubone.os_5560_object789734"
>>>>>>>>>>>>>> should
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>> help it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> this gives:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root at ceph003 ~]# ceph-kvstore-tool /var/lib/ceph/osd/ceph-67/
>>>>>>>>>>>>> current/
>>>>>>>>>>>>> list
>>>>>>>>>>>>> _GHOBJTOSEQ_ | grep 5560_object789734 -A 100
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011BDA6!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object789734!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011C027!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1330170!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011C6FD!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object227366!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011CB03!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1363631!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011CDF0!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1573957!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011D02C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1019282!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011E2B5!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1283563!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011E511!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object273736!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011E547!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1170628!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011EAAB!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object256335!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011F446!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1484196!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0011FC59!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object884178!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001203F3!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object853746!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001208E3!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object36633!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00120B37!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1235337!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001210B6!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1661351!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001210CB!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object238126!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012184C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object339943!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00121916!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1047094!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001219C1!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object520642!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001222BB!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object639565!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001223AA!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object231080!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012243C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object858050!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012289C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object241796!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00122D28!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object7462!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00122DFE!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object243798!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00122EFC!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object109512!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001232D7!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object653973!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001234A3!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1378169!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00123714!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object512925!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001237D9!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object23289!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00123854!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1108852!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00123971!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object704026!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00123F75!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object250441!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00124083!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object706178!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001240FA!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object316952!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012447D!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object538734!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001244D9!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object789215!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001247CD!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object265993!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00124897!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object610597!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00124BE4!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object691723!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00124C9B!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1306135!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00124E1D!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object520580!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012534C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object659767!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00125A81!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object184060!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00125E77!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1292867!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00126562!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1201410!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00126B34!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1657326!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00127383!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1269787!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00127396!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object500115!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001277F8!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object394932!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001279DD!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object252963!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00127B40!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object936811!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00127BAC!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1481773!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012894E!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object999885!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00128D05!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object943667!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012908A!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object212990!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00129519!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object437596!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00129716!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1585330!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00129798!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object603505!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001299C9!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object808800!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00129B7A!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object23193!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00129B9A!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1158397!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012A932!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object542450!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012B77A!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object195480!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012BE8C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object312911!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012BF74!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1563783!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012C65C!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1123980!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012C6FE!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_3411_object913!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012CCAD!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object400863!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012CDBB!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object789667!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012D14B!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1020723!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012D95B!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object106293!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012E3C8!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1355526!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012E5B3!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1491348!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012F2BB!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object338872!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012F374!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1337264!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012FBE5!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1512395!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012FCE3!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_8961_object298610!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0012FEB6!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object120824!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001301CA!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object816326!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00130263!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object777163!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00130529!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1413173!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001317D9!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object809510!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0013204F!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object471416!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00132400!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object695087!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00132A19!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object591945!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00132BF8!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object302000!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00132F5B!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1645443!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00133B8B!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object761911!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!0013433E!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object1467727!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00134446!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object791960!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00134678!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object677078!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00134A96!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object254923!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!001355D0!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_31461_object321528!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00135690!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4919_object36935!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00135B62!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object1228272!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00135C72!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_4812_object2180!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00135DEE!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object425705!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00136366!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object141569!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _GHOBJTOSEQ_:3%e0s0_head!00136371!!3!!benchmark_data_
>>>>>>>>>>>>> ceph001%ecubone%eos_5560_object564213!head
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> 100 rows seemed true for me. I found the min list objects is
>>>>>>>>>>>> 1024.
>>>>>>>>>>>> Please could you run
>>>>>>>>>>>> "ceph-kvstore-tool /var/lib/ceph/osd/ceph-67/current/ list
>>>>>>>>>>>> _GHOBJTOSEQ_| grep 6adb1100 -A 1024"
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I got the output in attachment
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Or should I run this immediately after the osd is crashed,
>>>>>>>>>>>>>>> (because
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> maybe
>>>>>>>>>>>>>>> rebalanced? I did already restarted the cluster)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't know if it is related, but before I could all do
>>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>> had
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>> something else: A monitor did run out if disk space, using
>>>>>>>>>>>>>>> 8GB
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> his
>>>>>>>>>>>>>>> store.db folder (lot of sst files). Other monitors are also
>>>>>>>>>>>>>>> near
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> level.
>>>>>>>>>>>>>>> Never had that problem on previous setups before. I recreated
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> monitor
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> now it uses 3.8GB.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It exists some duplicate data which needed to be compacted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another idea, maybe you can make KeyValueStore's stripe size
>>>>>>>>>>>>>> align
>>>>>>>>>>>>>> with EC stripe size.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can I do that? Is there some documentation about that?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ceph --show-config | grep keyvaluestore
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> debug_keyvaluestore = 0/0
>>>>>>>>>>>> keyvaluestore_queue_max_ops = 50
>>>>>>>>>>>> keyvaluestore_queue_max_bytes = 104857600
>>>>>>>>>>>> keyvaluestore_debug_check_backend = false
>>>>>>>>>>>> keyvaluestore_op_threads = 2
>>>>>>>>>>>> keyvaluestore_op_thread_timeout = 60
>>>>>>>>>>>> keyvaluestore_op_thread_suicide_timeout = 180
>>>>>>>>>>>> keyvaluestore_default_strip_size = 4096
>>>>>>>>>>>> keyvaluestore_max_expected_write_size = 16777216
>>>>>>>>>>>> keyvaluestore_header_cache_size = 4096
>>>>>>>>>>>> keyvaluestore_backend = leveldb
>>>>>>>>>>>>
>>>>>>>>>>>> keyvaluestore_default_strip_size is the wanted
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I haven't think deeply and maybe I will try it later.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Kenneth
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- Message from Sage Weil <sweil at redhat.com> ---------
>>>>>>>>>>>>>>> Date: Fri, 15 Aug 2014 06:10:34 -0700 (PDT)
>>>>>>>>>>>>>>> From: Sage Weil <sweil at redhat.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>>>>>>>>> To: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>> Cc: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>,
>>>>>>>>>>>>>>> ceph-users at lists.ceph.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, 15 Aug 2014, Haomai Wang wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Kenneth,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I don't find valuable info in your logs, it lack of the
>>>>>>>>>>>>>>>>> necessary
>>>>>>>>>>>>>>>>> debug output when accessing crash code.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But I scan the encode/decode implementation in
>>>>>>>>>>>>>>>>> GenericObjectMap
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> find something bad.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For example, two oid has same hash and their name is:
>>>>>>>>>>>>>>>>> A: "rb.data.123"
>>>>>>>>>>>>>>>>> B: "rb-123"
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In ghobject_t compare level, A < B. But GenericObjectMap
>>>>>>>>>>>>>>>>> encode
>>>>>>>>>>>>>>>>> "."
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> "%e", so the key in DB is:
>>>>>>>>>>>>>>>>> A: _GHOBJTOSEQ_:blah!51615000!!none!!rb%edata%e123!head
>>>>>>>>>>>>>>>>> B: _GHOBJTOSEQ_:blah!51615000!!none!!rb-123!head
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> A > B
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And it seemed that the escape function is useless and
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>> disabled.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not sure whether Kenneth's problem is touching this
>>>>>>>>>>>>>>>>> bug.
>>>>>>>>>>>>>>>>> Because
>>>>>>>>>>>>>>>>> this scene only occur when the object set is very large and
>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> two object has same hash value.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Kenneth, could you have time to run "ceph-kv-store
>>>>>>>>>>>>>>>>> [path-to-osd]
>>>>>>>>>>>>>>>>> list
>>>>>>>>>>>>>>>>> _GHOBJTOSEQ_| grep 6adb1100 -A 100". ceph-kv-store is a
>>>>>>>>>>>>>>>>> debug
>>>>>>>>>>>>>>>>> tool
>>>>>>>>>>>>>>>>> which can be compiled from source. You can clone ceph repo
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> run
>>>>>>>>>>>>>>>>> "./authongen.sh; ./configure; cd src; make
>>>>>>>>>>>>>>>>> ceph-kvstore-tool".
>>>>>>>>>>>>>>>>> "path-to-osd" should be "/var/lib/ceph/osd-[id]/current/".
>>>>>>>>>>>>>>>>> "6adb1100"
>>>>>>>>>>>>>>>>> is from your verbose log and the next 100 rows should know
>>>>>>>>>>>>>>>>> necessary
>>>>>>>>>>>>>>>>> infos.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You can also get ceph-kvstore-tool from the 'ceph-tests'
>>>>>>>>>>>>>>>> package.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi sage, do you think we need to provided with upgrade
>>>>>>>>>>>>>>>> function
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hmm, we might. This only affects the key/value encoding
>>>>>>>>>>>>>>>> right?
>>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>> FileStore is using its own function to map these to file
>>>>>>>>>>>>>>>> names?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you open a ticket in the tracker for this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>> sage
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Aug 14, 2014 at 7:36 PM, Kenneth Waegeman
>>>>>>>>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>> ---------
>>>>>>>>>>>>>>>>>> Date: Thu, 14 Aug 2014 19:11:55 +0800
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>>>>>>>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Could you add config "debug_keyvaluestore = 20/20" to the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> crashed
>>>>>>>>>>>>>>>>>>> osd
>>>>>>>>>>>>>>>>>>> and replay the command causing crash?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I would like to get more debug infos! Thanks.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I included the log in attachment!
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Thu, Aug 14, 2014 at 4:41 PM, Kenneth Waegeman
>>>>>>>>>>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have:
>>>>>>>>>>>>>>>>>>>> osd_objectstore = keyvaluestore-dev
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> in the global section of my ceph.conf
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [root at ceph002 ~]# ceph osd erasure-code-profile get
>>>>>>>>>>>>>>>>>>>> profile11
>>>>>>>>>>>>>>>>>>>> directory=/usr/lib64/ceph/erasure-code
>>>>>>>>>>>>>>>>>>>> k=8
>>>>>>>>>>>>>>>>>>>> m=3
>>>>>>>>>>>>>>>>>>>> plugin=jerasure
>>>>>>>>>>>>>>>>>>>> ruleset-failure-domain=osd
>>>>>>>>>>>>>>>>>>>> technique=reed_sol_van
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> the ecdata pool has this as profile
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pool 3 'ecdata' erasure size 11 min_size 8 crush_ruleset
>>>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>>> object_hash
>>>>>>>>>>>>>>>>>>>> rjenkins pg_num 128 pgp_num 128 last_change 161 flags
>>>>>>>>>>>>>>>>>>>> hashpspool
>>>>>>>>>>>>>>>>>>>> stripe_width 4096
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ECrule in crushmap
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> rule ecdata {
>>>>>>>>>>>>>>>>>>>> ruleset 2
>>>>>>>>>>>>>>>>>>>> type erasure
>>>>>>>>>>>>>>>>>>>> min_size 3
>>>>>>>>>>>>>>>>>>>> max_size 20
>>>>>>>>>>>>>>>>>>>> step set_chooseleaf_tries 5
>>>>>>>>>>>>>>>>>>>> step take default-ec
>>>>>>>>>>>>>>>>>>>> step choose indep 0 type osd
>>>>>>>>>>>>>>>>>>>> step emit
>>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>>> root default-ec {
>>>>>>>>>>>>>>>>>>>> id -8 # do not change unnecessarily
>>>>>>>>>>>>>>>>>>>> # weight 140.616
>>>>>>>>>>>>>>>>>>>> alg straw
>>>>>>>>>>>>>>>>>>>> hash 0 # rjenkins1
>>>>>>>>>>>>>>>>>>>> item ceph001-ec weight 46.872
>>>>>>>>>>>>>>>>>>>> item ceph002-ec weight 46.872
>>>>>>>>>>>>>>>>>>>> item ceph003-ec weight 46.872
>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Cheers!
>>>>>>>>>>>>>>>>>>>> Kenneth
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ----- Message from Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>>>> ---------
>>>>>>>>>>>>>>>>>>>> Date: Thu, 14 Aug 2014 10:07:50 +0800
>>>>>>>>>>>>>>>>>>>> From: Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>>>> Subject: Re: ceph cluster inconsistency?
>>>>>>>>>>>>>>>>>>>> To: Kenneth Waegeman <Kenneth.Waegeman at ugent.be>
>>>>>>>>>>>>>>>>>>>> Cc: ceph-users <ceph-users at lists.ceph.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Kenneth,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Could you give your configuration related to EC and
>>>>>>>>>>>>>>>>>>>>> KeyValueStore?
>>>>>>>>>>>>>>>>>>>>> Not sure whether it's bug on KeyValueStore
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Thu, Aug 14, 2014 at 12:06 AM, Kenneth Waegeman
>>>>>>>>>>>>>>>>>>>>> <Kenneth.Waegeman at ugent.be> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I was doing some tests with rados bench on a Erasure
>>>>>>>>>>>>>>>>>>>>>> Coded
>>>>>>>>>>>>>>>>>>>>>> pool
>>>>>>>>>>>>>>>>>>>>>> (using
>>>>>>>>>>>>>>>>>>>>>> keyvaluestore-dev objectstore) on 0.83, and I see some
>>>>>>>>>>>>>>>>>>>>>> strangs
>>>>>>>>>>>>>>>>>>>>>> things:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [root at ceph001 ~]# ceph status
>>>>>>>>>>>>>>>>>>>>>> cluster 82766e04-585b-49a6-a0ac-c13d9ffd0a7d
>>>>>>>>>>>>>>>>>>>>>> health HEALTH_WARN too few pgs per osd (4 < min 20)
>>>>>>>>>>>>>>>>>>>>>> monmap e1: 3 mons at
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> {ceph001=10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,
>>>>>>>>>>>>>>>>>>>>>> ceph003=10.141.8.182:6789/0},
>>>>>>>>>>>>>>>>>>>>>> election epoch 6, quorum 0,1,2 ceph001,ceph002,ceph003
>>>>>>>>>>>>>>>>>>>>>> mdsmap e116: 1/1/1 up
>>>>>>>>>>>>>>>>>>>>>> {0=ceph001.cubone.os=up:active},
>>>>>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>>>>> up:standby
>>>>>>>>>>>>>>>>>>>>>> osdmap e292: 78 osds: 78 up, 78 in
>>>>>>>>>>>>>>>>>>>>>> pgmap v48873: 320 pgs, 4 pools, 15366 GB data,
>>>>>>>>>>>>>>>>>>>>>> 3841
>>>>>>>>>>>>>>>>>>>>>> kobjects
>>>>>>>>>>>>>>>>>>>>>> 1381 GB used, 129 TB / 131 TB avail
>>>>>>>>>>>>>>>>>>>>>> 320 active+clean
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> There is around 15T of data, but only 1.3 T usage.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This is also visible in rados:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [root at ceph001 ~]# rados df
>>>>>>>>>>>>>>>>>>>>>> pool name category KB
>>>>>>>>>>>>>>>>>>>>>> objects
>>>>>>>>>>>>>>>>>>>>>> clones
>>>>>>>>>>>>>>>>>>>>>> degraded unfound rd rd KB
>>>>>>>>>>>>>>>>>>>>>> wr
>>>>>>>>>>>>>>>>>>>>>> wr
>>>>>>>>>>>>>>>>>>>>>> KB
>>>>>>>>>>>>>>>>>>>>>> data - 0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 0 0 0 0 0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ecdata - 16113451009
>>>>>>>>>>>>>>>>>>>>>> 3933959
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 0 0 1 1 3935632
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 16116850711
>>>>>>>>>>>>>>>>>>>>>> metadata - 2
>>>>>>>>>>>>>>>>>>>>>> 20
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 0 0 33 36 21
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 8
>>>>>>>>>>>>>>>>>>>>>> rbd - 0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 0 0 0 0 0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> total used 1448266016 3933979
>>>>>>>>>>>>>>>>>>>>>> total avail 139400181016
>>>>>>>>>>>>>>>>>>>>>> total space 140848447032
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Another (related?) thing: if I do rados -p ecdata ls,
>>>>>>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>>>>>>> trigger
>>>>>>>>>>>>>>>>>>>>>> osd
>>>>>>>>>>>>>>>>>>>>>> shutdowns (each time):
>>>>>>>>>>>>>>>>>>>>>> I get a list followed by an error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_8961_object243839
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object801983
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_31461_object856489
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_8961_object202232
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_4919_object33199
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object807797
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_4919_object74729
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_31461_object1264121
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object1318513
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object1202111
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_31461_object939107
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_31461_object729682
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object122915
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object76521
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object113261
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_31461_object575079
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object671042
>>>>>>>>>>>>>>>>>>>>>> benchmark_data_ceph001.cubone.os_5560_object381146
>>>>>>>>>>>>>>>>>>>>>> 2014-08-13 17:57:48.736150 7f65047b5700 0 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.180:0/1023295 >>
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6839/4471 pipe(0x7f64fc019b20 sd=5 :0 s=1
>>>>>>>>>>>>>>>>>>>>>> pgs=0
>>>>>>>>>>>>>>>>>>>>>> cs=0
>>>>>>>>>>>>>>>>>>>>>> l=1
>>>>>>>>>>>>>>>>>>>>>> c=0x7f64fc019db0).fault
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> And I can see this in the log files:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -25> 2014-08-13 17:52:56.323908 7f8a97fa4700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.182:6827/64670 <== osd.57
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:0/15796
>>>>>>>>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping e220 stamp 2014-08-13 17:52:56.323092)
>>>>>>>>>>>>>>>>>>>>>> v2
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> 47+0+0
>>>>>>>>>>>>>>>>>>>>>> (3227325175 0 0) 0xf475940 con 0xee89fa0
>>>>>>>>>>>>>>>>>>>>>> -24> 2014-08-13 17:52:56.323938 7f8a97fa4700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.182:6827/64670 --> 10.141.8.182:0/15796 --
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping_reply
>>>>>>>>>>>>>>>>>>>>>> e220
>>>>>>>>>>>>>>>>>>>>>> stamp 2014-08-13 17:52:56.323092) v2 -- ?+0 0xf815b00
>>>>>>>>>>>>>>>>>>>>>> con
>>>>>>>>>>>>>>>>>>>>>> 0xee89fa0
>>>>>>>>>>>>>>>>>>>>>> -23> 2014-08-13 17:52:56.324078 7f8a997a7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6840/64670 <== osd.57
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:0/15796
>>>>>>>>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping e220 stamp 2014-08-13 17:52:56.323092)
>>>>>>>>>>>>>>>>>>>>>> v2
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> 47+0+0
>>>>>>>>>>>>>>>>>>>>>> (3227325175 0 0) 0xf132bc0 con 0xee8a680
>>>>>>>>>>>>>>>>>>>>>> -22> 2014-08-13 17:52:56.324111 7f8a997a7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6840/64670 --> 10.141.8.182:0/15796 --
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping_reply
>>>>>>>>>>>>>>>>>>>>>> e220
>>>>>>>>>>>>>>>>>>>>>> stamp 2014-08-13 17:52:56.323092) v2 -- ?+0 0xf811a40
>>>>>>>>>>>>>>>>>>>>>> con
>>>>>>>>>>>>>>>>>>>>>> 0xee8a680
>>>>>>>>>>>>>>>>>>>>>> -21> 2014-08-13 17:52:56.584461 7f8a997a7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6840/64670 <== osd.29
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.181:0/12142
>>>>>>>>>>>>>>>>>>>>>> 47
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping e220 stamp 2014-08-13 17:52:56.583010)
>>>>>>>>>>>>>>>>>>>>>> v2
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> 47+0+0
>>>>>>>>>>>>>>>>>>>>>> (3355887204 0 0) 0xf655940 con 0xee88b00
>>>>>>>>>>>>>>>>>>>>>> -20> 2014-08-13 17:52:56.584486 7f8a997a7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6840/64670 --> 10.143.8.181:0/12142 --
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping_reply
>>>>>>>>>>>>>>>>>>>>>> e220
>>>>>>>>>>>>>>>>>>>>>> stamp 2014-08-13 17:52:56.583010) v2 -- ?+0 0xf132bc0
>>>>>>>>>>>>>>>>>>>>>> con
>>>>>>>>>>>>>>>>>>>>>> 0xee88b00
>>>>>>>>>>>>>>>>>>>>>> -19> 2014-08-13 17:52:56.584498 7f8a97fa4700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.182:6827/64670 <== osd.29
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.181:0/12142
>>>>>>>>>>>>>>>>>>>>>> 47
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping e220 stamp 2014-08-13 17:52:56.583010)
>>>>>>>>>>>>>>>>>>>>>> v2
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> 47+0+0
>>>>>>>>>>>>>>>>>>>>>> (3355887204 0 0) 0xf20e040 con 0xee886e0
>>>>>>>>>>>>>>>>>>>>>> -18> 2014-08-13 17:52:56.584526 7f8a97fa4700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.143.8.182:6827/64670 --> 10.143.8.181:0/12142 --
>>>>>>>>>>>>>>>>>>>>>> osd_ping(ping_reply
>>>>>>>>>>>>>>>>>>>>>> e220
>>>>>>>>>>>>>>>>>>>>>> stamp 2014-08-13 17:52:56.583010) v2 -- ?+0 0xf475940
>>>>>>>>>>>>>>>>>>>>>> con
>>>>>>>>>>>>>>>>>>>>>> 0xee886e0
>>>>>>>>>>>>>>>>>>>>>> -17> 2014-08-13 17:52:56.594448 7f8a798c7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6839/64670 >> :/0 pipe(0xec15f00 sd=74
>>>>>>>>>>>>>>>>>>>>>> :6839
>>>>>>>>>>>>>>>>>>>>>> s=0
>>>>>>>>>>>>>>>>>>>>>> pgs=0
>>>>>>>>>>>>>>>>>>>>>> cs=0
>>>>>>>>>>>>>>>>>>>>>> l=0
>>>>>>>>>>>>>>>>>>>>>> c=0xee856a0).accept sd=74 10.141.8.180:47641/0
>>>>>>>>>>>>>>>>>>>>>> -16> 2014-08-13 17:52:56.594921 7f8a798c7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6839/64670 <== client.7512
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.180:0/1018433
>>>>>>>>>>>>>>>>>>>>>> 1
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220) v4 ==== 151+0+39
>>>>>>>>>>>>>>>>>>>>>> (1972163119
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 4174233976) 0xf3bca40 con 0xee856a0
>>>>>>>>>>>>>>>>>>>>>> -15> 2014-08-13 17:52:56.594957 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.594874, event:
>>>>>>>>>>>>>>>>>>>>>> header_read,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -14> 2014-08-13 17:52:56.594970 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.594880, event:
>>>>>>>>>>>>>>>>>>>>>> throttled,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -13> 2014-08-13 17:52:56.594978 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.594917, event:
>>>>>>>>>>>>>>>>>>>>>> all_read,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -12> 2014-08-13 17:52:56.594986 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 0.000000, event: dispatched, op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1
>>>>>>>>>>>>>>>>>>>>>> [pgls
>>>>>>>>>>>>>>>>>>>>>> start_epoch 0] 3.0 ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -11> 2014-08-13 17:52:56.595127 7f8a90795700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.595104, event:
>>>>>>>>>>>>>>>>>>>>>> reached_pg,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -10> 2014-08-13 17:52:56.595159 7f8a90795700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.595153, event: started,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -9> 2014-08-13 17:52:56.602179 7f8a90795700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6839/64670 --> 10.141.8.180:0/1018433 --
>>>>>>>>>>>>>>>>>>>>>> osd_op_reply(1
>>>>>>>>>>>>>>>>>>>>>> [pgls
>>>>>>>>>>>>>>>>>>>>>> start_epoch 0] v164'30654 uv30654 ondisk = 0) v6 --
>>>>>>>>>>>>>>>>>>>>>> ?+0
>>>>>>>>>>>>>>>>>>>>>> 0xec16180
>>>>>>>>>>>>>>>>>>>>>> con
>>>>>>>>>>>>>>>>>>>>>> 0xee856a0
>>>>>>>>>>>>>>>>>>>>>> -8> 2014-08-13 17:52:56.602211 7f8a90795700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 299, time: 2014-08-13 17:52:56.602205, event: done,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:1 [pgls start_epoch 0] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -7> 2014-08-13 17:52:56.614839 7f8a798c7700 1 --
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.182:6839/64670 <== client.7512
>>>>>>>>>>>>>>>>>>>>>> 10.141.8.180:0/1018433
>>>>>>>>>>>>>>>>>>>>>> 2
>>>>>>>>>>>>>>>>>>>>>> ====
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220) v4 ==== 151+0+89
>>>>>>>>>>>>>>>>>>>>>> (3460833343
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2600845095) 0xf3bcec0 con 0xee856a0
>>>>>>>>>>>>>>>>>>>>>> -6> 2014-08-13 17:52:56.614864 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 2014-08-13 17:52:56.614789, event:
>>>>>>>>>>>>>>>>>>>>>> header_read,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -5> 2014-08-13 17:52:56.614874 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 2014-08-13 17:52:56.614792, event:
>>>>>>>>>>>>>>>>>>>>>> throttled,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -4> 2014-08-13 17:52:56.614884 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 2014-08-13 17:52:56.614835, event:
>>>>>>>>>>>>>>>>>>>>>> all_read,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -3> 2014-08-13 17:52:56.614891 7f8a798c7700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 0.000000, event: dispatched, op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2
>>>>>>>>>>>>>>>>>>>>>> [pgls
>>>>>>>>>>>>>>>>>>>>>> start_epoch 220] 3.0 ack+read+known_if_redirected
>>>>>>>>>>>>>>>>>>>>>> e220)
>>>>>>>>>>>>>>>>>>>>>> -2> 2014-08-13 17:52:56.614972 7f8a92f9a700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 2014-08-13 17:52:56.614958, event:
>>>>>>>>>>>>>>>>>>>>>> reached_pg,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> -1> 2014-08-13 17:52:56.614993 7f8a92f9a700 5 -- op
>>>>>>>>>>>>>>>>>>>>>> tracker
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> ,
>>>>>>>>>>>>>>>>>>>>>> seq:
>>>>>>>>>>>>>>>>>>>>>> 300, time: 2014-08-13 17:52:56.614986, event: started,
>>>>>>>>>>>>>>>>>>>>>> op:
>>>>>>>>>>>>>>>>>>>>>> osd_op(client.7512.0:2 [pgls start_epoch 220] 3.0
>>>>>>>>>>>>>>>>>>>>>> ack+read+known_if_redirected e220)
>>>>>>>>>>>>>>>>>>>>>> 0> 2014-08-13 17:52:56.617087 7f8a92f9a700 -1
>>>>>>>>>>>>>>>>>>>>>> os/GenericObjectMap.cc:
>>>>>>>>>>>>>>>>>>>>>> In function 'int GenericObjectMap::list_objects(const
>>>>>>>>>>>>>>>>>>>>>> coll_t&,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int, std::vector<ghobject_t>*, ghobject_t*)' thread
>>>>>>>>>>>>>>>>>>>>>> 7f8a92f9a700
>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>> 2014-08-13 17:52:56.615073
>>>>>>>>>>>>>>>>>>>>>> os/GenericObjectMap.cc: 1118: FAILED assert(start <=
>>>>>>>>>>>>>>>>>>>>>> header.oid)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ceph version 0.83 (78ff1f0a5dfd3c5850805b40217385
>>>>>>>>>>>>>>>>>>>>>> 64c36c92b8)
>>>>>>>>>>>>>>>>>>>>>> 1: (GenericObjectMap::list_objects(coll_t const&,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> std::vector<ghobject_t, std::allocator<ghobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x474)
>>>>>>>>>>>>>>>>>>>>>> [0x98f774]
>>>>>>>>>>>>>>>>>>>>>> 2: (KeyValueStore::collection_list_partial(coll_t,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t, std::vector<ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> std::allocator<ghobject_t>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x274) [0x8c5b54]
>>>>>>>>>>>>>>>>>>>>>> 3: (PGBackend::objects_list_partial(hobject_t const&,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t,
>>>>>>>>>>>>>>>>>>>>>> std::vector<hobject_t, std::allocator<hobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> hobject_t*)+0x1c9)
>>>>>>>>>>>>>>>>>>>>>> [0x862de9]
>>>>>>>>>>>>>>>>>>>>>> 4:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_pg_op(std::tr1::shared_ptr<OpRequest>)+
>>>>>>>>>>>>>>>>>>>>>> 0xea5)
>>>>>>>>>>>>>>>>>>>>>> [0x7f67f5]
>>>>>>>>>>>>>>>>>>>>>> 5:
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_op(std::tr1:
>>>>>>>>>>>>>>>>>>>>>> :shared_ptr<OpRequest>)+0x1f3)
>>>>>>>>>>>>>>>>>>>>>> [0x8177b3]
>>>>>>>>>>>>>>>>>>>>>> 6: (ReplicatedPG::do_request(std:
>>>>>>>>>>>>>>>>>>>>>> :tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x5d5) [0x7b8045]
>>>>>>>>>>>>>>>>>>>>>> 7: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
>>>>>>>>>>>>>>>>>>>>>> std::tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x47d)
>>>>>>>>>>>>>>>>>>>>>> [0x62bf8d]
>>>>>>>>>>>>>>>>>>>>>> 8: (OSD::ShardedOpWQ::_process(unsigned int,
>>>>>>>>>>>>>>>>>>>>>> ceph::heartbeat_handle_d*)+0x35c) [0x62c56c]
>>>>>>>>>>>>>>>>>>>>>> 9:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned
>>>>>>>>>>>>>>>>>>>>>> int)+0x8cd)
>>>>>>>>>>>>>>>>>>>>>> [0xa776fd]
>>>>>>>>>>>>>>>>>>>>>> 10:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
>>>>>>>>>>>>>>>>>>>>>> [0xa79980]
>>>>>>>>>>>>>>>>>>>>>> 11: (()+0x7df3) [0x7f8aac71fdf3]
>>>>>>>>>>>>>>>>>>>>>> 12: (clone()+0x6d) [0x7f8aab1963dd]
>>>>>>>>>>>>>>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS
>>>>>>>>>>>>>>>>>>>>>> <executable>`
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> interpret this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ceph version 0.83 (78ff1f0a5dfd3c5850805b40217385
>>>>>>>>>>>>>>>>>>>>>> 64c36c92b8)
>>>>>>>>>>>>>>>>>>>>>> 1: /usr/bin/ceph-osd() [0x99b466]
>>>>>>>>>>>>>>>>>>>>>> 2: (()+0xf130) [0x7f8aac727130]
>>>>>>>>>>>>>>>>>>>>>> 3: (gsignal()+0x39) [0x7f8aab0d5989]
>>>>>>>>>>>>>>>>>>>>>> 4: (abort()+0x148) [0x7f8aab0d7098]
>>>>>>>>>>>>>>>>>>>>>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165)
>>>>>>>>>>>>>>>>>>>>>> [0x7f8aab9e89d5]
>>>>>>>>>>>>>>>>>>>>>> 6: (()+0x5e946) [0x7f8aab9e6946]
>>>>>>>>>>>>>>>>>>>>>> 7: (()+0x5e973) [0x7f8aab9e6973]
>>>>>>>>>>>>>>>>>>>>>> 8: (()+0x5eb9f) [0x7f8aab9e6b9f]
>>>>>>>>>>>>>>>>>>>>>> 9: (ceph::__ceph_assert_fail(char const*, char const*,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> char
>>>>>>>>>>>>>>>>>>>>>> const*)+0x1ef) [0xa8805f]
>>>>>>>>>>>>>>>>>>>>>> 10: (GenericObjectMap::list_objects(coll_t const&,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> std::vector<ghobject_t, std::allocator<ghobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x474)
>>>>>>>>>>>>>>>>>>>>>> [0x98f774]
>>>>>>>>>>>>>>>>>>>>>> 11: (KeyValueStore::collection_list_partial(coll_t,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t, std::vector<ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> std::allocator<ghobject_t>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x274) [0x8c5b54]
>>>>>>>>>>>>>>>>>>>>>> 12: (PGBackend::objects_list_partial(hobject_t const&,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t,
>>>>>>>>>>>>>>>>>>>>>> std::vector<hobject_t, std::allocator<hobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> hobject_t*)+0x1c9)
>>>>>>>>>>>>>>>>>>>>>> [0x862de9]
>>>>>>>>>>>>>>>>>>>>>> 13:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_pg_op(std::tr1::shared_ptr<OpRequest>)+
>>>>>>>>>>>>>>>>>>>>>> 0xea5)
>>>>>>>>>>>>>>>>>>>>>> [0x7f67f5]
>>>>>>>>>>>>>>>>>>>>>> 14:
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_op(std::tr1:
>>>>>>>>>>>>>>>>>>>>>> :shared_ptr<OpRequest>)+0x1f3)
>>>>>>>>>>>>>>>>>>>>>> [0x8177b3]
>>>>>>>>>>>>>>>>>>>>>> 15:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x5d5) [0x7b8045]
>>>>>>>>>>>>>>>>>>>>>> 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
>>>>>>>>>>>>>>>>>>>>>> std::tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x47d)
>>>>>>>>>>>>>>>>>>>>>> [0x62bf8d]
>>>>>>>>>>>>>>>>>>>>>> 17: (OSD::ShardedOpWQ::_process(unsigned int,
>>>>>>>>>>>>>>>>>>>>>> ceph::heartbeat_handle_d*)+0x35c) [0x62c56c]
>>>>>>>>>>>>>>>>>>>>>> 18:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned
>>>>>>>>>>>>>>>>>>>>>> int)+0x8cd)
>>>>>>>>>>>>>>>>>>>>>> [0xa776fd]
>>>>>>>>>>>>>>>>>>>>>> 19:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
>>>>>>>>>>>>>>>>>>>>>> [0xa79980]
>>>>>>>>>>>>>>>>>>>>>> 20: (()+0x7df3) [0x7f8aac71fdf3]
>>>>>>>>>>>>>>>>>>>>>> 21: (clone()+0x6d) [0x7f8aab1963dd]
>>>>>>>>>>>>>>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS
>>>>>>>>>>>>>>>>>>>>>> <executable>`
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> interpret this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --- begin dump of recent events ---
>>>>>>>>>>>>>>>>>>>>>> 0> 2014-08-13 17:52:56.714214 7f8a92f9a700 -1 ***
>>>>>>>>>>>>>>>>>>>>>> Caught
>>>>>>>>>>>>>>>>>>>>>> signal
>>>>>>>>>>>>>>>>>>>>>> (Aborted) **
>>>>>>>>>>>>>>>>>>>>>> in thread 7f8a92f9a700
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ceph version 0.83 (78ff1f0a5dfd3c5850805b40217385
>>>>>>>>>>>>>>>>>>>>>> 64c36c92b8)
>>>>>>>>>>>>>>>>>>>>>> 1: /usr/bin/ceph-osd() [0x99b466]
>>>>>>>>>>>>>>>>>>>>>> 2: (()+0xf130) [0x7f8aac727130]
>>>>>>>>>>>>>>>>>>>>>> 3: (gsignal()+0x39) [0x7f8aab0d5989]
>>>>>>>>>>>>>>>>>>>>>> 4: (abort()+0x148) [0x7f8aab0d7098]
>>>>>>>>>>>>>>>>>>>>>> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165)
>>>>>>>>>>>>>>>>>>>>>> [0x7f8aab9e89d5]
>>>>>>>>>>>>>>>>>>>>>> 6: (()+0x5e946) [0x7f8aab9e6946]
>>>>>>>>>>>>>>>>>>>>>> 7: (()+0x5e973) [0x7f8aab9e6973]
>>>>>>>>>>>>>>>>>>>>>> 8: (()+0x5eb9f) [0x7f8aab9e6b9f]
>>>>>>>>>>>>>>>>>>>>>> 9: (ceph::__ceph_assert_fail(char const*, char const*,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> char
>>>>>>>>>>>>>>>>>>>>>> const*)+0x1ef) [0xa8805f]
>>>>>>>>>>>>>>>>>>>>>> 10: (GenericObjectMap::list_objects(coll_t const&,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> std::vector<ghobject_t, std::allocator<ghobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x474)
>>>>>>>>>>>>>>>>>>>>>> [0x98f774]
>>>>>>>>>>>>>>>>>>>>>> 11: (KeyValueStore::collection_list_partial(coll_t,
>>>>>>>>>>>>>>>>>>>>>> ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t, std::vector<ghobject_t,
>>>>>>>>>>>>>>>>>>>>>> std::allocator<ghobject_t>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> *,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> ghobject_t*)+0x274) [0x8c5b54]
>>>>>>>>>>>>>>>>>>>>>> 12: (PGBackend::objects_list_partial(hobject_t const&,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> int,
>>>>>>>>>>>>>>>>>>>>>> snapid_t,
>>>>>>>>>>>>>>>>>>>>>> std::vector<hobject_t, std::allocator<hobject_t> >*,
>>>>>>>>>>>>>>>>>>>>>> hobject_t*)+0x1c9)
>>>>>>>>>>>>>>>>>>>>>> [0x862de9]
>>>>>>>>>>>>>>>>>>>>>> 13:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_pg_op(std::tr1::shared_ptr<OpRequest>)+
>>>>>>>>>>>>>>>>>>>>>> 0xea5)
>>>>>>>>>>>>>>>>>>>>>> [0x7f67f5]
>>>>>>>>>>>>>>>>>>>>>> 14:
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_op(std::tr1:
>>>>>>>>>>>>>>>>>>>>>> :shared_ptr<OpRequest>)+0x1f3)
>>>>>>>>>>>>>>>>>>>>>> [0x8177b3]
>>>>>>>>>>>>>>>>>>>>>> 15:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x5d5) [0x7b8045]
>>>>>>>>>>>>>>>>>>>>>> 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>,
>>>>>>>>>>>>>>>>>>>>>> std::tr1::shared_ptr<OpRequest>,
>>>>>>>>>>>>>>>>>>>>>> ThreadPool::TPHandle&)+0x47d)
>>>>>>>>>>>>>>>>>>>>>> [0x62bf8d]
>>>>>>>>>>>>>>>>>>>>>> 17: (OSD::ShardedOpWQ::_process(unsigned int,
>>>>>>>>>>>>>>>>>>>>>> ceph::heartbeat_handle_d*)+0x35c) [0x62c56c]
>>>>>>>>>>>>>>>>>>>>>> 18:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::shardedthreadpool_worker(unsigned
>>>>>>>>>>>>>>>>>>>>>> int)+0x8cd)
>>>>>>>>>>>>>>>>>>>>>> [0xa776fd]
>>>>>>>>>>>>>>>>>>>>>> 19:
>>>>>>>>>>>>>>>>>>>>>> (ShardedThreadPool::WorkThreadSharded::entry()+0x10)
>>>>>>>>>>>>>>>>>>>>>> [0xa79980]
>>>>>>>>>>>>>>>>>>>>>> 20: (()+0x7df3) [0x7f8aac71fdf3]
>>>>>>>>>>>>>>>>>>>>>> 21: (clone()+0x6d) [0x7f8aab1963dd]
>>>>>>>>>>>>>>>>>>>>>> NOTE: a copy of the executable, or `objdump -rdS
>>>>>>>>>>>>>>>>>>>>>> <executable>`
>>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>>>>> needed
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> interpret this.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I guess this has something to do with using the dev
>>>>>>>>>>>>>>>>>>>>>> Keyvaluestore?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Kenneth
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>>>>>> ceph-users mailing list
>>>>>>>>>>>>>>>>>>>>>> ceph-users at lists.ceph.com
>>>>>>>>>>>>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Wheat
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ----- End message from Haomai Wang
>>>>>>>>>>>>>>>>>>>> <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>>>>>>>>> Kenneth Waegeman
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Wheat
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com>
>>>>>>>>>>>>>>>>>> -----
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>>>>>>> Kenneth Waegeman
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Wheat
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> ceph-users mailing list
>>>>>>>>>>>>>>>>> ceph-users at lists.ceph.com
>>>>>>>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ----- End message from Sage Weil <sweil at redhat.com> -----
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Met vriendelijke groeten,
>>>
>>>
>>>
>>>>>>>>>>>>>>> Kenneth Waegeman
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Wheat
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>> Kenneth Waegeman
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Wheat
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>> Kenneth Waegeman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards,
>>>>>>>>>>
>>>>>>>>>> Wheat
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Best Regards,
>>>>>>>>>
>>>>>>>>> Wheat
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Met vriendelijke groeten,
>>>>>>>> Kenneth Waegeman
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Wheat
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Met vriendelijke groeten,
>>>>>> Kenneth Waegeman
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Best Regards,
>>>>>
>>>>> Wheat
>>>>
>>>>
>>>>
>>>>
>>>> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>>>>
>>>> --
>>>>
>>>> Met vriendelijke groeten,
>>>> Kenneth Waegeman
>>>
>>>
>>>
>>>
>>> ----- End message from Kenneth Waegeman <Kenneth.Waegeman at UGent.be> -----
>>>
>>>
>>> --
>>>
>>> Met vriendelijke groeten,
>>> Kenneth Waegeman
>>>
>>
>>
>>
>> --
>> Best Regards,
>>
>> Wheat
>
>
>
> ----- End message from Haomai Wang <haomaiwang at gmail.com> -----
>
> --
>
> Met vriendelijke groeten,
> Kenneth Waegeman
>
>
--
Best Regards,
Wheat
