Hi,
I noticed that problem is more frequent at nigth where traffic is smaller.
Mabye it is caused by scrubbing (multiple scrubbings on one osd) and
to small "filestore op threads" or something other num threads
settings in my config:
osd heartbeat grace = 15
filestore flush min = 0
filestore flusher = false
filestore fiemap = false
filestore op threads = 4
filestore queue max ops = 4096
filestore queue max bytes = 10485760
filestore queue committing max bytes = 10485760
osd op threads = 8
osd disk threads = 4
osd recovery threads = 1
osd recovery max active = 1
osd recovery op priority = 1
osd client op priority = 100
osd max backfills = 1
--
Regards
Dominik
2013/7/4 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
> I reported bug: http://tracker.ceph.com/issues/5504
>
> --
> Regards
> Dominik
>
> 2013/7/2 Dominik Mostowiec <dominikmostowiec@xxxxxxxxx>:
>> Hi,
>> Some osd.87 performance graphs:
>> https://www.dropbox.com/s/o07wae2041hu06l/osd_87_performance.PNG
>> After 11.05 I have restarted it.
>>
>> Mons .., maybe this is the problem.
>>
>> --
>> Regards
>> Dominik
>>
>> 2013/7/2 Andrey Korolyov <andrey@xxxxxxx>:
>>> Hi Dominik,
>>>
>>> What`s about performance on the osd.87 at this moment, do you have any
>>> related measurements?
>>>
>>> As for mine version of this issue, seems that quorum has some kind of
>>> degradation over time - when I restarted mons, problem has gone and peering
>>> time lowered by factor of ten or so. Also seems that the problem has a
>>> cumulative origin in the quorum - I did disk replacement over last week and
>>> every time peering gets worse and worse. I assume that it`s a time to put
>>> more or less formalized problems to the bugtracker:
>>> - such degradation over a time plus stuck placement groups,
>>> - newer kind of problem related to the epochs too - restarting one mon
>>> resulting to slight dataplacement change at the moment when _first rebooted_
>>> monitor came up, not shown up with one hour delays between quorum restart.
>>>
>>>
>>>
>>>
>>> On Tue, Jul 2, 2013 at 1:37 PM, Dominik Mostowiec
>>> <dominikmostowiec@xxxxxxxxx> wrote:
>>>>
>>>> Hi,
>>>> I got it.
>>>>
>>>> ceph health details
>>>> HEALTH_WARN 3 pgs peering; 3 pgs stuck inactive; 5 pgs stuck unclean;
>>>> recovery 64/38277874 degraded (0.000%)
>>>> pg 5.df9 is stuck inactive for 138669.746512, current state peering,
>>>> last acting [87,2,151]
>>>> pg 5.a82 is stuck inactive for 138638.121867, current state peering,
>>>> last acting [151,87,42]
>>>> pg 5.80d is stuck inactive for 138621.069523, current state peering,
>>>> last acting [151,47,87]
>>>> pg 5.df9 is stuck unclean for 138669.746761, current state peering,
>>>> last acting [87,2,151]
>>>> pg 5.ae2 is stuck unclean for 139479.810499, current state active,
>>>> last acting [87,151,28]
>>>> pg 5.7b6 is stuck unclean for 139479.693271, current state active,
>>>> last acting [87,105,2]
>>>> pg 5.a82 is stuck unclean for 139479.713859, current state peering,
>>>> last acting [151,87,42]
>>>> pg 5.80d is stuck unclean for 139479.800820, current state peering,
>>>> last acting [151,47,87]
>>>> pg 5.df9 is peering, acting [87,2,151]
>>>> pg 5.a82 is peering, acting [151,87,42]
>>>> pg 5.80d is peering, acting [151,47,87]
>>>> recovery 64/38277874 degraded (0.000%)
>>>>
>>>>
>>>> osd pg query for 5.df9:
>>>> { "state": "peering",
>>>> "up": [
>>>> 87,
>>>> 2,
>>>> 151],
>>>> "acting": [
>>>> 87,
>>>> 2,
>>>> 151],
>>>> "info": { "pgid": "5.df9",
>>>> "last_update": "119454'58844953",
>>>> "last_complete": "119454'58844953",
>>>> "log_tail": "119454'58843952",
>>>> "last_backfill": "MAX",
>>>> "purged_snaps": "[]",
>>>> "history": { "epoch_created": 365,
>>>> "last_epoch_started": 119456,
>>>> "last_epoch_clean": 119456,
>>>> "last_epoch_split": 117806,
>>>> "same_up_since": 119458,
>>>> "same_interval_since": 119458,
>>>> "same_primary_since": 119458,
>>>> "last_scrub": "119442'58732630",
>>>> "last_scrub_stamp": "2013-06-29 20:02:24.817352",
>>>> "last_deep_scrub": "119271'57224023",
>>>> "last_deep_scrub_stamp": "2013-06-23 02:04:49.654373",
>>>> "last_clean_scrub_stamp": "2013-06-29 20:02:24.817352"},
>>>> "stats": { "version": "119454'58844953",
>>>> "reported": "119458'42382189",
>>>> "state": "peering",
>>>> "last_fresh": "2013-06-30 20:35:29.489826",
>>>> "last_change": "2013-06-30 20:35:28.469854",
>>>> "last_active": "2013-06-30 20:33:24.126599",
>>>> "last_clean": "2013-06-30 20:33:24.126599",
>>>> "last_unstale": "2013-06-30 20:35:29.489826",
>>>> "mapping_epoch": 119455,
>>>> "log_start": "119454'58843952",
>>>> "ondisk_log_start": "119454'58843952",
>>>> "created": 365,
>>>> "last_epoch_clean": 365,
>>>> "parent": "0.0",
>>>> "parent_split_bits": 0,
>>>> "last_scrub": "119442'58732630",
>>>> "last_scrub_stamp": "2013-06-29 20:02:24.817352",
>>>> "last_deep_scrub": "119271'57224023",
>>>> "last_deep_scrub_stamp": "2013-06-23 02:04:49.654373",
>>>> "last_clean_scrub_stamp": "2013-06-29 20:02:24.817352",
>>>> "log_size": 135341,
>>>> "ondisk_log_size": 135341,
>>>> "stats_invalid": "0",
>>>> "stat_sum": { "num_bytes": 1010563373,
>>>> "num_objects": 3099,
>>>> "num_object_clones": 0,
>>>> "num_object_copies": 0,
>>>> "num_objects_missing_on_primary": 0,
>>>> "num_objects_degraded": 0,
>>>> "num_objects_unfound": 0,
>>>> "num_read": 302,
>>>> "num_read_kb": 0,
>>>> "num_write": 32264,
>>>> "num_write_kb": 798650,
>>>> "num_scrub_errors": 0,
>>>> "num_objects_recovered": 8235,
>>>> "num_bytes_recovered": 2085653757,
>>>> "num_keys_recovered": 249061471},
>>>> "stat_cat_sum": {},
>>>> "up": [
>>>> 87,
>>>> 2,
>>>> 151],
>>>> "acting": [
>>>> 87,
>>>> 2,
>>>> 151]},
>>>> "empty": 0,
>>>> "dne": 0,
>>>> "incomplete": 0,
>>>> "last_epoch_started": 119454},
>>>> "recovery_state": [
>>>> { "name": "Started\/Primary\/Peering\/GetLog",
>>>> "enter_time": "2013-06-30 20:35:28.545478",
>>>> "newest_update_osd": 2},
>>>> { "name": "Started\/Primary\/Peering",
>>>> "enter_time": "2013-06-30 20:35:28.469841",
>>>> "past_intervals": [
>>>> { "first": 119453,
>>>> "last": 119454,
>>>> "maybe_went_rw": 1,
>>>> "up": [
>>>> 87,
>>>> 2,
>>>> 151],
>>>> "acting": [
>>>> 87,
>>>> 2,
>>>> 151]},
>>>> { "first": 119455,
>>>> "last": 119457,
>>>> "maybe_went_rw": 1,
>>>> "up": [
>>>> 2,
>>>> 151],
>>>> "acting": [
>>>> 2,
>>>> 151]}],
>>>> "probing_osds": [
>>>> 2,
>>>> 87,
>>>> 151],
>>>> "down_osds_we_would_probe": [],
>>>> "peering_blocked_by": []},
>>>> { "name": "Started",
>>>> "enter_time": "2013-06-30 20:35:28.469765"}]}
>>>>
>>>>
>>>> For other PGs: https://www.dropbox.com/s/q5iv8lwzecioy3d/pg_query.tar.tz
>>>>
>>>> --
>>>> Regards
>>>> Dominik
>>>>
>>>> 2013/6/30 Andrey Korolyov <andrey@xxxxxxx>:
>>>> > That`s not a loop as it looks, sorry - I had reproduced issue many
>>>> > times and there is no such cpu-eating behavior in most cases, only
>>>> > locked pgs are presented. Also I may celebrate returning of 'wrong
>>>> > down mark' bug, at least for the 0.61.4 tag. For first one, I`ll send
>>>> > a link with core as quick as I will be able to reproduce it on my test
>>>> > env, and second one linked with 100% disk utilization, so I`m not sure
>>>> > if this is right behavior or wrong.
>>>> >
>>>> > On Sat, Jun 29, 2013 at 1:28 AM, Sage Weil <sage@xxxxxxxxxxx> wrote:
>>>> >> On Sat, 29 Jun 2013, Andrey Korolyov wrote:
>>>> >>> There is almost same problem with the 0.61 cluster, at least with same
>>>> >>> symptoms. Could be reproduced quite easily - remove an osd and then
>>>> >>> mark it as out and with quite high probability one of neighbors will
>>>> >>> be stuck at the end of peering process with couple of peering pgs with
>>>> >>> primary copy on it. Such osd process seems to be stuck in some kind of
>>>> >>> lock, eating exactly 100% of one core.
>>>> >>
>>>> >> Which version?
>>>> >> Can you attach with gdb and get a backtrace to see what it is chewing
>>>> >> on?
>>>> >>
>>>> >> Thanks!
>>>> >> sage
>>>> >>
>>>> >>
>>>> >>>
>>>> >>> On Thu, Jun 13, 2013 at 8:42 PM, Gregory Farnum <greg@xxxxxxxxxxx>
>>>> >>> wrote:
>>>> >>> > On Thu, Jun 13, 2013 at 6:33 AM, S?awomir Skowron <szibis@xxxxxxxxx>
>>>> >>> > wrote:
>>>> >>> >> Hi, sorry for late response.
>>>> >>> >>
>>>> >>> >> https://docs.google.com/file/d/0B9xDdJXMieKEdHFRYnBfT3lCYm8/view
>>>> >>> >>
>>>> >>> >> Logs in attachment, and on google drive, from today.
>>>> >>> >>
>>>> >>> >> https://docs.google.com/file/d/0B9xDdJXMieKEQzVNVHJ1RXFXZlU/view
>>>> >>> >>
>>>> >>> >> We have such problem today. And new logs are on google drive with
>>>> >>> >> today date.
>>>> >>> >>
>>>> >>> >> Strange is that problematic osd.71 have about 10-15%, more space
>>>> >>> >> used
>>>> >>> >> then other osd in cluster.
>>>> >>> >>
>>>> >>> >> Today in one hour osd.71 fails 3 times in mon log, and after third
>>>> >>> >> recovery has been stuck, and many 500 errors appears in http layer
>>>> >>> >> on
>>>> >>> >> top of rgw. When it's stuck, restarting osd71, osd.23, and osd.108,
>>>> >>> >> all from stucked pg, helps, but i run even repair on this osd, just
>>>> >>> >> in
>>>> >>> >> case.
>>>> >>> >>
>>>> >>> >> I have some theory, that on this pg is rgw index of objects, or one
>>>> >>> >> of
>>>> >>> >> osd in this pg, have some problems with local filesystem or drive
>>>> >>> >> bellow (raid controller reports nothing about that), but i do not
>>>> >>> >> see
>>>> >>> >> any problem in system.
>>>> >>> >>
>>>> >>> >> How can we find in which pg/osd index of objects in rgw bucket
>>>> >>> >> exist ??
>>>> >>> >
>>>> >>> > You can find the location of any named object by grabbing the OSD
>>>> >>> > map
>>>> >>> > from the cluster and using the osdmaptool: "osdmaptool <mapfile>
>>>> >>> > --test-map-object <objname> --pool <poolid>".
>>>> >>> >
>>>> >>> > You're not providing any context for your issue though, so we really
>>>> >>> > can't help. What symptoms are you observing?
>>>> >>> > -Greg
>>>> >>> > Software Engineer #42 @ http://inktank.com | http://ceph.com
>>>> >>> > _______________________________________________
>>>> >>> > ceph-users mailing list
>>>> >>> > ceph-users@xxxxxxxxxxxxxx
>>>> >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >>> _______________________________________________
>>>> >>> ceph-users mailing list
>>>> >>> ceph-users@xxxxxxxxxxxxxx
>>>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >>>
>>>> > _______________________________________________
>>>> > ceph-users mailing list
>>>> > ceph-users@xxxxxxxxxxxxxx
>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>
>>>>
>>>>
>>>> --
>>>> Pozdrawiam
>>>> Dominik
>>>
>>>
>>
>>
>>
>> --
>> Pozdrawiam
>> Dominik
>
>
>
> --
> Pozdrawiam
> Dominik
--
Pozdrawiam
Dominik
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
