On Fri, Feb 15, 2013 at 7:02 PM, Kevin Decherf <kevin@xxxxxxxxxxxx> wrote: > On Wed, Feb 13, 2013 at 10:19:36AM -0800, Gregory Farnum wrote: >> On Wed, Feb 13, 2013 at 3:47 AM, Kevin Decherf <kevin@xxxxxxxxxxxx> wrote: >> > On Mon, Feb 11, 2013 at 12:25:59PM -0800, Gregory Farnum wrote: >> >> On Mon, Feb 11, 2013 at 10:54 AM, Kevin Decherf <kevin@xxxxxxxxxxxx> wrote: >> >> > Furthermore, I observe another strange thing more or less related to the >> >> > storms. >> >> > >> >> > During a rsync command to write ~20G of data on Ceph and during (and >> >> > after) the storm, one OSD sends a lot of data to the active MDS >> >> > (400Mbps peak each 6 seconds). After a quick check, I found that when I >> >> > stop osd.23, osd.14 stops its peaks. >> >> >> >> This is consistent with Sam's suggestion that MDS is thrashing its >> >> cache, and is grabbing a directory object off of the OSDs. How large >> >> are the directories you're using? If they're a significant fraction of >> >> your cache size, it might be worth enabling the (sadly less stable) >> >> directory fragmentation options, which will split them up into smaller >> >> fragments that can be independently read and written to disk. >> > >> > I set mds cache size to 400000 but now I observe ~900Mbps peaks from >> > osd.14 to the active mds, osd.18 and osd.2. >> > >> > osd.14 shares some pg with osd.18 and osd.2: >> > http://pastebin.com/raw.php?i=uBAcTcu4 >> >> The high bandwidth from OSD to MDS really isn't a concern — that's the >> MDS asking for data and getting it back quickly! We're concerned about >> client responsiveness; has that gotten better? > > It seems better now, I didn't see any storm so far. > > But we observe high latency on some of our clients (with no load). Does > it exist any documentation on how to read the perfcounters_dump output? > I would like to know if the MDS still has any problem with its cache or > if the latency comes from elsewhere. You can use perfcounters_schema to get the output format in json, using something like json_pp to make it somewhat readable for a human. You can post the dump output as well and we can try to help you parse/understand what its telling you. -sam > > -- > Kevin Decherf - @Kdecherf > GPG C610 FE73 E706 F968 612B E4B2 108A BD75 A81E 6E2F > http://kdecherf.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
