Welcome to ceph-large

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]


Thanks everyone for joining the list!

As I mentioned in the ceph-users post, this list is for people running
large Ceph clusters (>500 OSDs) to discuss issues and experiences that you
don't run into with small clusters.  Please try and keep conversations
which are related to Ceph, but not specific to running a large cluster on
the ceph-users list.

Personally I run two different Ceph clusters that currently have over
1,300 OSDs each.  Recently I've run into two different bugs which I
believe most of us have either run into or will run into.

The first issue was related to excessive OSD maps being kept for every OSD
which resulted in quite a bit of each OSD's storage being wasted (I saw up
to 20%).  By default 500 OSD maps are stored per OSD, but I saw up to
200,000 OSD maps on some OSDs.  This was made worse by the size of the
clusters since the size of each OSD map was ~1 MB.  Here's a link to the


Which was fixed in the 0.94.8 release, but that brings me to the next bug
we ran into...

When we attempted to upgrade our clusters from 0.94.6 to the 0.94.9
release we saw a huge number of slow requests at the start of the upgrade.
 This was caused by a change to the OSD map encoding (introduced in
0.94.7) which caused all the 0.94.6 OSDs to request full OSD maps from the
0.94.9 mon nodes instead of incremental ones.  This flooded the outgoing
network connection on all the mon nodes any time there was an update to
the OSD map for a couple minutes at a time.  This caused all sorts of
problems and even resulted in an incomplete pg at one point.  The link for
this bug is here:


The solution was to upgrade all the OSDs to 0.94.9 first and then upgrade
the mon nodes.

Now that we're on 0.94.9 things are working pretty well, but next up is to
look into upgrading them to Jewel.  Has anyone gone through that process
and willing to share their experiences?


Ceph-large mailing list

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFS]

  Powered by Linux