> Op 19 juli 2016 om 11:55 schreef M Ranga Swami Reddy <swamireddy@xxxxxxxxx>: > > > Thanks for detail... > When an OSD is 95% full, then that specific OSD's write IO blocked. > No, the *whole* cluster will block. In the OSDMap the flag 'full' is set which causes all I/O to stop (even read!) until you make sure the OSD drops below 95%. Wido > Thanks > Swami > > On Tue, Jul 19, 2016 at 3:07 PM, Christian Balzer <chibi@xxxxxxx> wrote: > > > > Hello, > > > > On Tue, 19 Jul 2016 14:23:32 +0530 M Ranga Swami Reddy wrote: > > > >> >> Using ceph cluster with 100+ OSDs and cluster is filled with 60% data. > >> >> One of the OSD is 95% full. > >> >> If an OSD is 95% full, is it impact the any storage operation? Is this > >> >> impacts on VM/Instance? > >> > >> >Yes, one OSD will impact whole cluster. It will block write operations to the cluster > >> > >> Thanks for clarification. Really?? Is this(OSD 95%) full designed to > >> block write I/O of ceph cluster? > >> > > Really. > > To be more precise, any I/O that touches any PG on that OSD will block. > > So with a sufficiently large cluster you may have some, few, I/Os still go > > through as they don't use that OSD at all. > > > > That's why: > > > > 1. Ceph has the near-full warning (which of course may need to be > > adjusted to correctly reflect things, especially with smaller clusters). > > Once you get that warning, you NEED to take action immediately. > > > > 2. You want to graph the space utilization of all your OSDs with something > > like graphite. That allows you to spot trends of uneven data distribution > > early and thus react early to it. > > I re-weight (CRUSH re-weight, as this is permanent and my clusters aren't > > growing frequently) OSDs so they they are at least within 10% of each > > other. > > > > Christian > >> Because I have around 251 OSDs out which one OSD is 95% full, but > >> other 250 OSDs not in near full also... > >> > >> Thanks > >> Swami > >> > >> > >> On Tue, Jul 19, 2016 at 2:17 PM, Henrik Korkuc <lists@xxxxxxxxx> wrote: > >> > On 16-07-19 11:44, M Ranga Swami Reddy wrote: > >> >> > >> >> Hi, > >> >> Using ceph cluster with 100+ OSDs and cluster is filled with 60% data. > >> >> One of the OSD is 95% full. > >> >> If an OSD is 95% full, is it impact the any storage operation? Is this > >> >> impacts on VM/Instance? > >> > > >> > Yes, one OSD will impact whole cluster. It will block write operations to > >> > the cluster > >> >> > >> >> Immediately I have reduced the OSD weight, which was filled with 95 % > >> >> data. After re-weight, data rebalanaced and OSD came to normal state > >> >> (ie < 80%) with 1 hour time frame. > >> >> > >> >> > >> >> Thanks > >> >> Swami > >> >> _______________________________________________ > >> >> ceph-users mailing list > >> >> ceph-users@xxxxxxxxxxxxxx > >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > >> > > >> > > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@xxxxxxxxxxxxxx > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > > -- > > Christian Balzer Network/Systems Engineer > > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications > > http://www.gol.com/ > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |