Thanks for the correction...so even one OSD reaches to 95% full, the total ceph cluster IO (R/W) will be blocked...Ideally read IO should work... Thanks Swami On Tue, Jul 19, 2016 at 3:41 PM, Wido den Hollander <wido@xxxxxxxx> wrote: > >> Op 19 juli 2016 om 11:55 schreef M Ranga Swami Reddy <swamireddy@xxxxxxxxx>: >> >> >> Thanks for detail... >> When an OSD is 95% full, then that specific OSD's write IO blocked. >> > > No, the *whole* cluster will block. In the OSDMap the flag 'full' is set which causes all I/O to stop (even read!) until you make sure the OSD drops below 95%. > > Wido > >> Thanks >> Swami >> >> On Tue, Jul 19, 2016 at 3:07 PM, Christian Balzer <chibi@xxxxxxx> wrote: >> > >> > Hello, >> > >> > On Tue, 19 Jul 2016 14:23:32 +0530 M Ranga Swami Reddy wrote: >> > >> >> >> Using ceph cluster with 100+ OSDs and cluster is filled with 60% data. >> >> >> One of the OSD is 95% full. >> >> >> If an OSD is 95% full, is it impact the any storage operation? Is this >> >> >> impacts on VM/Instance? >> >> >> >> >Yes, one OSD will impact whole cluster. It will block write operations to the cluster >> >> >> >> Thanks for clarification. Really?? Is this(OSD 95%) full designed to >> >> block write I/O of ceph cluster? >> >> >> > Really. >> > To be more precise, any I/O that touches any PG on that OSD will block. >> > So with a sufficiently large cluster you may have some, few, I/Os still go >> > through as they don't use that OSD at all. >> > >> > That's why: >> > >> > 1. Ceph has the near-full warning (which of course may need to be >> > adjusted to correctly reflect things, especially with smaller clusters). >> > Once you get that warning, you NEED to take action immediately. >> > >> > 2. You want to graph the space utilization of all your OSDs with something >> > like graphite. That allows you to spot trends of uneven data distribution >> > early and thus react early to it. >> > I re-weight (CRUSH re-weight, as this is permanent and my clusters aren't >> > growing frequently) OSDs so they they are at least within 10% of each >> > other. >> > >> > Christian >> >> Because I have around 251 OSDs out which one OSD is 95% full, but >> >> other 250 OSDs not in near full also... >> >> >> >> Thanks >> >> Swami >> >> >> >> >> >> On Tue, Jul 19, 2016 at 2:17 PM, Henrik Korkuc <lists@xxxxxxxxx> wrote: >> >> > On 16-07-19 11:44, M Ranga Swami Reddy wrote: >> >> >> >> >> >> Hi, >> >> >> Using ceph cluster with 100+ OSDs and cluster is filled with 60% data. >> >> >> One of the OSD is 95% full. >> >> >> If an OSD is 95% full, is it impact the any storage operation? Is this >> >> >> impacts on VM/Instance? >> >> > >> >> > Yes, one OSD will impact whole cluster. It will block write operations to >> >> > the cluster >> >> >> >> >> >> Immediately I have reduced the OSD weight, which was filled with 95 % >> >> >> data. After re-weight, data rebalanaced and OSD came to normal state >> >> >> (ie < 80%) with 1 hour time frame. >> >> >> >> >> >> >> >> >> Thanks >> >> >> Swami >> >> >> _______________________________________________ >> >> >> ceph-users mailing list >> >> >> ceph-users@xxxxxxxxxxxxxx >> >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > >> >> > >> >> > >> >> > _______________________________________________ >> >> > ceph-users mailing list >> >> > ceph-users@xxxxxxxxxxxxxx >> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> _______________________________________________ >> >> ceph-users mailing list >> >> ceph-users@xxxxxxxxxxxxxx >> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> > >> > >> > -- >> > Christian Balzer Network/Systems Engineer >> > chibi@xxxxxxx Global OnLine Japan/Rakuten Communications >> > http://www.gol.com/ >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |