Hi Wade, What IO are you seeing on the OSD devices when this happens (see e.g. iostat), are there short periods of high read IOPS where (almost) no writes occur? What does your memory usage look like (including slab)? Cheers, On 16 June 2016 at 22:14, Wade Holler <wade.holler@xxxxxxxxx> wrote: > Hi All, > > I have a repeatable condition when the object count in a pool gets to > 320-330 million the object write time dramatically and almost > instantly increases as much as 10X, exhibited by fs_apply_latency > going from 10ms to 100s of ms. > > Can someone point me in a direction / have an explanation ? > > I can add a new pool and it performs normally. > > Config is generally > 3 Nodes 24 physical core each, 768GB Ram each, 16 OSD / node , all SSD > with NVME for journals. Centos 7.2, XFS > > Jewell is the release; inserting objects with librados via some Python > test code. > > Best Regards > Wade > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Cheers, ~Blairo -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
