I see the same behavior with the threshold of around 20M objects for 4 nodes, 16 OSDs, 32TB, hdd-based cluster. The issue dates back to hammer. Sent from my Windows 10 phone From: Blair Bethwaite Hi Wade, What IO are you seeing on the OSD devices when this happens (see e.g. iostat), are there short periods of high read IOPS where (almost) no writes occur? What does your memory usage look like (including slab)? Cheers, On 16 June 2016 at 22:14, Wade Holler <wade.holler@xxxxxxxxx> wrote: > Hi All, > > I have a repeatable condition when the object count in a pool gets to > 320-330 million the object write time dramatically and almost > instantly increases as much as 10X, exhibited by fs_apply_latency > going from 10ms to 100s of ms. > > Can someone point me in a direction / have an explanation ? > > I can add a new pool and it performs normally. > > Config is generally > 3 Nodes 24 physical core each, 768GB Ram each, 16 OSD / node , all SSD > with NVME for journals. Centos 7.2, XFS > > Jewell is the release; inserting objects with librados via some Python > test code. > > Best Regards > Wade > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- Cheers, ~Blairo _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
