Ok. Of the 202 pgs on this example OSD, 65 of them have around ~160k files 137 ( the rest ) -have around ~80k files On Thu, Jun 16, 2016 at 10:47 AM, Wade Holler <wade.holler@xxxxxxxxx> wrote: > Wido, > > I am walking an Example OSD now and counting the files. 3096 PGs for this pool. > So far file counts inside the pool.pg_head directories are all coming > in around ~80k. > > Is this an issue ? > > I will report back with all pg_head file counts in this example OSD > once it finishes. > > Best Regards, > Wade > > > On Thu, Jun 16, 2016 at 9:38 AM, Wido den Hollander <wido@xxxxxxxx> wrote: >> >>> Op 16 juni 2016 om 14:14 schreef Wade Holler <wade.holler@xxxxxxxxx>: >>> >>> >>> Hi All, >>> >>> I have a repeatable condition when the object count in a pool gets to >>> 320-330 million the object write time dramatically and almost >>> instantly increases as much as 10X, exhibited by fs_apply_latency >>> going from 10ms to 100s of ms. >>> >> >> My first guess is the filestore splitting and the amount of files per directory. >> >> You have 3*16=48 OSDs, is that correct? With roughly 100 PGs per OSD you have let's say 4800 PGs in total? >> >> That means you have ~66k objects per PG. >> >>> Can someone point me in a direction / have an explanation ? >> >> If you take a look at one of the OSDs, are there a huge amount of files in a single directory? Look inside the 'current' directory on that OSDs. >> >> Wido >> >>> >>> I can add a new pool and it performs normally. >>> >>> Config is generally >>> 3 Nodes 24 physical core each, 768GB Ram each, 16 OSD / node , all SSD >>> with NVME for journals. Centos 7.2, XFS >>> >>> Jewell is the release; inserting objects with librados via some Python >>> test code. >>> >>> Best Regards >>> Wade >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
