do you try to restart osd to se the memory usage? On Fri, Oct 7, 2016 at 1:04 PM, David Burns <dburns@xxxxxxxxxxxxxx> wrote: > Hello all, > > We have a small 160TB Ceph cluster used only as a test s3 storage repository for media content. > > Problem > Since upgrading from Firefly to Hammer we are experiencing very high OSD memory use of 2-3 GB per TB of OSD storage - typical OSD memory 6-10GB. > We have had to increase swap space to bring the cluster to a basic functional state. Clearly this will significantly impact system performance and precludes starting all OSDs simultaneously. > > Hardware > 4 x storage nodes with 16 OSDs/node. OSD nodes are reasonable spec SMC storage servers with dual Xeon CPUs. Storage is 16 x 3TB SAS disks in each node. > Installed RAM is 72GB (2 nodes) & 80GB (2 nodes). (We note that the installed RAM is at least 50% higher than the Ceph recommended 1 GB RAM per TB of storage.) > > Software > OSD node OS is CentOS 6.8 (with updates). One node has been updated to CentOS 7.2 - no change in memory usage was observed. > > "ceph -v" -> ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90) > (all Ceph packages downloaded from download.ceph.com) > > The cluster has achieved status HEALTH_OK so we don’t believe this relates to increased memory due to recovery. > > History > Emperor 0.72.2 -> Firefly 0.80.10 -> Hammer 0.94.6 -> Hammer 0.94.7 -> Hammer 0.94.9 > > OSD per process memory is observed to increase substantially during load_pgs phase. > > Use of "ceph tell 'osd.*' heap release” has minimal effect - there is no substantial memory in the heap or cache freelists. > > More information can be found in bug #17228 (link http://tracker.ceph.com/issues/17228) > > Any feedback or guidance to further understanding the high memory usage would be welcomed. > > Thanks > > David > > > -- > FetchTV Pty Ltd, Level 5, 61 Lavender Street, Milsons Point, NSW 2061 > > <http://www.fetchtv.com.au/> > > This email is sent by FetchTV Pty Ltd (ABN 36 130 669 500). The contents of > this communication may be > confidential, legally privileged and/or copyright material. If you are not > the intended recipient, any use, > disclosure or copying of this communication is expressed prohibited. If you > have received this email in error, > please notify the sender and delete it immediately. > > <http://facebook.com/fetchtv> > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |