On Fri, Jul 6, 2018 at 8:38 AM, Matthew Stroud <mattstroud@xxxxxxxxxxxxx> wrote: > > Thanks for the reply. > > > > Actually we are using fiber channel (it’s so much more performant than iscsi in our tests) as the primary storage and this is serving up traffic for RBD for openstack, so this isn’t for backups. > > > > Our biggest bottle neck is trying utilize the host and/or osd process correctly. The disks are running at sub-millisecond, with about 90% of the IO being pulled from the array’s cache (a.k.a. not even hitting the disks). According to the host, we never get north of 20% disk utilization, unless there is a deep scrub going on. > > > > We have debated about putting the replica size to 2 instead of 3. However this isn’t much of a win for the purestorage which dedupes on the backend, so having copies of data are relatively free for that unit. 1 wouldn’t work because this is hosting a production work load. It is a mistake to use replica of 2 for production, when one of the copy is corrupted its hard to fix things. if you are concerned about storage overhead there is an option to use EC pools in luminous. To get back to your original question if you are comparing the network/disk utilization with FC numbers than that is wrong comparison, They are 2 different storage systems with different purposes, Ceph is scale out object storage system unlike FC systems where you can use commodity hardware and grow as you need, you generally dont need hba/fc enclosed disks but nothing stopping you from using your existing system. Also you generally dont need any raid mirroring configurations in the backend since ceph will handle the redundancy for you. scale out systems have more work to do than traditional FC systems. There are minimal configuration options for bluestore , what kind of disk/network utilization slowdown you are seeing? can you publish your numbers and test data? > > > Thanks, > > Matthew Stroud > > > > From: Maged Mokhtar <mmokhtar@xxxxxxxxxxx> > Date: Friday, July 6, 2018 at 7:01 AM > To: Matthew Stroud <mattstroud@xxxxxxxxxxxxx> > Cc: ceph-users <ceph-users@xxxxxxxxxxxxxx> > Subject: Re: Performance tuning for SAN SSD config > > > > > > On 2018-06-29 18:30, Matthew Stroud wrote: > > We back some of our ceph clusters with SAN SSD disk, particularly VSP G/F and Purestorage. I'm curious what are some settings we should look into modifying to take advantage of our SAN arrays. We had to manually set the class for the luns to SSD class which was a big improvement. However we still see situations where we get slow requests and the underlying disks and network are underutilized. > > > > More info about our setup. We are running centos 7 with Luminous as our ceph release. We have 4 osd nodes that have 5x2TB disks each and they are setup as bluestore. Our ceph.conf is attached with some information removed for security reasons. > > > > Thanks ahead of time. > > > > Thanks, > > Matthew Stroud > > > > ________________________________ > > > CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you. > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > If i understand correctly, you are using luns (via iSCSI) from your external SAN as OSDs and created a separate pool with these OSDs with device class SSD, you are using this pool for backup. > > Some comments: > > Using external disks as OSDs is probably not that common. It may be better to keep the SAN and Ceph cluster separate and have your backup tool access both, it will also be safer in case of disaster to the cluster your backup will be on a separate system. > What backup tool/script are you using ? it is better that this tool uses high queue depth, large block sizes and memory/page cache to increase performance during copies. > To try to pin down where your current bottleneck is, i would run benchmarks (eg fio) using the block sizes used by your backup tool on the raw luns before being added as OSDs (as pure iSCSI disks) as well as on both the main and backup pools. Have a resource tool (eg atop/systat/collectl) run during these tests to check for resources: disks %busy/cores %busy/io_wait > You probably can use replica count of 1 for the SAN OSDs since they include their own RAID redundancy. > > Maged > > > ________________________________ > > CONFIDENTIALITY NOTICE: This message is intended only for the use and review of the individual or entity to which it is addressed and may contain information that is privileged and confidential. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message solely to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone or return email. Thank you. > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |