Hi, I am just wondering if anyone has any thoughts on the questions below...I would like to order some additional hardware ASAP...and the order that I place may change depending on the feedback that I receive. Thanks again, Shain Sent from my iPhone > On Jan 9, 2015, at 2:45 PM, Shain Miley <smiley@xxxxxxx> wrote: > > Although it seems like having a regularly scheduled cron job to do a recursive directory listing may be ok for us as a bit of a work around...I am still in the processes of trying to improve performance. > > A few other questions have come up as a result. > > a)I am in the process of looking at specs for a new rbd 'headnode' that will be used to mount our 100TB rbd image. At some point in the future we may look into the performance, and multi client access that cephfs could offer...is there any reason that I would not be able to use this new server as both an rbd client and an mds server (assuming the hardware is good enough)? I know that some cluster functions should not and cannot be mixed on the same server...is this by any chance one of them? > > b)Currently the 100TB rbd image is acting as one large repository for our archive....this will only grow over time. I understand that ceph is pool based...however I am wondering if I would somehow see any better per rbd image performance...if for example...instead of having 1 x 100TB rbd image...I had 4 x 25TB rbd images (since we really could split these up based on our internal groups). > > c)Would adding a few ssd drives (in the right quantity) to each node help out with reads as well as writes? > > d)I am a bit confused about how to enable the rbd cache option on the client...is this change something that only needs to be made to the ceph.conf file on the rbd kernel client server...or do the mds and osd servers need the ceph.conf file modified as well and their services restarted? > > Other options that I might be looking into going forward are moving some of this data (the data actually needed by our php apps) to rgw...although that option adds some more complexity and unfamiliarity for our users. > > Thanks again for all the help so far. > > Shain > >> On 01/07/2015 03:40 PM, Shain Miley wrote: >> Just to follow up on this thread, the main reason that the rbd directory listing latency was an issue for us, was that we were seeing a large amount of IO delay in a PHP app that reads from that rbd image. >> >> It occurred to me (based on Roberts cache_dir suggestion below) that maybe doing a recursive find or a recursive directory listing inside the one folder in question might speed things up. >> >> After doing the recursive find...the directory listing seems much faster and the responsiveness of the PHP app has increased as well. >> >> Hopefully nothing else will need to be done here, however it seems that worst case...a daily or weekly cronjob that traverses the directory tree in that folder might be all we need. >> >> Thanks again for all the help. >> >> Shain >> >> >> >> Shain Miley | Manager of Systems and Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649 >> >> ________________________________________ >> From: ceph-users [ceph-users-bounces@xxxxxxxxxxxxxx] on behalf of Shain Miley [SMiley@xxxxxxx] >> Sent: Tuesday, January 06, 2015 8:16 PM >> To: Christian Balzer; ceph-users@xxxxxxxx >> Subject: Re: rbd directory listing performance issues >> >> Christian, >> >> Each of the OSD's server nodes are running on Dell R-720xd's with 64 GB or RAM. >> >> We have 107 OSD's so I have not checked all of them..however the ones I have checked with xfs_db, have shown anywhere from 1% to 4% fragmentation. >> >> I'll try to upgrade the client server to 32 or 64 GB of ram at some point soon...however at this point all the tuning that I have done has not yielded all that much in terms of results. >> >> It maybe a simple fact that I need to look into adding some SSD's, and the overall bottleneck here are the 4TB 7200 rpm disks we are using. >> >> In general, when looking at the graphs in Calamari, we see around 20ms latency (await) for our OSD's however there are lots of times where we see (via the graphs) spikes of 250ms to 400ms as well. >> >> Thanks again, >> >> Shain >> >> >> Shain Miley | Manager of Systems and Infrastructure, Digital Media | smiley@xxxxxxx | 202.513.3649 >> >> ________________________________________ >> From: Christian Balzer [chibi@xxxxxxx] >> Sent: Tuesday, January 06, 2015 7:34 PM >> To: ceph-users@xxxxxxxx >> Cc: Shain Miley >> Subject: Re: rbd directory listing performance issues >> >> Hello, >> >>> On Tue, 6 Jan 2015 15:29:50 +0000 Shain Miley wrote: >>> >>> Hello, >>> >>> We currently have a 12 node (3 monitor+9 OSD) ceph cluster, made up of >>> 107 x 4TB drives formatted with xfs. The cluster is running ceph version >>> 0.80.7: >> I assume journals on the same HDD then. >> >> How much memory per node? >> >> [snip] >>> A while back I created an 80 TB rbd image to be used as an archive >>> repository for some of our audio and video files. We are still seeing >>> good rados and rbd read and write throughput performance, however we >>> seem to be having quite a long delay in response times when we try to >>> list out the files in directories with a large number of folders, files, >>> etc. >>> >>> Subsequent directory listing times seem to run a lot faster (but I am >>> not sure for long that is the case before we see another instance of >>> slowness), however the initial directory listings can take 20 to 45 >>> seconds. >> Basically the same thing(s) that Robert said. >> How big is "large"? >> How much memory on the machine you're mounting this image? >> Ah, never mind, just saw your follow-up. >> >> Definitely add memory to this machine if you can. >> >> The initial listing is always going to be slow-ish of sorts depending on >> a number of things in the cluster. >> >> As in, how busy is it (IOPS)? With journals on disk your HDDs are going to >> be sluggish individually and your directory information might reside >> mostly in one object (on one OSD), thus limiting you to the speed of that >> particular disk. >> >> And this is also where the memory of your storage nodes comes in, if it is >> large enough your "hot" objects will get cached there as well. >> To see if that's the case (at least temporarily), drop the caches on all >> of your storage nodes (echo 3 > /proc/sys/vm/drop_caches), mount your >> image, do the "ls -l" until it's "fast", umount it, mount it again and do >> the listing again. >> In theory, unless your cluster is extremely busy or your storage node have >> very little pagecache, the re-mounted image should get all the info it >> needs from said pagecache on your storage nodes, never having to go to the >> actual OSD disks and thus be fast(er) than the initial test. >> >> Finally to potentially improve the initial scan that has to come from the >> disks obviously, see how fragmented your OSDs are and depending on the >> results defrag them. >> >> Christian >> -- >> Christian Balzer Network/Systems Engineer >> chibi@xxxxxxx Global OnLine Japan/Fusion Communications >> http://www.gol.com/ >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |