On 08/08/2014 03:44 PM, Dan Van Der Ster wrote: > Hi, > Here?s what we do to identify our top RBD users. > > First, enable log level 10 for the filestore so you can see all the IOs > coming from the VMs. Then use a script like this (used on a dumpling > cluster): > > https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl > > to summarize the osd logs and identify the top clients. > > Then its just a matter of scripting to figure out the ops/sec per > volume, but for us at least the main use-case has been to identify who > is responsible for a new peak in overall ops ? and daily-granular > statistics from the above script tends to suffice. > > BTW, do you throttle your clients? We found that its absolutely > necessary, since without a throttle just a few active VMs can eat up the > entire iops capacity of the cluster. +1 I'd strongly advise to set I/O limits for Instances. I've had multiple occasions where a runaway script inside a VM was hammering on the underlying storage killing all I/O. Not only with Ceph, but over the many years I've worked with storage. I/O == expensive CloudStack supports I/O limiting, so I recommend you set a limit. Set it to 750 write IOps for example. That way one Instance can't kill the whole cluster, but it still has enough I/O to run. (usually). Wido > > Cheers, Dan > > -- Dan van der Ster || Data & Storage Services || CERN IT Department -- > > > On 08 Aug 2014, at 13:51, Andrija Panic <andrija.panic at gmail.com > <mailto:andrija.panic at gmail.com>> wrote: > >> Hi, >> >> we just had some new clients, and have suffered very big degradation >> in CEPH performance for some reasons (we are using CloudStack). >> >> I'm wondering if there is way to monitor OP/s or similar usage by >> client connected, so we can isolate the heavy client ? >> >> Also, what is the general best practice to monitor these kind of >> changes in CEPH ? I'm talking about R/W or OP/s change or similar... >> >> Thanks, >> -- >> >> Andrija Pani? >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com <mailto:ceph-users at lists.ceph.com> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on
 |