Actually is very strange, since if i run the fio test on the client, and also un parallel run a iostat on all the OSD servers, i don't see any workload going on over the disks, I mean... nothing! 0.00....and also the fio script on the client is reacting very rare too: $ sudo fio --filename=/dev/rbd1 --direct=1 --rw=write --bs=4m --size=10G --iodepth=16 --ioengine=libaio --runtime=60 --group_reporting --name=file99 file99: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=16 fio-2.1.3 Starting 1 process Jobs: 1 (f=1): [W] [2.1% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 01h:26m:43s] It's seems like is doing nothing.. German Anders > --- Original message --- > Asunto: Re: [ceph-users] Performance really drops from 700MB/s to > 10MB/s > De: Mark Nelson <mark.nelson at inktank.com> > Para: <ceph-users at lists.ceph.com> > Fecha: Wednesday, 13/08/2014 11:00 > > On 08/13/2014 08:19 AM, German Anders wrote: >> >> Hi to all, >> >> I'm having a particular behavior on a new Ceph cluster. >> I've map >> a RBD to a client and issue some performance tests with fio, at this >> point everything goes just fine (also the results :) ), but then I try >> to run another new test on a new RBD on the same client, and suddenly >> the performance goes below 10MB/s and it took almost 10 minutes to >> complete a 10G file test, if I issue a *ceph -w* I don't see anything >> suspicious, any idea what can be happening here? > > When things are going fast, are your disks actually writing data out > as > fast as your client IO would indicate? (don't forgot to count > replication!) It may be that the great speed is just writing data > into > the tmpfs journals (if the test is only 10GB and spread across 36 > OSDs, > it could finish pretty quickly writing to tmpfs!). FWIW, tmpfs > journals > aren't very safe. It's not something you want to use outside of > testing > except in unusual circumstances. > > In your tests, when things are bad: it's generally worth checking to > see > if any one disk/osd is backed up relative to the others. There are a > couple of ways to accomplish this. the Ceph admin socket can tell you > information about each OSD ie how many outstanding IOs and a history > of > slow ops. You can also look at per-disk statistics with something > like > iostat or collectl. > > Hope this helps! > >> >> >> The cluster is made of: >> >> 3 x MON Servers >> 4 x OSD Servers (3TB SAS 6G disks for OSD daemons & tmpfs for Journal >> -> >> there's one tmpfs of 36GB that is share by 9 OSD daemons, on each >> server) >> 2 x Network SW (Cluster and Public) >> 10GbE speed on both networks >> >> The ceph.conf file is the following: >> >> [global] >> fsid = 56e56e4c-ea59-4157-8b98-acae109bebe1 >> mon_initial_members = cephmon01, cephmon02, cephmon03 >> mon_host = 10.97.10.1,10.97.10.2,10.97.10.3 >> auth_client_required = cephx >> auth_cluster_required = cephx >> auth_service_required = cephx >> filestore_xattr_use_omap = true >> public_network = 10.97.0.0/16 >> cluster_network = 192.168.10.0/24 >> osd_pool_default_size = 2 >> glance_api_version = 2 >> >> [mon] >> debug_optracker = 0 >> >> [mon.cephmon01] >> host = cephmon01 >> mon_addr = 10.97.10.1:6789 >> >> [mon.cephmon02] >> host = cephmon02 >> mon_addr = 10.97.10.2:6789 >> >> [mon.cephmon03] >> host = cephmon03 >> mon_addr = 10.97.10.3:6789 >> >> [osd] >> journal_dio = false >> osd_journal_size = 4096 >> fstype = btrfs >> debug_optracker = 0 >> >> [osd.0] >> host = cephosd01 >> devs = /dev/sdc1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.1] >> host = cephosd01 >> devs = /dev/sdd1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.2] >> host = cephosd01 >> devs = /dev/sdf1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.3] >> host = cephosd01 >> devs = /dev/sdg1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.4] >> host = cephosd01 >> devs = /dev/sdi1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.5] >> host = cephosd01 >> devs = /dev/sdj1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.6] >> host = cephosd01 >> devs = /dev/sdl1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.7] >> host = cephosd01 >> devs = /dev/sdm1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.8] >> host = cephosd01 >> devs = /dev/sdn1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.9] >> host = cephosd02 >> devs = /dev/sdc1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.10] >> host = cephosd02 >> devs = /dev/sdd1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.11] >> host = cephosd02 >> devs = /dev/sdf1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.12] >> host = cephosd02 >> devs = /dev/sdg1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.13] >> host = cephosd02 >> devs = /dev/sdi1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.14] >> host = cephosd02 >> devs = /dev/sdj1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.15] >> host = cephosd02 >> devs = /dev/sdl1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.16] >> host = cephosd02 >> devs = /dev/sdm1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.17] >> host = cephosd02 >> devs = /dev/sdn1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.18] >> host = cephosd03 >> devs = /dev/sdc1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.19] >> host = cephosd03 >> devs = /dev/sdd1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.20] >> host = cephosd03 >> devs = /dev/sdf1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.21] >> host = cephosd03 >> devs = /dev/sdg1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.22] >> host = cephosd03 >> devs = /dev/sdi1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.23] >> host = cephosd03 >> devs = /dev/sdj1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.24] >> host = cephosd03 >> devs = /dev/sdl1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.25] >> host = cephosd03 >> devs = /dev/sdm1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.26] >> host = cephosd03 >> devs = /dev/sdn1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.27] >> host = cephosd04 >> devs = /dev/sdc1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.28] >> host = cephosd04 >> devs = /dev/sdd1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.29] >> host = cephosd04 >> devs = /dev/sdf1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.30] >> host = cephosd04 >> devs = /dev/sdg1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.31] >> host = cephosd04 >> devs = /dev/sdi1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.32] >> host = cephosd04 >> devs = /dev/sdj1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.33] >> host = cephosd04 >> devs = /dev/sdl1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.34] >> host = cephosd04 >> devs = /dev/sdm1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [osd.35] >> host = cephosd04 >> devs = /dev/sdn1 >> osd_journal = /mnt/ramdisk/$cluster-$id-journal >> >> [client.volumes] >> keyring = /etc/ceph/ceph.client.volumes.keyring >> >> >> Thanks in advance, >> >> Best regards, >> >> *German Anders >> * >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users at lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > _______________________________________________ > ceph-users mailing list > ceph-users at lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.ceph.com/pipermail/ceph-users-ceph.com/attachments/20140813/a777484c/attachment.htm>
 |