Hi,
On 08/19/2015 11:01 AM, Christian Balzer wrote:
Hello,
That's a pretty small cluster all things considered, so your rather
intensive test setup is likely to run into any or all of the following
issues:
1) The amount of data you're moving around is going cause a lot of
promotions from and to the cache tier. This is expensive and slow.
2) EC coded pools are slow. You may have actually better results with a
"Ceph classic" approach, 2-4 HDDs per journal SSD. Also 6TB HDDs combined
with EC may look nice to you from a cost/density prospect, but more HDDs
means more IOPS and thus speed.
3) scrubbing (unless configured very aggressively down) will
impact your performance on top of the items above.
4) You already noted the kernel versus userland bit.
5) Having all your storage in a single JBOD chassis strikes me as ill
advised, though I don't think it's an actual bottleneck at 4x12Gb/s.
We use two of these (I forgot to mention that)
Each chasis has two internal controllers - both exposing all the disks
to the connected hosts. There are two osd nodes connected to each chasis.
When you ran the fio tests I assume nothing else was going on and the
dataset size would have fit easily into the cache pool, right?
Look at your nodes with atop or iostat, I venture all your HDDs are at
100%.
Christian
Yes, the problem was a full cache pool. I'm currently wondering on how
to tune the cache pool parameters so that the whole cluster doesn't slow
down that much when the cache is full...
I'm thinking of doing some tests on a pool w/o the cache tier so I can
compare the results. Any suggestions would be greatly appreciated..
J
--
Jacek Jarosiewicz
Administrator Systemów Informatycznych
----------------------------------------------------------------------------------------
SUPERMEDIA Sp. z o.o. z siedzibą w Warszawie
ul. Senatorska 13/15, 00-075 Warszawa
Sąd Rejonowy dla m.st.Warszawy, XII Wydział Gospodarczy Krajowego
Rejestru Sądowego,
nr KRS 0000029537; kapitał zakładowy 42.756.000 zł
NIP: 957-05-49-503
Adres korespondencyjny: ul. Jubilerska 10, 04-190 Warszawa
----------------------------------------------------------------------------------------
SUPERMEDIA -> http://www.supermedia.pl
dostep do internetu - hosting - kolokacja - lacza - telefonia
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
