On 19 Apr 2017 18:01, "Adam Carheden" <carheden@xxxxxxxx> wrote:
Does anyone know if XFS uses a single thread to write to it's journal?
You probably know this but just to avoid any confusion, the journal in this context isn't the metadata journaling in XFS, it's a separate journal written to by the OSD daemons
I think the number of threads per OSD is controlled by the 'osd op threads' setting which defaults to 2
I'm evaluating SSDs to buy as journal devices. I plan to have multiple
OSDs share a single SSD for journal.
I'm benchmarking several brands as
described here:
https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-tes t-if-your-ssd-is-suitable-as-a -journal-device/
It appears that sequential write speed using multiple threads varies
widely between brands. Here's what I have so far:
SanDisk SDSSDA240G, dd: 6.8 MB/s
SanDisk SDSSDA240G, fio 1 jobs: 6.7 MB/s
SanDisk SDSSDA240G, fio 2 jobs: 7.4 MB/s
SanDisk SDSSDA240G, fio 4 jobs: 7.5 MB/s
SanDisk SDSSDA240G, fio 8 jobs: 7.5 MB/s
SanDisk SDSSDA240G, fio 16 jobs: 7.5 MB/s
SanDisk SDSSDA240G, fio 32 jobs: 7.6 MB/s
SanDisk SDSSDA240G, fio 64 jobs: 7.6 MB/s
HFS250G32TND-N1A2A 30000P10, dd: 1.8 MB/s
HFS250G32TND-N1A2A 30000P10, fio 1 jobs: 4.8 MB/s
HFS250G32TND-N1A2A 30000P10, fio 2 jobs: 5.2 MB/s
HFS250G32TND-N1A2A 30000P10, fio 4 jobs: 9.5 MB/s
HFS250G32TND-N1A2A 30000P10, fio 8 jobs: 23.4 MB/s
HFS250G32TND-N1A2A 30000P10, fio 16 jobs: 7.2 MB/s
HFS250G32TND-N1A2A 30000P10, fio 32 jobs: 49.8 MB/s
HFS250G32TND-N1A2A 30000P10, fio 64 jobs: 70.5 MB/s
INTEL SSDSC2BB150G7, dd: 90.1 MB/s
INTEL SSDSC2BB150G7, fio 1 jobs: 91.0 MB/s
INTEL SSDSC2BB150G7, fio 2 jobs: 108.3 MB/s
INTEL SSDSC2BB150G7, fio 4 jobs: 134.2 MB/s
INTEL SSDSC2BB150G7, fio 8 jobs: 118.2 MB/s
INTEL SSDSC2BB150G7, fio 16 jobs: 39.9 MB/s
INTEL SSDSC2BB150G7, fio 32 jobs: 25.4 MB/s
INTEL SSDSC2BB150G7, fio 64 jobs: 15.8 MB/s
The SanDisk is slow, but speed is the same at any number of threads. The
Intel peaks at 4-6 threads and then declines rapidly into sub-par
performance (at least for a pricey "enterprise" drive). The SK Hynix is
slow at low numbers of threads but gets huge performance gains with more
threads. (This is all with one trial, but I have a script running
multiple trials across all drives today.)
I would avoid the SanDisk and Hynix. The s3500 isn't too bad. Perhaps consider going up to a 37xx and putting more OSDs on it. Of course with the caveat that you'll lose more OSDs if it goes down.
So if XFS has a single thread that does journaling, it looks like my
best option would be 1 intel SSD shared by 4-6 OSDs. If XFS already
throws multiple threads at the journal, then having OSDs share an Intel
drive will likely kill my SSD performance, but having as many OSDs as I
can cram in a chassis share the SK Hynix drive would get me great
performance for a fraction of the cost.
I don't think the hynix is going to give you great performance with multiple OSDs
Anyone have any related advice or experience to share regarding journal
SSD selection?
Need to know a bit more about the type of cluster you're planning to build, number of nodes, type of OSD, workload etc.
--
Adam Carheden
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
