Re: Recommendations for sharing a file system to a heterogeneous client network?

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



 
 
I opened a thread recently here asking about what can be generally accepted as 'ceph overhead' when using the file system. I wonder if the performance loss I have on a cephfs 1x replication pool compared to native performance is really so much. 5,6x to 2x slower than native disk performance
 
 
 

4k r ran.

4k w ran.

4k r seq.

4k w seq.

1024k r ran.

1024k w ran.

1024k r seq.

1024k w seq.

size

lat

iops

kB/s

lat

iops

kB/s

lat

iops

MB/s

lat

iops

MB/s

lat

iops

MB/s

lat

iops

MB/s

lat

iops

MB/s

lat

iops

MB/s

Cephfs

ssd rep. 3

2.78

1781

7297

1.42

700

2871

0.29

3314

13.6

0.04

889

3.64

4.3

231

243

0.08

132

139

4.23

235

247

6.99

142

150

Cephfs

ssd rep. 1

0.54

1809

7412

0.8

1238

5071

0.29

3325

13.6

0.56

1761

7.21

4.27

233

245

4.34

229

241

4.21

236

248

4.34

229

241

Samsung

MZK7KM480

480GB

0.09

10.2k

41600

0.05

17.9k

73200

0.05

18k

77.6

0.05

18.3k

75.1

2.06

482

506

2.16

460

483

1.98

502

527

2.13

466

489


(4 nodes, CentOS7, luminous)


   From: Maged Mokhtar [mailto:mmokhtar@xxxxxxxxxxx]
Sent: 15 January 2019 22:55
To: Ketil Froyn; ceph-users@xxxxxxxxxxxxxx
Subject: Re: Recommendations for sharing a file system to a heterogeneous client network?


Hi Ketil,

I have not tested the creation/deletion but the read/write performance was much better then the link you posted. Using CTDB setup based on Robert's presentation, we were getting 800 MB/s write performance for queue depth =1 and  2.2 GB/s  queue depth= 32  from a single CTDB/Samba gateway. For the QD=32 test we used 2 Windows clients to the same gateway (to avoid limitation from the Windows side). Tests were done using Microsoft diskspd tool at 4M blocks with cache off.  Gateway had 2x40 G nics : one for Windows network the other for CephFS client, each was doing 20 Gbps (50% utilization) cpu was 24 cores running at 85% utilization taken by the smbd process. We used Ubuntu 16.04 CTDB/Samba with a SUSE SLE15 kernel for kernel client. Ceph was Luminous 12.2.7.

Maged


On 15/01/2019 22:04, Ketil Froyn wrote:
Robert,

Thanks, this is really interesting. Do you also have any details on how a solution like this performs? I've been reading a thread about samba/cephfs performance, and the stats aren't great - especially when creating/deleting many files - but being a rookie, I'm not 100% clear on the hardware differences being benchmarked in the mentioned test.


Regards, Ketil

On Tue, Jan 15, 2019, 16:38 Robert Sander <r.sander@xxxxxxxxxxxxxxxxxxx wrote:
Hi Ketil,

use Samba/CIFS with multiple gateway machines clustered with CTDB.
CephFS can be mounted with Posix ACL support.

Slides from my last Ceph day talk are available here:
https://www.slideshare.net/Inktank_Ceph/ceph-day-berlin-unlimited-fileserver-with-samba-ctdb-and-cephfs

Regards
--
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 93818 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
-- 
Maged Mokhtar
CEO PetaSAN
4 Emad El Deen Kamel
Cairo 11371, Egypt
www.petasan.org
+201006979931
skype: maged.mokhtar

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux