Question:
I'm curious if there is anybody else out there running CephFS at the scale I'm planning for. I'd like to know some of the issues you didn't expect that I should be looking out for. I'd also like to simply see when CephFS hasn't worked out and why. Basically, give me your war stories.Problem Details:
Now that I'm out of my design phase and finished testing on VMs, I'm ready to drop $100k on a pilo. I'd like to get some sense of confidence from the community that this is going to work before I pull the trigger.
I'm planning to replace my 110 disk 300TB (usable) Oracle ZFS 7320 with CephFS by this time next year (hopefully by December). My workload is a mix of small and vary large files (100GB+ in size). We do fMRI analysis on DICOM image sets as well as other physio data collected from subjects. We also have plenty of spreadsheets, scripts, etc. Currently 90% of our analysis is I/O bound and generally sequential.
In deploying Ceph, I am hoping to see more throughput than the 7320 can currently provide. I'm also looking to get away from traditional file-systems that require forklift upgrades. That's where Ceph really shines for us.
I don't have a total file count, but I do know that we have about 500k directories.
Planned Architecture:
Storage Interconnect:
Brocade VDX 6940 (40 gig)
Access Switches for clients (servers):
Brocade VDX 6740 (10 gig)
Access Switches for clients (workstations):
Brocade ICX 7450
3x MON:
128GB RAM
128GB RAM
2x 200GB SSD for OS
2x 400GB P3700 for LevelDB
2x E5-2660v4
1x Dual Port 40Gb Ethernet
2x MDS:
128GB RAM
2x 200GB SSD for OS
2x 400GB P3700 for LevelDB (is this necessary?)
2x E5-2660v4
1x Dual Port 40Gb Ethernet
8x OSD:
128GB RAM
2x 200GB SSD for OS
2x 400GB P3700 for Journals
24x 6TB Enterprise SATA
2x E5-2660v4
1x Dual Port 40Gb Ethernet
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
