Hi Somnath, Unfortunately I don't have any figures or really any way to generate them to answer that question. However my gut feeling is that it is heavily dependent on the type of workload. Faster Ghz will lower per request latency but only up to the point where the CPU's start getting busy. Once you get to that point, probably just having more total Ghz (ie cores x Ghz) is more important. But if you never generate a high enough queue depth, then you will never saturate all the cores and so will miss peak performance. In the example in that article, I was using a queue depth of 1 and so was heavily dependent on frequency. In a lot of the tests you have been doing at 256+ queue depth I would imagine having more total Ghz is better. Benchmarks aside, in the real world a cluster which is fairly idle but serving OLTP workloads would probably be better suited to having very fast clocked cores and scale out with more nodes or sockets. For more batch processing type workloads where combined throughput is more important that individual requests I would put my money on lots of slower cores. Of course the Turbo Boost is going to skew any testing you try and do, as even the 22 core monsters can scale individual cores up to ~3.5ghz at low loads. Obviously the problem with these is that they get slower turbo as you load them up, so your latency will rise more than expected as queue depth increases. Hmmm....I think I have confused myself now!! > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Somnath Roy > Sent: 20 January 2016 16:00 > To: Mark Nelson <mnelson@xxxxxxxxxx>; ceph-users@xxxxxxxxxxxxxx > Subject: Re: SSD OSDs - more Cores or more GHz > > Yes, thanks for the data.. > BTW, Nick, do we know what is more important more cpu core or more > frequency ? > For example, We have Xeon cpus available with a bit less frequency but with > more cores /socket , so, which one we should be going with for OSD servers > ? > > Thanks & Regards > Somnath > > -----Original Message----- > From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of > Mark Nelson > Sent: Wednesday, January 20, 2016 6:54 AM > To: ceph-users@xxxxxxxxxxxxxx > Subject: Re: SSD OSDs - more Cores or more GHz > > Excellent testing Nick! > > Mark > > On 01/20/2016 08:18 AM, Nick Fisk wrote: > > See this benchmark I did last year > > > > http://www.sys-pro.co.uk/ceph-storage-fast-cpus-ssd-performance/ > > > > > >> -----Original Message----- > >> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf > >> Of Oliver Dzombic > >> Sent: 20 January 2016 13:33 > >> To: ceph-users@xxxxxxxx > >> Subject: Re: SSD OSDs - more Cores or more GHz > >> > >> Hi, > >> > >> to be honest, i never made real benchmarks about that. > >> > >> But to me, i doubt that the higher frequency of a cpu will have a "real" > >> impact on ceph's performance. > >> > >> I mean, yes, mathematically, just like Wade pointed out, its true. > >>> frequency = < latency > >> > >> But when we compare CPU's of the same model, with different > frequencies. > >> > >> How much time ( in nano seconds ), do we save ? > >> I mean i have really no numbers here. > >> > >> But the difference between a 2,1 GHz and a 2,9 GHz ( Low End Xeon E5 > >> / High End Xeon E5 ) ( when it comes to delay in "memory/what ever" > >> allocation ), will be, inside an Linux OS, quiet small. And i mean > >> nano seconds tiny/non existing small. > >> But again, thats just my guess. Of course, if we talk about complete > >> different CPU Models ( E5 vs. I7 vs. AMD vs. what ever ) we will have > >> different 1st/2nd level Caches in CPU, different > Architecture/RAM/everything. > >> > >> But we are talking here about pure frequency issues. So we compare > >> identical CPU Models, just with different frequencies. > >> > >> And there, the difference, especially inside an OS and inside a > >> productive environment must be nearly not existing. > >> > >> I can not imagine how much an OSD / HDD needs to be hammered, that a > >> server is in general not totally overloaded and that the higher > >> frequency will make a measureable difference. > >> > >> ---- > >> > >> But again, i have here no numbers/benchmarks that could proove this > >> pure theory of mine. > >> > >> In the very end, more cores will usually mean more GHz frequency in > sum. > >> > >> So maybe the whole discussion is very theoretically, because usually > >> we wont run in a situation where we have to choose frequency vs. cores. > >> > >> Simply because more cores always means more frequency in sum. > >> > >> Except you compare totally different cpu models and generations, and > >> this is even more worst theoretically and maybe pointless since the > >> different cpu generations have totally different inner architecture > >> which has a great impact in overall performance ( aside from numbers of > frequency and cores ). > >> > >> -- > >> Mit freundlichen Gruessen / Best regards > >> > >> Oliver Dzombic > >> IP-Interactive > >> > >> mailto:info@xxxxxxxxxxxxxxxxx > >> > >> Anschrift: > >> > >> IP Interactive UG ( haftungsbeschraenkt ) Zum Sonnenberg 1-3 > >> 63571 Gelnhausen > >> > >> HRB 93402 beim Amtsgericht Hanau > >> Geschäftsführung: Oliver Dzombic > >> > >> Steuer Nr.: 35 236 3622 1 > >> UST ID: DE274086107 > >> > >> > >> Am 20.01.2016 um 14:14 schrieb Wade Holler: > >>> Great commentary. > >>> > >>> While it is fundamentally true that higher clock speed equals lower > >>> latency, I'm my practical experience we are more often interested in > >>> latency at the concurrency profile of the applications. > >>> > >>> So in this regard I favor more cores when I have to choose, such > >>> that we can support more concurrent operations at a queue depth of 0. > >>> > >>> Cheers > >>> Wade > >>> On Wed, Jan 20, 2016 at 7:58 AM Jan Schermer <jan@xxxxxxxxxxx > >>> <mailto:jan@xxxxxxxxxxx>> wrote: > >>> > >>> I'm using Ceph with all SSDs, I doubt you have to worry about speed > that > >>> much with HDD (it will be abysmall either way). > >>> With SSDs you need to start worrying about processor caches and > >> memory > >>> colocation in NUMA systems, linux scheduler is not really that smart > >>> right now. > >>> Yes, the process will get its own core, but it might be a different > >>> core every > >>> time it spins up, this increases latencies considerably if you start > >>> hammering > >>> the OSDs on the same host. > >>> > >>> But as always, YMMV ;-) > >>> > >>> Jan > >>> > >>> > >>> > On 20 Jan 2016, at 13:28, Oliver Dzombic <info@xxxxxxxxxxxxxxxxx > >>> <mailto:info@xxxxxxxxxxxxxxxxx>> wrote: > >>> > > >>> > Hi Jan, > >>> > > >>> > actually the linux kernel does this automatically anyway ( sending > new > >>> > processes to "empty/low used" cores ). > >>> > > >>> > A single scrubbing/recovery or what ever process wont take more > than > >>> > 100% CPU ( one core ) because technically this processes are not > >>> able to > >>> > run multi thread. > >>> > > >>> > Of course, if you configure your ceph to have ( up to ) 8 backfill > >>> > processes, then 8 processes will start, which can utilize ( up to ) 8 > >>> > CPU cores. > >>> > > >>> > But still, the single process wont be able to use more than one > >>> cpu core. > >>> > > >>> > --- > >>> > > >>> > In a situation where you have 2x E5-2620v3 for example, you have > 2x 6 > >>> > Cores x 2 HT Units = 24 Threads ( vCores ). > >>> > > >>> > So if you use inside such a system 24 OSD's every OSD will have ( > >>> > mathematically ) its "own" CPU Core automatically. > >>> > > >>> > Such a combination will perform better compared if you are using 1x > E5 > >>> > CPU with a much higher frequency ( but still the same amout of > >>> cores ). > >>> > > >>> > This kind of CPU's are so fast, that the physical HDD ( no matter if > >>> > SAS/SSD/ATA ) will not be able to overload the cpu ( no matter > >>> which cpu > >>> > you use of this kind ). > >>> > > >>> > Its like if you are playing games. If the game is running smooth, it > >>> > does not matter if its running on a 4 GHz machine on 40% > >>> utilization or > >>> > on a 2 GHz machine with 80% utilization. Is running smooth, it can > not > >>> > do better :-) > >>> > > >>> > So if your data is coming as fast as the HDD can physical deliver it, > >>> > its not important if the cpu runs with 2, 3, 4, 200 Ghz. Its > >>> already the > >>> > max of what the HDD can deliver. > >>> > > >>> > So as long as the HDD's dont get faster, the CPU's does not need to > be > >>> > faster. > >>> > > >>> > The Ceph storage is usually just delivering data, not running a > >>> > commercial webserver/what ever beside that. > >>> > > >>> > So if you are deciding what CPU you have to choose, you only have > to > >>> > think about how fast your HDD devices are. So that the CPU does > not > >>> > become the bottleneck. > >>> > > >>> > And the more cores you have, the lower is the chance, that > different > >>> > requests will block each other. > >>> > > >>> > ---- > >>> > > >>> > So all in all, Core > Frequency, always. ( As long as you use > >>> fast/up to > >>> > date CPUs ). If you are using old cpu's, of course you have to > >>> make sure > >>> > that the performance of the cpu ( which does by the way not only > >>> depend > >>> > on the frequency ) is sufficient that its not breaking the HDD > >>> data output. > >>> > > >>> > > >>> > > >>> > -- > >>> > Mit freundlichen Gruessen / Best regards > >>> > > >>> > Oliver Dzombic > >>> > IP-Interactive > >>> > > >>> > mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx> > >>> > > >>> > Anschrift: > >>> > > >>> > IP Interactive UG ( haftungsbeschraenkt ) > >>> > Zum Sonnenberg 1-3 > >>> > 63571 Gelnhausen > >>> > > >>> > HRB 93402 beim Amtsgericht Hanau > >>> > Geschäftsführung: Oliver Dzombic > >>> > > >>> > Steuer Nr.: 35 236 3622 1 > >>> > UST ID: DE274086107 > >>> > > >>> > > >>> > Am 20.01.2016 um 13:10 schrieb Jan Schermer: > >>> >> This is very true, but do you actually exclusively pin the cores > >>> to the OSD daemons so they don't interfere? > >>> >> I don't think may people do that, it wouldn't work with more than > >>> a handful of OSDs. > >>> >> The OSD might typicaly only need <100% of one core, but during > >>> startup or some reshuffling it's beneficial > >>> >> to allow it to get more (>400%), and that will interfere with > >>> whatever else was pinned there... > >>> >> > >>> >> Jan > >>> >> > >>> >>> On 20 Jan 2016, at 13:07, Oliver Dzombic <info@xxxxxxxxxxxxxxxxx > >>> <mailto:info@xxxxxxxxxxxxxxxxx>> wrote: > >>> >>> > >>> >>> Hi, > >>> >>> > >>> >>> Cores > Frequency > >>> >>> > >>> >>> If you think about recovery / scrubbing tasks its better when a > >>> cpu core > >>> >>> can be assigned to do this. > >>> >>> > >>> >>> Compared to a situation where the same cpu core needs to > >>> recovery/scrub > >>> >>> and still deliver the productive content at the same time. > >>> >>> > >>> >>> The more you can create a situation where an osd has its "own" > >>> cpu core, > >>> >>> the better it is. Modern CPU's are anyway so fast, that even > >>> SSDs cant > >>> >>> run the CPU's to their limit. > >>> >>> > >>> >>> -- > >>> >>> Mit freundlichen Gruessen / Best regards > >>> >>> > >>> >>> Oliver Dzombic > >>> >>> IP-Interactive > >>> >>> > >>> >>> mailto:info@xxxxxxxxxxxxxxxxx <mailto:info@xxxxxxxxxxxxxxxxx> > >>> >>> > >>> >>> Anschrift: > >>> >>> > >>> >>> IP Interactive UG ( haftungsbeschraenkt ) > >>> >>> Zum Sonnenberg 1-3 > >>> >>> 63571 Gelnhausen > >>> >>> > >>> >>> HRB 93402 beim Amtsgericht Hanau > >>> >>> Geschäftsführung: Oliver Dzombic > >>> >>> > >>> >>> Steuer Nr.: 35 236 3622 1 > >>> >>> UST ID: DE274086107 > >>> >>> > >>> >>> > >>> >>> Am 20.01.2016 um 10:01 schrieb Götz Reinicke - IT Koordinator: > >>> >>>> Hi folks, > >>> >>>> > >>> >>>> we plan to use more ssd OSDs in our first cluster layout > >>> instead of SAS > >>> >>>> osds. (more IO is needed than space) > >>> >>>> > >>> >>>> short question: What would influence the performance more? > >> more > >>> Cores or > >>> >>>> more GHz/Core. > >>> >>>> > >>> >>>> Or is it as always: Depeds on the total of > >>> OSDs/nodes/repl-level/etc ... :) > >>> >>>> > >>> >>>> If needed, I can give some more detailed information on the > layout. > >>> >>>> > >>> >>>> Thansk for feedback . Götz > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> _______________________________________________ > >>> >>>> ceph-users mailing list > >>> >>>> ceph-users@xxxxxxxxxxxxxx <mailto:ceph- > users@xxxxxxxxxxxxxx> > >>> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> >>>> > >>> >>> _______________________________________________ > >>> >>> ceph-users mailing list > >>> >>> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > >>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > _______________________________________________ > >>> > ceph-users mailing list > >>> > ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > >>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx> > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@xxxxxxxxxxxxxx > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@xxxxxxxxxxxxxx > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |