I'll turn HT off and see if it gives any boost. Couple of extra notes : - Yes turbo was on. - Got my hands on a Xeon W-2155. Same drives, same hba, same kernel and got close to 2GiB/s. Overall turbo frequency, especially AVX512, on this cpu is much higher. - As Roy mentioned, I did try the same exercise with four NVMe drives and got 1.2 GiB/s on a single core but in this case group_thread_cnt at 2 gave me somewhere close to 2.4 GiB/s which I think is close to the limit. - With a second identical RAID6, the raid controller could achieve 2 * 1.8 GiB/s on the same exercise. JDG -----Original Message----- From: Roger Heflin [mailto:rogerheflin@xxxxxxxxx] Sent: mercredi 20 février 2019 02:02 To: Roy Sigurd Karlsbakk <roy@xxxxxxxxxxxxx> Cc: Jean De Gyns <Jean.DeGyns@xxxxxxxxxx>; Song Liu <liu.song.a23@xxxxxxxxx>; Wilson Jonathan <i400sjon@xxxxxxxxx>; Linux Raid <linux-raid@xxxxxxxxxxxxxxx> Subject: Re: RAID6 : Sequential Write Performance The raid card likely has a custom asic for a lot of the calculations related to raid. Custom designed for a specific purpose hardware is hard to beat in simple cases like this. XOR hardware is pretty easy to build so they could have several XOR units pipelined to speed up the process. As you mentioned keeping it all on the same socket but different non-ht cores is probably best. And is also probably best to use the socket with the raid card on it, but make sure each of the raid cards interrupts are isolated and not using the same core(s) the raid processes are using. And if the kernel being used has interrupt clustering either get on with it removed (newer ones) or use the interrupt remap option with it disabled to it disables the cluster interrupt. The cluster interrupt seems to defeat irqbalancing either manually or by irqbalance, and you may end up with several interrupts on the same core and not moving. On Tue, Feb 19, 2019 at 6:42 PM Roy Sigurd Karlsbakk <roy@xxxxxxxxxxxxx> wrote: > > I'm not sure what it takes to make the raid code fully multithreaded, but it should help on this occation. Perhaps it would be sufficient to move the interrupts to a single socket, but I think he tried that - I was asking him because of the load of CPU interconnects and the lower bandwidth compared to main memory. > > Still, a RAID controller has a slower CPU and less memory bandwidth, so it really should be slower, but isn't. I'm not quite sure why this is the case. > > Vennlig hilsen > > roy > -- > Roy Sigurd Karlsbakk > (+47) 98013356 > http://blogg.karlsbakk.net/ > GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt > -- > Hið góða skaltu í stein höggva, hið illa í snjó rita. > > ----- Original Message ----- > > From: "Roger Heflin" <rogerheflin@xxxxxxxxx> > > To: "Roy Sigurd Karlsbakk" <roy@xxxxxxxxxxxxx> > > Cc: "Jean De Gyns" <Jean.DeGyns@xxxxxxxxxx>, "Song Liu" <liu.song.a23@xxxxxxxxx>, "Wilson Jonathan" > > <i400sjon@xxxxxxxxx>, "Linux Raid" <linux-raid@xxxxxxxxxxxxxxx> > > Sent: Wednesday, 20 February, 2019 01:26:07 > > Subject: Re: RAID6 : Sequential Write Performance > > > One though I have is if you can figure out which core is being maxed > > out then turn off its hyperthread. On the single cpu speed > > benchmarks I have ran turning off the shared idle hyperthread gains > > several % on cpu bound benchmarks. If the hyperthread is being > > actively used by some other process then it should gain quite a bit > > more. You may also want to run turbostat and verify that the > > processor in question is running full speed and/or turboboosting (it > > probably is if everything is working right). > > > > On Tue, Feb 19, 2019 at 5:33 PM Roy Sigurd Karlsbakk <roy@xxxxxxxxxxxxx> wrote: > >> > >> > Is this process going to be running all of the time, or just run for a > >> > few minutes/hours at a time? If for a few minutes at a time you may > >> > want to use a raid0 ssd array for the data collection and then have > >> > another process to move that data onto the raid6. If the data is > >> > split multiple files (a few gb each) then the backend process can > >> > move the finished files just behind the main process and if the > >> > files are small enough then they will still be in file cache and > >> > you won't have to read off of the ssd array, and you will be > >> > isolated from random blips on the spinning disks. Give you are > >> > talking about 12 disks, if one of those spinning disk blips it > >> > will be whatever timeout you set on the disk assuming you have a > >> > disk that allows the timeout to be set (the lowest I have found for that timeout is 0.1seconds). > >> > >> He did use SSDs to check this, with RAID-6, and the performance > >> issue was the same, CPU-bound > >> > >> Vennlig hilsen > >> > >> roy > >> -- > >> Roy Sigurd Karlsbakk > >> (+47) 98013356 > >> http://blogg.karlsbakk.net/ > >> GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt > >> -- > >> Hið góða skaltu í stein höggva, hið illa í snjó rita.
