I will verify, but I'm pretty sure they are still sitting with the same bios - I did an hpe ilo get of the one bios and pushed it to the other once I saw good individual SSD performance with FIO. I'm always fearful going out to these lists because there is much more that I don't know than I do, but at least my problems are different than "I'm running all of these desktop drives on a system with non-ECC memory and I just lost all of my movies, can you help me :)" -----Original Message----- From: Jeff Johnson <jeff.johnson@xxxxxxxxxxxxxxxxx> Sent: Wednesday, January 26, 2022 3:53 PM To: linux-raid@xxxxxxxxxxxxxxx Cc: Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> Subject: [URL Verdict: Neutral]Re: [Non-DoD Source] Re: Showing my ignorance - kernel workers All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ---- It might be worthwhile to check the BIOS settings on the two Rome servers to make sure the settings match, paying particular attention to NUMA and ioapic settings. Background: Caution-https://developer.amd.com/wp-content/resources/56745_0.80.pdf --Jeff On Wed, Jan 26, 2022 at 12:40 PM Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> wrote: > > Both dual socket AMD Romes. Identical in every way. NUMAs per socket set to 1 in the BIOS. I'm using the exact same 10 drives on each system and they are PCIe Gen4 HPE OEM of SAMSUNG.... > > -----Original Message----- > From: Jani Partanen <jiipee@xxxxxxxxxxx> > Sent: Wednesday, January 26, 2022 3:32 PM > To: Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx>; > linux-raid@xxxxxxxxxxxxxxx > Subject: [Non-DoD Source] Re: Showing my ignorance - kernel workers > > Hello, are both systems identical what comes to hardware? Mainly mobo. > > If no and they are dual socket systems, then it may be that one of the systems is designed to route all PCI-e via one socket so that all drive slots can be used just 1 socked populated. And another is designed so taht only half of the drive slots works when only 1 socket is populated. > At least I have read something like this previously from this list. > > // JiiPee > > > Finlayson, James M CIV (USA) kirjoitti 26/01/2022 klo 22.17: > > I apologize in advance if you can point me to something I can read about mdraid besides the source code. I'm beyond the bounds of my understanding of Linux. Background, I do a bunch of NUMA aware computing. I have two systems configured identically with a NUMA node 0 focused RAID5 LUN containing NUMA node 0 nvme drives and a NUMA node 1 focused RAID5 LUN identically configured. 9+1 nvme, 128KB stripe, xfs sitting on top, 64KB O_DIRECT reads from the application. > -- ------------------------------ Jeff Johnson Co-Founder Aeon Computing jeff.johnson@xxxxxxxxxxxxxxxxx Caution-www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage
