It might be worthwhile to check the BIOS settings on the two Rome servers to make sure the settings match, paying particular attention to NUMA and ioapic settings. Background: https://developer.amd.com/wp-content/resources/56745_0.80.pdf --Jeff On Wed, Jan 26, 2022 at 12:40 PM Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx> wrote: > > Both dual socket AMD Romes. Identical in every way. NUMAs per socket set to 1 in the BIOS. I'm using the exact same 10 drives on each system and they are PCIe Gen4 HPE OEM of SAMSUNG.... > > -----Original Message----- > From: Jani Partanen <jiipee@xxxxxxxxxxx> > Sent: Wednesday, January 26, 2022 3:32 PM > To: Finlayson, James M CIV (USA) <james.m.finlayson4.civ@xxxxxxxx>; linux-raid@xxxxxxxxxxxxxxx > Subject: [Non-DoD Source] Re: Showing my ignorance - kernel workers > > Hello, are both systems identical what comes to hardware? Mainly mobo. > > If no and they are dual socket systems, then it may be that one of the systems is designed to route all PCI-e via one socket so that all drive slots can be used just 1 socked populated. And another is designed so taht only half of the drive slots works when only 1 socket is populated. > At least I have read something like this previously from this list. > > // JiiPee > > > Finlayson, James M CIV (USA) kirjoitti 26/01/2022 klo 22.17: > > I apologize in advance if you can point me to something I can read about mdraid besides the source code. I'm beyond the bounds of my understanding of Linux. Background, I do a bunch of NUMA aware computing. I have two systems configured identically with a NUMA node 0 focused RAID5 LUN containing NUMA node 0 nvme drives and a NUMA node 1 focused RAID5 LUN identically configured. 9+1 nvme, 128KB stripe, xfs sitting on top, 64KB O_DIRECT reads from the application. > -- ------------------------------ Jeff Johnson Co-Founder Aeon Computing jeff.johnson@xxxxxxxxxxxxxxxxx www.aeoncomputing.com t: 858-412-3810 x1001 f: 858-412-3845 m: 619-204-9061 4170 Morena Boulevard, Suite C - San Diego, CA 92117 High-Performance Computing / Lustre Filesystems / Scale-out Storage
