1. How much ram has each node 2. do they have fibre attached storage, and which, how many disks, raid-controller cache size, or is it local disks, and again, what raid controller and how many disks are you using? 3. when running tiotest, does it does it depend from where you run it on what performance you see? Do those four threads run one on each node or is it one node running four threads? My first reaction to the numbers is: 2500MB/s sounds like it's not coming from disk but from RAM. Michael -----Original Message----- From: linux-cluster-bounces@xxxxxxxxxx [mailto:linux-cluster-bounces@xxxxxxxxxx] On Behalf Of Patrick Caulfield Sent: Thursday, February 09, 2006 6:18 AM To: linux clustering Subject: Re: Node lag Frank Schliefer wrote: > Hi, > > after setting up an four node cluster we have one node that is way > slower than the other 3 nodes. > > We using eg. tiotest for benchmarking the GFS. > > Normal Node: > Tiotest results for 4 concurrent io threads: > ,----------------------------------------------------------------------. > | Item | Time | Rate | Usr CPU | Sys CPU | > +-----------------------+----------+--------------+----------+---------+ > | Write 40 MBs | 0.2 s | 227.426 MB/s | 36.4 % | 384.4 % | > | Random Write 16 MBs | 0.1 s | 143.405 MB/s | 58.7 % | 146.9 % | > | Read 40 MBs | 0.0 s | 2558.199 MB/s | 307.0 % | 1228.0 % | > | Random Read 16 MBs | 0.0 s | 2685.169 MB/s | 550.0 % | 1374.9 % | > `----------------------------------------------------------------------' > > > Slow Node: > Tiotest results for 4 concurrent io threads: > ,----------------------------------------------------------------------. > | Item | Time | Rate | Usr CPU | Sys CPU | > +-----------------------+----------+--------------+----------+---------+ > | Write 40 MBs | 1.4 s | 27.687 MB/s | 2.2 % | 121.8 % | > | Random Write 16 MBs | 4.2 s | 3.695 MB/s | 0.0 % | 7.9 % | > | Read 40 MBs | 0.0 s | 2228.288 MB/s | 89.1 % | 1337.1 % | > | Random Read 16 MBs | 0.0 s | 2252.739 MB/s | 230.7 % | 692.1 % | > `----------------------------------------------------------------------' > > any hints why this could happen ?? > > Using kernel 2.6.15.2 (sorry no RH) It would be helpful if you could give us more information about your installation: disk topology, lock manager in use (and which nodes are lockservers if using GULM) and whether it matters which nodes are started first or not. -- patrick -- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster -- Linux-cluster@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/linux-cluster
