Re: OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

I can see ~17% hardware interrupts which I find a little high - can you make sure all load is spread over all your cores (/proc/interrupts)?

What about disk util once you restart them? Are they all 100% utilized or is it 'only' mostly cpu-bound?

Also you're running a monitor on this node - how is the load on the nodes where you run a monitor compared to those where you dont?

Cheers,
Martin


On Thu, Mar 20, 2014 at 10:18 AM, Quenten Grasso <qgrasso@xxxxxxxxxx> wrote:

Hi All,

 

I left out my OS/kernel version, Ubuntu 12.04.4 LTS w/ Kernel 3.10.33-031033-generic (We upgrade our kernels to 3.10 due to Dell Drivers).

 

Here’s an example of starting all the OSD’s after a reboot.

 

top - 09:10:51 up 2 min,  1 user,  load average: 332.93, 112.28, 39.96

Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie

Cpu(s): 50.3%us, 32.5%sy,  0.0%ni,  0.0%id,  0.0%wa, 17.2%hi,  0.0%si,  0.0%st

Mem:  32917276k total,  6331224k used, 26586052k free,     1332k buffers

Swap: 33496060k total,        0k used, 33496060k free,  1474084k cached

 

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

15875 root      20   0  910m 381m  50m S   60  1.2   0:50.57 ceph-osd

2996 root      20   0  867m 330m  44m S   59  1.0   0:58.32 ceph-osd

4502 root      20   0  907m 372m  47m S   58  1.2   0:55.14 ceph-osd

12465 root      20   0  949m 418m  55m S   58  1.3   0:51.79 ceph-osd

4171 root      20   0  886m 348m  45m S   57  1.1   0:56.17 ceph-osd

3707 root      20   0  941m 405m  50m S   57  1.3   0:59.68 ceph-osd

3560 root      20   0  924m 394m  51m S   56  1.2   0:59.37 ceph-osd

4318 root      20   0  965m 435m  55m S   56  1.4   0:54.80 ceph-osd

3337 root      20   0  935m 407m  51m S   56  1.3   1:01.96 ceph-osd

3854 root      20   0  897m 366m  48m S   55  1.1   1:00.55 ceph-osd

3143 root      20   0 1364m 424m  24m S   16  1.3   1:08.72 ceph-osd

2509 root      20   0  652m 261m  62m S    2  0.8   0:26.42 ceph-mon

    4 root      20   0     0    0    0 S    0  0.0   0:00.08 kworker/0:0

 

Regards,

Quenten Grasso

 

From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of Quenten Grasso
Sent: Tuesday, 18 March 2014 10:19 PM
To: 'ceph-users@xxxxxxxxxxxxxx'
Subject: OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"

 

Hi All,

 

I’m trying to troubleshoot a strange issue with my Ceph cluster.

 

We’re Running Ceph Version 0.72.2

All Nodes are Dell R515’s w/ 6C AMD CPU w/ 32GB Ram, 12 x 3TB NearlineSAS Drives and 2 x 100GB Intel DC S3700 SSD’s for Journals.

All Pools have a replica of 2 or better. I.e. metadata replica of 3.

 

I have 55 OSD’s in the cluster across 5 nodes. When I restart the OSD’s on a single node (any node) the load average of that node shoots up to 230+ and the whole cluster starts blocking IO requests until it settles down and its fine again.

 

Any ideas on why the load average goes so crazy & starts to block IO?

 

 

<snips from my ceph.conf>

[osd]

        osd data = "">

        osd journal size = 15000

        osd mkfs type = xfs

        osd mkfs options xfs = "-i size=2048 -f"

        osd mount options xfs = "rw,noexec,nodev,noatime,nodiratime,barrier=0,inode64,logbufs=8,logbsize=256k"

        osd max backfills = 5

        osd recovery max active = 3

 

[osd.0]

        host = pbnerbd01

        public addr = 10.100.96.10

        cluster addr = 10.100.128.10

        osd journal = /dev/disk/by-id/scsi-36b8ca3a0eaa2660019deaf8d3a40bec4-part1

        devs = /dev/sda4

</end>

 

Thanks,

Quenten

 


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux