Re: OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"

Martin B Nielsen <martin@xxxxxxxxxxx> · Sun, 23 Mar 2014 11:06:42 +0100

Hi,
I can see ~17% hardware interrupts which I find a little high - can you make sure all load is spread over all your cores (/proc/interrupts)?

What about disk util once you restart them? Are they all 100% utilized or is it 'only' mostly cpu-bound?

Also you're running a monitor on this node - how is the load on the nodes where you run a monitor compared to those where you dont?

Cheers,
Martin

On Thu, Mar 20, 2014 at 10:18 AM, Quenten Grasso <qgrasso@xxxxxxxxxx> wrote:

Hi All,

I left out my OS/kernel version, Ubuntu 12.04.4 LTS w/ Kernel 3.10.33-031033-generic (We upgrade our kernels to 3.10 due to Dell Drivers).

Here’s an example of starting all the OSD’s after a reboot.

top - 09:10:51 up 2 min,  1 user,  load average: 332.93, 112.28, 39.96
Tasks: 310 total,   1 running, 309 sleeping,   0 stopped,   0 zombie
Cpu(s): 50.3%us, 32.5%sy,  0.0%ni,  0.0%id,  0.0%wa, 17.2%hi,  0.0%si,  0.0%st
Mem:  32917276k total,  6331224k used, 26586052k free,     1332k buffers
Swap: 33496060k total,        0k used, 33496060k free,  1474084k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15875 root      20   0  910m 381m  50m S   60  1.2   0:50.57 ceph-osd
2996 root      20   0  867m 330m  44m S   59  1.0   0:58.32 ceph-osd
4502 root      20   0  907m 372m  47m S   58  1.2   0:55.14 ceph-osd
12465 root      20   0  949m 418m  55m S   58  1.3   0:51.79 ceph-osd
4171 root      20   0  886m 348m  45m S   57  1.1   0:56.17 ceph-osd
3707 root      20   0  941m 405m  50m S   57  1.3   0:59.68 ceph-osd
3560 root      20   0  924m 394m  51m S   56  1.2   0:59.37 ceph-osd
4318 root      20   0  965m 435m  55m S   56  1.4   0:54.80 ceph-osd
3337 root      20   0  935m 407m  51m S   56  1.3   1:01.96 ceph-osd
3854 root      20   0  897m 366m  48m S   55  1.1   1:00.55 ceph-osd
3143 root      20   0 1364m 424m  24m S   16  1.3   1:08.72 ceph-osd
2509 root      20   0  652m 261m  62m S    2  0.8   0:26.42 ceph-mon
    4 root      20   0     0    0    0 S    0  0.0   0:00.08 kworker/0:0

Regards,
Quenten Grasso

From: ceph-users-bounces@xxxxxxxxxxxxxx [mailto:ceph-users-bounces@xxxxxxxxxxxxxx]
On Behalf Of Quenten Grasso

Sent: Tuesday, 18 March 2014 10:19 PM

To: 'ceph-users@xxxxxxxxxxxxxx'

Subject:  OSD Restarts cause excessively high load average and "requests are blocked > 32 sec"

Hi All,

I’m trying to troubleshoot a strange issue with my Ceph cluster.

We’re Running Ceph Version 0.72.2
All Nodes are Dell R515’s w/ 6C AMD CPU w/ 32GB Ram, 12 x 3TB NearlineSAS Drives and 2 x 100GB Intel DC S3700 SSD’s for Journals.
All Pools have a replica of 2 or better. I.e. metadata replica of 3.

I have 55 OSD’s in the cluster across 5 nodes. When I restart the OSD’s on a single node (any node) the load average of that node shoots up to 230+ and the whole cluster starts blocking IO requests until it settles down and its fine again.

Any ideas on why the load average goes so crazy & starts to block IO?

<snips from my ceph.conf>
[osd]
        osd data = "">
        osd journal size = 15000
        osd mkfs type = xfs
        osd mkfs options xfs = "-i size=2048 -f"
        osd mount options xfs = "rw,noexec,nodev,noatime,nodiratime,barrier=0,inode64,logbufs=8,logbsize=256k"
        osd max backfills = 5
        osd recovery max active = 3

[osd.0]
        host = pbnerbd01
        public addr = 10.100.96.10
        cluster addr = 10.100.128.10
        osd journal = /dev/disk/by-id/scsi-36b8ca3a0eaa2660019deaf8d3a40bec4-part1
        devs = /dev/sda4
</end>

Thanks,
Quenten 

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com