Adding ceph-users.
On Mon, Sep 7, 2015 at 11:31 PM, Vickey Singh <vickey.singh22693@xxxxxxxxx> wrote:
On Mon, Sep 7, 2015 at 10:04 PM, Udo Lembke <ulembke@xxxxxxxxxxxx> wrote:Hi Vickey,Thanks for your time in replying to my problem.I had the same rados bench output after changing the motherboard of the monitor node with the lowest IP...
Due to the new mainboard, I assume the hw-clock was wrong during startup. Ceph health show no errors, but all VMs aren't able to do IO (very high load on the VMs - but no traffic).
I stopped the mon, but this don't changed anything. I had to restart all other mons to get IO again. After that I started the first mon also (with the right time now) and all worked fine again...Thanks i will try to restart all OSD / MONS and report back , if it solves my problem
Another posibility:
Do you use journal on SSDs? Perhaps the SSDs can't write to garbage collection?No i don't have journals on SSD , they are on the same OSD disk.
Udo
On 07.09.2015 16:36, Vickey Singh wrote:
Dear Experts
Can someone please help me , why my cluster is not able write data.
See the below output cur MB/S is 0 and Avg MB/s is decreasing.
Ceph Hammer 0.94.2CentOS 6 (3.10.69-1)
The Ceph status says OPS are blocked , i have tried checking , what all i know
- System resources ( CPU , net, disk , memory ) -- All normal- 10G network for public and cluster network -- no saturation- Add disks are physically healthy- No messages in /var/log/messages OR dmesg- Tried restarting OSD which are blocking operation , but no luck- Tried writing through RBD and Rados bench , both are giving same problemm
Please help me to fix this problem.
# rados bench -p rbd 60 writeMaintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or 0 objectsObject prefix: benchmark_data_stor1_1791844sec Cur ops started finished avg MB/s cur MB/s last lat avg lat0 0 0 0 0 0 - 01 16 125 109 435.873 436 0.022076 0.06978642 16 139 123 245.948 56 0.246578 0.06744073 16 139 123 163.969 0 - 0.06744074 16 139 123 122.978 0 - 0.06744075 16 139 123 98.383 0 - 0.06744076 16 139 123 81.9865 0 - 0.06744077 16 139 123 70.2747 0 - 0.06744078 16 139 123 61.4903 0 - 0.06744079 16 139 123 54.6582 0 - 0.067440710 16 139 123 49.1924 0 - 0.067440711 16 139 123 44.7201 0 - 0.067440712 16 139 123 40.9934 0 - 0.067440713 16 139 123 37.8401 0 - 0.067440714 16 139 123 35.1373 0 - 0.067440715 16 139 123 32.7949 0 - 0.067440716 16 139 123 30.7451 0 - 0.067440717 16 139 123 28.9364 0 - 0.067440718 16 139 123 27.3289 0 - 0.067440719 16 139 123 25.8905 0 - 0.06744072015-09-07 15:54:52.694071min lat: 0.022076 max lat: 0.46117 avg lat: 0.0674407sec Cur ops started finished avg MB/s cur MB/s last lat avg lat20 16 139 123 24.596 0 - 0.067440721 16 139 123 23.4247 0 - 0.067440722 16 139 123 22.36 0 - 0.067440723 16 139 123 21.3878 0 - 0.067440724 16 139 123 20.4966 0 - 0.067440725 16 139 123 19.6768 0 - 0.067440726 16 139 123 18.92 0 - 0.067440727 16 139 123 18.2192 0 - 0.067440728 16 139 123 17.5686 0 - 0.067440729 16 139 123 16.9628 0 - 0.067440730 16 139 123 16.3973 0 - 0.067440731 16 139 123 15.8684 0 - 0.067440732 16 139 123 15.3725 0 - 0.067440733 16 139 123 14.9067 0 - 0.067440734 16 139 123 14.4683 0 - 0.067440735 16 139 123 14.0549 0 - 0.067440736 16 139 123 13.6645 0 - 0.067440737 16 139 123 13.2952 0 - 0.067440738 16 139 123 12.9453 0 - 0.067440739 16 139 123 12.6134 0 - 0.06744072015-09-07 15:55:12.697124min lat: 0.022076 max lat: 0.46117 avg lat: 0.0674407sec Cur ops started finished avg MB/s cur MB/s last lat avg lat40 16 139 123 12.2981 0 - 0.067440741 16 139 123 11.9981 0 - 0.0674407
cluster 86edf8b8-b353-49f1-ab0a-a4827a9ea5e8health HEALTH_WARN1 requests are blocked > 32 secmonmap e3: 3 mons at {stor0111=10.100.1.111:6789/0,stor0113=10.100.1.113:6789/0,stor011election epoch 32, quorum 0,1,2 stor0111,stor0113,stor0115osdmap e19536: 50 osds: 50 up, 50 inpgmap v928610: 2752 pgs, 9 pools, 30476 GB data, 4183 kobjects91513 GB used, 47642 GB / 135 TB avail2752 active+clean
Tried using RBD
# dd if=/dev/zero of=file1 bs=4K count=10000 oflag=direct10000+0 records in10000+0 records out40960000 bytes (41 MB) copied, 24.5529 s, 1.7 MB/s
# dd if=/dev/zero of=file1 bs=1M count=100 oflag=direct
100+0 records in100+0 records out104857600 bytes (105 MB) copied, 1.05602 s, 9.3 MB/s
# dd if=/dev/zero of=file1 bs=1G count=1 oflag=direct1+0 records in1+0 records out1073741824 bytes (1.1 GB) copied, 293.551 s, 3.7 MB/s]#
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
