Thanks llya.The following is the solution.
3.8 is missing a *ton* of fixes, I'd strongy recommend an upgrade to
4.0+.
If the osdc output is still the same, try marking osd28 down with "ceph
down 28" (it'll come back automatically) and triggering some I/O (e.g.
a small read from a file you can open). You should see
libceph: osd28 down
libceph: osd28 up
in the dmesg after the I/O is triggered.
Attach
# ceph -s
# find /sys/kernel/debug/ceph -type f -print -exec cat {} \;
when you are done.
2016-04-15 17:42 GMT+08:00 lin zhou <hnuzhoulin2@xxxxxxxxx>:
> some thing goes well now.
> after I reboot osd.28.
>
> the rbd show from :
> /sys/kernel/debug/ceph/409059ba-797e-46da-bc2f-83e3c7779094.client400179/osdc
> 155198393 osd28 2.2e56 rb.0.9e5ab.6b8b4567.00000001f432 write
>
> I can read the file which I can not read before reboot osd.28
>
> 2016-04-15 17:37 GMT+08:00 lin zhou <hnuzhoulin2@xxxxxxxxx>:
>> the increment log in dmesg is :
>> [25592034.504614] libceph: osd44 192.168.43.15:6823 socket closed (con
>> state OPEN)
>> [25592545.157129] libceph: osd17 192.168.43.13:6832 socket closed (con
>> state OPEN)
>> [25593569.346612] libceph: osd28 down
>> [25593573.750922] libceph: osd28 up
>> [25593578.317884] EXT4-fs (rbd17): re-mounted. Opts:
>> grpjquota=quota.group,usrjquota=quota.user,jqfmt=vfsv1
>>
>>
>> /sys/kernel/debug/ceph/409059ba-797e-46da-bc2f-83e3c7779094.client400179/osdc
>> 155198396 osd38 2.2d68 rb.0.5b06d.6b8b4567.00000001f433 write
>> 155198529 osd87 2.27d6 rb.0.578cc.6b8b4567.0000000c0021 write
>> 155198530 osd80 2.33e rb.0.578c3.6b8b4567.0000000c0021 write
>> 155198531 osd16 2.79ce rb.0.5486a.6b8b4567.000000006421 write
>> 155198532 osd22 2.b35e rb.0.899f7.6b8b4567.00000000322f write
>> 155198533 osd26 2.ea40 rb.0.2b68d4.6b8b4567.000000040022 write
>> 155198534 osd20 2.713d rb.0.578d5.6b8b4567.0000000c0021 write
>> 155198535 osd26 2.e436 rb.0.54935.6b8b4567.000000006421 write
>> 155198536 osd56 2.cc9d rb.0.56fb6.6b8b4567.00000001f421 write
>> 155198537 osd80 2.936b rb.0.5486d.6b8b4567.000000006421 write
>> 155198539 osd51 2.d1bd rb.0.51ae2.6b8b4567.000000018623 write
>> 155198586 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> 155198587 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> 155198597 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> 155198598 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> 155199106 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> 155199460 osd40 2.f093 rb.0.899f7.6b8b4567.000000000470 write
>> /sys/kernel/debug/ceph/409059ba-797e-46da-bc2f-83e3c7779094.client400179/monc
>> have osdmap 32321
>> want next osdmap
>> root@musicgci5:~# dmesg |less
>> root@musicgci5:~# ceph -s
>> cluster 409059ba-797e-46da-bc2f-83e3c7779094
>> health HEALTH_OK
>> monmap e1: 3 mons at
>> {musicgci2=192.168.43.12:6789/0,musicgci3=192.168.43.13:6789/0,musicgci4=192.168.43.14:6789/0},
>> election epoch 70, quorum 0,1,2 musicgci2,musicgci3,musicgci4
>> osdmap e32321: 69 osds: 69 up, 69 in
>> pgmap v39523780: 18748 pgs: 18748 active+clean; 48326 GB data, 141
>> TB used, 46976 GB / 187 TB avail; 1000KB/s wr, 10op/s
>> mdsmap e1: 0/0/1 up
>>
>> 2016-04-15 17:33 GMT+08:00 Ilya Dryomov <idryomov@xxxxxxxxx>:
>>> On Fri, Apr 15, 2016 at 11:18 AM, lin zhou <hnuzhoulin2@xxxxxxxxx> wrote:
>>>> root@musicgci5:~# uname -a
>>>> Linux musicgci5 3.8.0-31-generic #46~precise1 SMP Wed Sep 25 23:05:54
>>>> CST 2013 x86_64 x86_64 x86_64 GNU/Linux
>>>> root@musicgci5:~# lsb_release -a
>>>> No LSB modules are available.
>>>> Distributor ID: Ubuntu
>>>> Description: Ubuntu 12.04.3 LTS
>>>> Release: 12.04
>>>> Codename: precise
>>>> root@musicgci5:~# ceph -v
>>>> ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)
>>>
>>> 3.8 is missing a *ton* of fixes, I'd strongy recommend an upgrade to
>>> 4.0+.
>>>
>>> If the osdc output is still the same, try marking osd28 down with "ceph
>>> down 28" (it'll come back automatically) and triggering some I/O (e.g.
>>> a small read from a file you can open). You should see
>>>
>>> libceph: osd28 down
>>> libceph: osd28 up
>>>
>>> in the dmesg after the I/O is triggered.
>>>
>>> Attach
>>>
>>> # ceph -s
>>> # find /sys/kernel/debug/ceph -type f -print -exec cat {} \;
>>>
>>> when you are done.
>>>
>>> Thanks,
>>>
>>> Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
