We'll have to run the test again for the OSD log data. Forgot to make
sure that the ceph log partition wasn't full.
client: /sys/kernel/debug/ceph/e88d2684-47c1-5a64-a275-6e375d11b557.client1242818/osdc
REQUESTS 1 homeless 0
11475 osd2 37.a819967 [2]/2 [2]/2
rbd_header.d960431b30e2f 0x400019 3 0'0 call
LINGER REQUESTS
91 osd0 37.851f81e1 [0]/0 [0]/0
rbd_header.d94f26cf2eafd 0x24 0 WC/0
93 osd0 37.98ca7eab [0]/0 [0]/0
rbd_header.d94b96a21ce28 0x24 0 WC/0
106 osd0 37.9720d758 [0]/0 [0]/0
rbd_header.d94f53a7da731 0x24 0 WC/0
104 osd1 37.de8088a1 [1]/1 [1]/1
rbd_header.d94f52e0f9b4d 0x24 0 WC/0
105 osd1 37.db9af301 [1]/1 [1]/1
rbd_header.d94f1ed40c97 0x24 0 WC/0
14 osd2 37.a819967 [2]/2 [2]/2
rbd_header.d960431b30e2f 0x24 2 WC/0
96 osd2 37.8fb9befc [2]/2 [2]/2
rbd_header.d94f03028192c 0x24 0 WC/0
82 osd3 37.370c3798 [3]/3 [3]/3
rbd_header.d94da25e9605b 0x24 0 WC/0
87 osd4 37.9c510a15 [4]/4 [4]/4
rbd_header.d94f079de7a55 0x24 0 WC/0
85 osd5 37.832091ad [5]/5 [5]/5
rbd_header.d94f22d15792a 0x24 0 WC/0
94 osd5 37.344d5f3 [5]/5 [5]/5 rbd_header.d94f6b30f4a
0x24 0 WC/0
103 osd5 37.4cb8bb74 [5]/5 [5]/5
rbd_header.d94ef2bac496b 0x24 0 WC/0
77 osd6 37.7c480437 [6]/6 [6]/6
rbd_header.d94c057a9331d 0x24 1 WC/0
88 osd7 37.58634cdc [7]/7 [7]/7
rbd_header.d94da55fd8a34 0x24 1 WC/0
98 osd7 37.a61c68b [7]/7 [7]/7
rbd_header.d94ce7d11bf7f 0x24 0 WC/0
rbd image '3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7':
size 9536 MB in 2385 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.d960431b30e2f
format: 2
features: layering, striping
flags:
stripe unit: 4096 kB
stripe count: 1
osdmaptool: osdmap file '/tmp/osdmap'
object '3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7'
-> 37.c6 -> [7]
On Tue, Jul 26, 2016 at 3:27 PM, Victor Payno <vpayno@xxxxxxxxxx> wrote:
> We'll have to run the test again for the OSD log data. Forgot to make sure
> that the ceph log partition wasn't full.
>
>
> client:
> /sys/kernel/debug/ceph/e88d2684-47c1-5a64-a275-6e375d11b557.client1242818/osdc
>
> REQUESTS 1 homeless 0
> 11475 osd2 37.a819967 [2]/2 [2]/2 rbd_header.d960431b30e2f
> 0x400019 3 0'0 call
> LINGER REQUESTS
> 91 osd0 37.851f81e1 [0]/0 [0]/0 rbd_header.d94f26cf2eafd
> 0x24 0 WC/0
> 93 osd0 37.98ca7eab [0]/0 [0]/0 rbd_header.d94b96a21ce28
> 0x24 0 WC/0
> 106 osd0 37.9720d758 [0]/0 [0]/0 rbd_header.d94f53a7da731
> 0x24 0 WC/0
> 104 osd1 37.de8088a1 [1]/1 [1]/1 rbd_header.d94f52e0f9b4d
> 0x24 0 WC/0
> 105 osd1 37.db9af301 [1]/1 [1]/1 rbd_header.d94f1ed40c97 0x24
> 0 WC/0
> 14 osd2 37.a819967 [2]/2 [2]/2 rbd_header.d960431b30e2f
> 0x24 2 WC/0
> 96 osd2 37.8fb9befc [2]/2 [2]/2 rbd_header.d94f03028192c
> 0x24 0 WC/0
> 82 osd3 37.370c3798 [3]/3 [3]/3 rbd_header.d94da25e9605b
> 0x24 0 WC/0
> 87 osd4 37.9c510a15 [4]/4 [4]/4 rbd_header.d94f079de7a55
> 0x24 0 WC/0
> 85 osd5 37.832091ad [5]/5 [5]/5 rbd_header.d94f22d15792a
> 0x24 0 WC/0
> 94 osd5 37.344d5f3 [5]/5 [5]/5 rbd_header.d94f6b30f4a 0x24
> 0 WC/0
> 103 osd5 37.4cb8bb74 [5]/5 [5]/5 rbd_header.d94ef2bac496b
> 0x24 0 WC/0
> 77 osd6 37.7c480437 [6]/6 [6]/6 rbd_header.d94c057a9331d
> 0x24 1 WC/0
> 88 osd7 37.58634cdc [7]/7 [7]/7 rbd_header.d94da55fd8a34
> 0x24 1 WC/0
> 98 osd7 37.a61c68b [7]/7 [7]/7 rbd_header.d94ce7d11bf7f
> 0x24 0 WC/0
>
>
> rbd image
> '3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7':
> size 9536 MB in 2385 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.d960431b30e2f
> format: 2
> features: layering, striping
> flags:
> stripe unit: 4096 kB
> stripe count: 1
>
>
> osdmaptool: osdmap file '/tmp/osdmap'
> object '3f370dbabff91bbb7ff23ae7a96e5cb414cac3408013cefed6d4b627b5eed9c7'
> -> 37.c6 -> [7]
>
>
>
>
>
>
> On Tue, Jul 26, 2016 at 11:11 AM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote:
>>
>> On Tue, Jul 26, 2016 at 7:58 PM, Patrick McLean <patrickm@xxxxxxxxxx>
>> wrote:
>> > Hi Ilya,
>> >
>> > We discovered this weekend that enabling lockdep in the kernel makes the
>> > issue go away. We are working on reproducing without lockdep, and
>> > isolating
>> > the issue in the OSD logs. We should be have OSD debug logs this week.
>>
>> I'm going to need the "cat /sys/kernel/debug/ceph/*/osdc" output, the
>> osd log for the osd from that output, and the output of "echo w" and
>> "echo t" to /proc/sysrq-trigger.
>>
>> Thanks,
>>
>> Ilya
>
>
>
>
> --
> Victor Payno
> ビクター·ペイン
>
> Sr. Release Engineer
> シニアリリースエンジニア
>
>
>
> Gaikai, a Sony Computer Entertainment Company ∆○×□
> ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
> 65 Enterprise
> Aliso Viejo, CA 92656 USA
>
> Web: www.gaikai.com
> Email: vpayno@xxxxxxxxxx
> Phone: (949) 330-6850
--
Victor Payno
ビクター·ペイン
Sr. Release Engineer
シニアリリースエンジニア
Gaikai, a Sony Computer Entertainment Company ∆○×□
ガイカイ、ソニー・コンピュータエンタテインメント傘下会社
65 Enterprise
Aliso Viejo, CA 92656 USA
Web: www.gaikai.com
Email: vpayno@xxxxxxxxxx
Phone: (949) 330-6850
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
