On Thu, Jun 7, 2018 at 4:33 PM, Tracy Reed <treed@xxxxxxxxxxxxxxx> wrote:
> On Thu, Jun 07, 2018 at 02:05:31AM PDT, Ilya Dryomov spake thusly:
>> > find /sys/kernel/debug/ceph -type f -print -exec cat {} \;
>>
>> Can you paste the entire output of that command?
>>
>> Which kernel are you running on the client box?
>
> Kernel is Linux cpu04.mydomain.com 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
This is a *very* old kernel.
>
> output is:
>
> # find /sys/kernel/debug/ceph -type f -print -exec cat {} \;
> /sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/osdmap
> epoch 232455
> flags
> pool 0 pg_num 2500 (4095) read_tier -1 write_tier -1
> pool 2 pg_num 512 (511) read_tier -1 write_tier -1
> pool 3 pg_num 128 (127) read_tier -1 write_tier -1
> pool 4 pg_num 100 (127) read_tier -1 write_tier -1
> osd0 10.0.5.3:6801 54% (exists, up) 100%
> osd1 10.0.5.3:6812 57% (exists, up) 100%
> osd2 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd3 10.0.5.4:6812 50% (exists, up) 100%
> osd4 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd5 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd6 10.0.5.9:6861 37% (exists, up) 100%
> osd7 10.0.5.9:6876 28% (exists, up) 100%
> osd8 10.0.5.9:6864 43% (exists, up) 100%
> osd9 10.0.5.9:6836 30% (exists, up) 100%
> osd10 10.0.5.9:6820 22% (exists, up) 100%
> osd11 10.0.5.9:6844 54% (exists, up) 100%
> osd12 10.0.5.9:6803 43% (exists, up) 100%
> osd13 10.0.5.9:6826 41% (exists, up) 100%
> osd14 10.0.5.9:6853 37% (exists, up) 100%
> osd15 10.0.5.9:6872 36% (exists, up) 100%
> osd16 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd17 10.0.5.9:6812 44% (exists, up) 100%
> osd18 10.0.5.9:6817 48% (exists, up) 100%
> osd19 10.0.5.9:6856 33% (exists, up) 100%
> osd20 10.0.5.9:6808 46% (exists, up) 100%
> osd21 10.0.5.9:6871 41% (exists, up) 100%
> osd22 10.0.5.9:6816 49% (exists, up) 100%
> osd23 10.0.5.9:6823 56% (exists, up) 100%
> osd24 10.0.5.9:6800 54% (exists, up) 100%
> osd25 10.0.5.9:6848 54% (exists, up) 100%
> osd26 10.0.5.9:6840 37% (exists, up) 100%
> osd27 10.0.5.9:6883 69% (exists, up) 100%
> osd28 10.0.5.9:6833 39% (exists, up) 100%
> osd29 10.0.5.9:6809 38% (exists, up) 100%
> osd30 10.0.5.9:6829 51% (exists, up) 100%
> osd31 10.0.5.11:6828 47% (exists, up) 100%
> osd32 10.0.5.11:6848 25% (exists, up) 100%
> osd33 10.0.5.11:6802 56% (exists, up) 100%
> osd34 10.0.5.11:6840 35% (exists, up) 100%
> osd35 10.0.5.11:6856 32% (exists, up) 100%
> osd36 10.0.5.11:6832 26% (exists, up) 100%
> [88/1848]
> osd37 10.0.5.11:6868 42% (exists, up) 100%
> osd38 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd39 10.0.5.11:6812 52% (exists, up) 100%
> osd40 10.0.5.11:6864 44% (exists, up) 100%
> osd41 10.0.5.11:6801 25% (exists, up) 100%
> osd42 10.0.5.11:6872 39% (exists, up) 100%
> osd43 10.0.5.13:6809 38% (exists, up) 100%
> osd44 10.0.5.11:6844 47% (exists, up) 100%
> osd45 10.0.5.11:6816 20% (exists, up) 100%
> osd46 10.0.5.3:6800 58% (exists, up) 100%
> osd47 10.0.5.2:6808 43% (exists, up) 100%
> osd48 10.0.5.2:6804 44% (exists, up) 100%
> osd49 10.0.5.2:6812 44% (exists, up) 100%
> osd50 10.0.5.2:6800 47% (exists, up) 100%
> osd51 10.0.5.4:6808 43% (exists, up) 100%
> osd52 10.0.5.12:6815 41% (exists, up) 100%
> osd53 10.0.5.11:6820 24% (up) 100%
> osd54 10.0.5.11:6876 34% (exists, up) 100%
> osd55 10.0.5.11:6836 48% (exists, up) 100%
> osd56 10.0.5.11:6824 31% (exists, up) 100%
> osd57 10.0.5.11:6860 48% (exists, up) 100%
> osd58 10.0.5.11:6852 35% (exists, up) 100%
> osd59 10.0.5.11:6800 42% (exists, up) 100%
> osd60 10.0.5.11:6880 58% (exists, up) 100%
> osd61 10.0.5.3:6803 52% (exists, up) 100%
> osd62 10.0.5.12:6800 42% (exists, up) 100%
> osd63 10.0.5.12:6819 46% (exists, up) 100%
> osd64 10.0.5.12:6809 44% (exists, up) 100%
> osd65 10.0.5.13:6800 44% (exists, up) 100%
> osd66 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd67 10.0.5.13:6808 50% (exists, up) 100%
> osd68 10.0.5.4:6804 41% (exists, up) 100%
> osd69 10.0.5.4:6800 39% (exists, up) 100%
> osd70 10.0.5.13:6804 42% (exists, up) 100%
> osd71 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd72 (unknown sockaddr family 0) 0% (doesn't exist) 100%
> osd73 10.0.5.16:6825 92% (exists, up) 100%
> osd74 10.0.5.16:6846 100% (exists, up) 100%
> osd75 10.0.5.16:6811 98% (exists, up) 100%
> osd76 10.0.5.16:6815 100% (exists, up) 100%
> osd77 10.0.5.16:6835 93% (exists, up) 100%
> osd78 10.0.5.16:6802 97% (exists, up) 100%
> osd79 10.0.5.16:6858 100% (exists, up) 100%
> osd80 10.0.5.16:6839 91% (exists, up) 100%
> osd81 10.0.5.16:6801 100% (exists, up) 100%
> osd82 10.0.5.16:6820 99% (exists, up) 100%
> osd83 10.0.5.16:6852 98% (exists, up) 100%
> [41/1848]
> osd84 10.0.5.16:6862 93% (exists, up) 100%
> osd85 10.0.5.16:6800 96% (exists, up) 100%
> /sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/monmap
> epoch 12
> mon0 10.0.5.2:6789
> mon1 10.0.5.4:6789
> mon2 10.0.5.13:6789
> /sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/osdc
> 6699389 osd1 2.ba8d973e
> rbd_header.dd3b556b8b4567
> 5305738'894263730634752 watch
> 6995225 osd1 2.ce007d3e rbd_data.dd3b556b8b4567.0000000000000b59 set-alloc-hint,write
> 6995664 osd1 2.ce007d3e rbd_data.dd3b556b8b4567.0000000000000b59 set-alloc-hint,write
> 7392381 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 7392382 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 7998200 osd1 2.edcb4966 rbd_data.1fea336b8b4567.0000000000001293 set-alloc-hint,write
> 8878724 osd1 2.940e2cda rbd_data.93285b6b8b4567.000000000000133e set-alloc-hint,write
> 9192095 osd1 2.80947fa7 rbd_data.1fea336b8b4567.0000000000000b7d set-alloc-hint,write
> 9192099 osd1 2.80947fa7 rbd_data.1fea336b8b4567.0000000000000b7d set-alloc-hint,write
> 9192100 osd1 2.80947fa7 rbd_data.1fea336b8b4567.0000000000000b7d set-alloc-hint,write
> 9192101 osd1 2.80947fa7 rbd_data.1fea336b8b4567.0000000000000b7d set-alloc-hint,write
> 9192102 osd1 2.80947fa7 rbd_data.1fea336b8b4567.0000000000000b7d set-alloc-hint,write
> 9219332 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 9219333 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 9234005 osd1 2.4af69687 rbd_data.1c55496b8b4567.000000000000129a set-alloc-hint,write
> 9266352 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 9266353 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 9266354 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 9423525 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 11040736 osd1 2.a5a53fbe rbd_data.93285b6b8b4567.00000000000012fa set-alloc-hint,write
> 11169813 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 11169814 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 12371185 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 12670812 osd1 2.da87333e rbd_data.93285b6b8b4567.0000000000001314 set-alloc-hint,write
> 12958952 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 12958953 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 12958954 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 12958955 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109849 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109856 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109886 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109887 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109888 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109892 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109893 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109894 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109895 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109896 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109927 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109945 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16109946 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271489 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271490 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271491 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271492 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271496 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271497 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271498 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271499 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 16271500 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32154589 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32154590 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32155075 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32155250 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32156442 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 32156983 osd1 2.5d28d036 rbd_data.1c55496b8b4567.00000000000008cb set-alloc-hint,write
> 33982347 osd1 2.678ef636 rbd_data.93285b6b8b4567.000000000000139d set-alloc-hint,write
These lines indicate in-flight requests. Looks like there may have
been a problem with osd1 in the past, as some of these are much older
than others. Try bouncing osd1 with "ceph osd down 1" (it should
come back up automatically) and see if that clears up this batch.
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
