On Thu, Jun 07, 2018 at 09:30:23AM PDT, Jason Dillaman spake thusly:
> I think what Ilya is saying is that it's a very old RHEL 7-based
> kernel (RHEL 7.1?). For example, the current RHEL 7.5 kernel includes
> numerous improvements that have been backported from the current
> upstream kernel.
Ah, I understand now. My VM servers tend not to get upgraded often as
restarting all of the VMs is a hassle. I'll fix that. Do we think that
is related to my issues? It has worked reliably for ages as far as
mapping rbd goes.
I still have the following in flight requests. I set osd.73 out as
suggested and even went and restarted the osd process on the node. It
doesn't seem to have had any effect. And I still have unkillable
processes blocking on mapped rbd devices. I guess I can patch/reboot
this box which would likely clear this up but that's going to have to
wait a week or so and involves downtime for 21 VMs which is less than
ideal. I would love to get this fixed, finish transferring images from
iscsi storage to ceph rbd, then I can retire the iscsi storage and have
some surplus amps so I can bring some more VM servers online so I can
live migrate these VMs in the future allowing easier reboots/upgrades as
that's the real limiting factor here.
# find /sys/kernel/debug/ceph -type f -print -exec cat {} \;
# [70/1950]
/sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/osdmap
epoch 232501
flags
pool 0 pg_num 2500 (4095) read_tier -1 write_tier -1
pool 2 pg_num 512 (511) read_tier -1 write_tier -1
pool 3 pg_num 128 (127) read_tier -1 write_tier -1
pool 4 pg_num 100 (127) read_tier -1 write_tier -1
osd0 10.0.5.3:6801 54% (exists, up) 100%
osd1 10.0.5.3:6812 57% (exists, up) 100%
osd2 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd3 10.0.5.4:6812 50% (exists, up) 100%
osd4 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd5 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd6 10.0.5.9:6861 37% (exists, up) 100%
osd7 10.0.5.9:6876 28% (exists, up) 100%
osd8 10.0.5.9:6864 43% (exists, up) 100%
osd9 10.0.5.9:6836 30% (exists, up) 100%
osd10 10.0.5.9:6820 22% (exists, up) 100%
osd11 10.0.5.9:6844 54% (exists, up) 100%
osd12 10.0.5.9:6803 43% (exists, up) 100%
osd13 10.0.5.9:6826 41% (exists, up) 100%
osd14 10.0.5.9:6853 37% (exists, up) 100%
osd15 10.0.5.9:6872 36% (exists, up) 100%
osd16 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd17 10.0.5.9:6812 44% (exists, up) 100%
osd18 10.0.5.9:6817 48% (exists, up) 100%
osd19 10.0.5.9:6856 33% (exists, up) 100%
osd20 10.0.5.9:6808 46% (exists, up) 100%
osd21 10.0.5.9:6871 41% (exists, up) 100%
osd22 10.0.5.9:6816 49% (exists, up) 100%
osd23 10.0.5.9:6823 56% (exists, up) 100%
osd24 10.0.5.9:6800 54% (exists, up) 100%
osd25 10.0.5.9:6848 54% (exists, up) 100%
osd26 10.0.5.9:6840 37% (exists, up) 100%
osd27 10.0.5.9:6883 69% (exists, up) 100%
osd28 10.0.5.9:6833 39% (exists, up) 100%
osd29 10.0.5.9:6809 38% (exists, up) 100%
osd30 10.0.5.9:6829 51% (exists, up) 100%
osd31 10.0.5.11:6828 47% (exists, up) 100%
osd32 10.0.5.11:6848 25% (exists, up) 100%
osd33 10.0.5.11:6802 56% (exists, up) 100%
osd34 10.0.5.11:6840 35% (exists, up) 100%
osd35 10.0.5.11:6856 32% (exists, up) 100%
osd36 10.0.5.11:6832 26% (exists, up) 100%
osd37 10.0.5.11:6868 42% (exists, up) 100%
osd38 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd39 10.0.5.11:6812 52% (exists, up) 100%
[23/1950]
osd40 10.0.5.11:6864 44% (exists, up) 100%
osd41 10.0.5.11:6801 25% (exists, up) 100%
osd42 10.0.5.11:6872 39% (exists, up) 100%
osd43 10.0.5.13:6809 38% (exists, up) 100%
osd44 10.0.5.11:6844 47% (exists, up) 100%
osd45 10.0.5.11:6816 20% (exists, up) 100%
osd46 10.0.5.3:6800 58% (exists, up) 100%
osd47 10.0.5.2:6808 43% (exists, up) 100%
osd48 10.0.5.2:6804 44% (exists, up) 100%
osd49 10.0.5.2:6812 44% (exists, up) 100%
osd50 10.0.5.2:6800 47% (exists, up) 100%
osd51 10.0.5.4:6808 43% (exists, up) 100%
osd52 10.0.5.12:6815 41% (exists, up) 100%
osd53 10.0.5.11:6820 24% (up) 100%
osd54 10.0.5.11:6876 34% (exists, up) 100%
osd55 10.0.5.11:6836 48% (exists, up) 100%
osd56 10.0.5.11:6824 31% (exists, up) 100%
osd57 10.0.5.11:6860 48% (exists, up) 100%
osd58 10.0.5.11:6852 35% (exists, up) 100%
osd59 10.0.5.11:6800 42% (exists, up) 100%
osd60 10.0.5.11:6880 58% (exists, up) 100%
osd61 10.0.5.3:6803 52% (exists, up) 100%
osd62 10.0.5.12:6800 42% (exists, up) 100%
osd63 10.0.5.12:6819 46% (exists, up) 100%
osd64 10.0.5.12:6809 44% (exists, up) 100%
osd65 10.0.5.13:6800 44% (exists, up) 100%
osd66 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd67 10.0.5.13:6808 50% (exists, up) 100%
osd68 10.0.5.4:6804 41% (exists, up) 100%
osd69 10.0.5.4:6800 39% (exists, up) 100%
osd70 10.0.5.13:6804 42% (exists, up) 100%
osd71 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd72 (unknown sockaddr family 0) 0% (doesn't exist) 100%
osd73 10.0.5.16:6826 92% (exists, up) 100%
osd74 10.0.5.16:6846 100% (exists, up) 100%
osd75 10.0.5.16:6811 98% (exists, up) 100%
osd76 10.0.5.16:6815 100% (exists, up) 100%
osd77 10.0.5.16:6835 93% (exists, up) 100%
osd78 10.0.5.16:6802 97% (exists, up) 100%
osd79 10.0.5.16:6858 100% (exists, up) 100%
osd80 10.0.5.16:6839 91% (exists, up) 100%
osd81 10.0.5.16:6801 100% (exists, up) 100%
osd82 10.0.5.16:6820 99% (exists, up) 100%
osd83 10.0.5.16:6852 98% (exists, up) 100%
osd84 10.0.5.16:6862 93% (exists, up) 100%
osd85 10.0.5.16:6800 96% (exists, up) 100%
/sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/monmap
epoch 12
mon0 10.0.5.2:6789
mon1 10.0.5.4:6789
mon2 10.0.5.13:6789
/sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/osdc
34533231 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533233 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533234 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533235 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533236 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533237 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533238 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533239 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34533241 osd73 0.f0ae1f02 rbd_data.51f32238e1f29.00000000000013de set-alloc-hint,write
34919983 osd67 0.f4cdfa38 rbd_header.51f32238e1f29 5613'998386622791680 watch
34919984 osd6 2.5aca5ef2 rbd_header.93285b6b8b4567 4422885'943544185389056 watch
34919985 osd67 2.4dbc6037 rbd_header.5f75476b8b4567 28922'998386622791680 watch
34919986 osd1 2.ba8d973e rbd_header.dd3b556b8b4567 5305738'894263730634752 watch
/sys/kernel/debug/ceph/b2b00aae-f00d-41b4-a29b-58859aa41375.client31276017/monc
have osdmap 232501
want next osdmap
--
Tracy Reed
http://tracyreed.org
Digital signature attached for your safety.
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
