|
Hello, List.
I met very big trouble during ceph upgrade from bobtail to cuttlefish. My OSDs started to crash to stale so LA went to 100+ on node, after I stop OSD I unable to launch it again because of errors. So, I started to reformat OSDs and eventually meet that I have "incomplete" PGs and It seems to me that I lost actual data (It seems some drives I use is buggy and lost partition and filesystem at all after reboot node as them was empty, and I placed them in all my nodes in one time). I use ceph as source for RBD images, so all my images are unaccessible and ceph health is: root@ceph-osd-1-2:~# ceph -s cluster e74c9e0b-40ce-45e4-904a-676cbaaf042d health HEALTH_WARN 536 pgs backfill; 28 pgs backfilling; 546 pgs degraded; 19 pgs down; 76 pgs incomplete; 33 pgs peering; 23 pgs stale; 109 pgs stuck inactive; 23 pgs stuck stale; 673 pgs stuck unclean; 11 requests are blocked > 32 sec; recovery 2769254/9737405 degraded (28.439%); recovering 76 o/s, 169MB/s; mds cluster is degraded; clock skew detected on mon.0, mon.1 monmap e1: 3 mons at {0=10.252.0.3:6789/0,1=10.252.0.4:6789/0,2=10.252.0.2:6789/0}, election epoch 5486, quorum 0,1,2 0,1,2 osdmap e52800: 48 osds: 41 up, 41 in pgmap v22294835: 1344 pgs: 671 active+clean, 18 active+remapped+wait_backfill, 150 active+degraded+wait_backfill, 12 down+peering, 14 stale+peering, 13 active+degraded+backfilling, 368 active+degraded+remapped+wait_backfill, 7 stale+down+peering, 74 incomplete, 15 active+degraded+remapped+backfilling, 2 stale+incomplete; 6883 GB data, 14627 GB used, 31364 GB / 46031 GB avail; 2769254/9737405 degraded (28.439%); recovering 76 o/s, 169MB/s mdsmap e564: 1/1/1 up {0=2=up:replay}, 2 up:standby Actually, my images stored in RBD pool, but when I try to access it I get: root@ceph-osd-1-2:~# rbd ls rbd: pool rbd doesn't contain rbd images root@ceph-osd-1-2:~# however, I can access images directly: root@ceph-osd-1-2:~# rbd info lun-000000017 rbd image 'lun-000000017': size 10000 MB in 320000 objects order 15 (32768 bytes objects) block_name_prefix: rb.0.3c07.238e1f29 format: 1 and what I really would like is to export data even with errors as usual files and try to check and fix some using fsck. As I understand I should someway fix "incomplete" pgs and "inactive" pgs but I don't really understand how to do it? root@ceph-osd-1-2:~# ceph pg dump | awk '{print $1, $14, $2 ,$6/1024/1024 , $9}' | grep '\.' | grep inc dumped all in format plain 2.1be [43,15,37] 1603 4018.79 incomplete 2.1aa [20,37,44] 4875 11930.5 incomplete 0.1a4 [45,37,27] 0 0 incomplete 2.1a1 [43,45,37] 988 2497.98 incomplete 2.1a0 [30,29,37] 0 0 incomplete 0.1a2 [30,29,37] 0 0 incomplete 1.1a3 [45,37,27] 0 0 incomplete 2.1a2 [45,37,27] 0 0 incomplete 1.1a1 [30,29,37] 0 0 incomplete 2.19d [18,42,24] 0 0 incomplete 0.19f [18,42,24] 0 0 incomplete 1.19e [18,42,24] 0 0 incomplete 0.198 [1,43,37] 0 0 incomplete 1.197 [1,43,37] 0 0 incomplete 2.196 [1,43,37] 0 0 incomplete 2.183 [28,30,37] 0 0 incomplete 2.15f [27,32,37] 0 0 incomplete 0.15c [42,31,37] 0 0 incomplete 1.15b [42,31,37] 0 0 incomplete 2.15a [42,31,37] 0 0 incomplete 2.137 [7,30,37] 0 0 incomplete 0.130 [41,31,25] 139 446.865 stale+incomplete 0.12d [31,37,27] 0 0 incomplete 1.12c [31,37,27] 0 0 incomplete 2.12b [31,37,27] 0 0 incomplete 0.124 [30,37,42] 0 0 incomplete 1.123 [30,37,42] 0 0 incomplete 2.122 [30,37,42] 0 0 incomplete 0.109 [31,37,27] 0 0 incomplete 1.108 [31,37,27] 0 0 incomplete 2.107 [31,37,27] 0 0 incomplete 2.f4 [12,37,20] 3132 7941.63 incomplete 2.ee [12,11,37] 3959 9825.61 incomplete 0.ec [32,37,42] 0 0 incomplete 1.eb [32,37,42] 0 0 incomplete 2.ea [32,37,42] 0 0 incomplete 2.de [45,37,21] 1228 2960.71 incomplete 2.d8 [9,28,37] 832 2029.15 incomplete 2.c1 [13,30,37] 0 0 incomplete 2.b9 [32,2,37] 2971 7305.59 incomplete 2.a4 [30,22,33] 1343 3365.41 incomplete 2.9e [43,37,45] 651 1633.18 incomplete 2.9a [30,2,37] 0 0 incomplete 2.95 [6,43,37] 0 0 incomplete 1.96 [6,43,37] 0 0 incomplete 0.97 [6,43,37] 4 8.00209 incomplete 2.91 [47,28,1] 4518 11066.5 stale+incomplete 0.64 [32,37,8] 0 0 incomplete 1.63 [32,37,8] 0 0 incomplete 2.62 [32,37,8] 0 0 incomplete 0.59 [32,37,21] 311 1010.57 incomplete 2.55 [27,16,37] 0 0 incomplete 0.57 [27,16,37] 0 0 incomplete 1.56 [27,16,37] 0 0 incomplete 2.57 [32,37,21] 42 72.6953 incomplete 2.4c [11,37,30] 3315 8320.87 incomplete 0.49 [18,7,37] 0 0 incomplete 1.48 [18,7,37] 0 0 incomplete 2.47 [18,7,37] 0 0 incomplete 2.3d [28,44,37] 0 0 incomplete 0.3f [28,44,37] 0 0 incomplete 1.3e [28,44,37] 0 0 incomplete 2.2a [43,16,37] 5379 13236.4 incomplete 0.18 [32,42,37] 0 0 incomplete 1.17 [32,42,37] 0 0 incomplete 2.16 [32,42,37] 0 0 incomplete 2.d [45,29,37] 0 0 incomplete 1.e [45,29,37] 0 0 incomplete 0.f [45,29,37] 0 0 incomplete 2.c [18,20,37] 1208 3002.21 incomplete 2.9 [45,37,27] 0 0 incomplete 1.a [45,37,27] 0 0 incomplete 0.b [45,37,27] 28 80.5886 incomplete 2.7 [22,37,30] 2889 7272.09 incomplete 0.1 [31,37,27] 39 135.707 incomplete 1.0 [31,37,27] 0 0 incomplete |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
