Hi Ilya, I had already recovered but I managed to recreate the problem again. I ran the commands against rbd_data.f54f9422698a8.0000000000000000 which was one of those listed in osdc this time. We have 2048 PGs in the pool so the list is long. As for when I fetched the object using rados, it grabbed it without issue. osd map: osdmap e6247 pool 'NAS-ic2gw01' (3) object 'rbd_data.f54f9422698a8.0000000000000000' -> pg 3.cac46c43 (3.443) -> up ([33,56], p33) acting ([33,56], p33) osdmaptool: pool 3 pg_num 2048 3.0 [23,138] 23 3.1 [96,76] 96 3.2 [16,62] 16 3.3 [75,150,57] 75 3.4 [85,30] 85 3.5 [138,93,55] 138 3.6 [134,91,64] 134 3.7 [129,56] 129 3.8 [150,63] 150 3.9 [129,84] 129 3.a [62,40,102] 62 3.b [75,105,43] 75 3.c [31,65,132] 31 3.d [122,62,69] 122 3.e [26,144] 26 3.f [87,4] 87 3.10 [82,134,24] 82 3.11 [134,38] 134 3.12 [45,118] 45 3.13 [144,43,20] 144 3.14 [66,40,144] 66 3.15 [67,102,43] 67 3.16 [50,124,53] 50 3.17 [122,51,84] 122 3.18 [61,142] 61 3.19 [9,0,31] 9 3.1a [93,72] 93 3.1b [35,46] 35 3.1c [5,48,17] 5 3.1d [14,134] 14 3.1e [136,72] 136 3.1f [138,59,93] 138 3.20 [34,20] 34 3.21 [148,55] 148 3.22 [24,63] 24 3.23 [68,33,58] 68 3.24 [25,19] 25 3.25 [32,108] 32 3.26 [122,38] 122 3.27 [138,72] 138 3.28 [89,60] 89 3.29 [61,11] 61 3.2a [1,60,122] 1 3.2b [42,64] 42 3.2c [15,88] 15 3.2d [6,100] 6 3.2e [21,8] 21 3.2f [87,0] 87 3.30 [74,3] 74 3.31 [76,58,120] 76 3.32 [108,90] 108 3.33 [48,94,41] 48 3.34 [4,124,91] 4 3.35 [70,83] 70 3.36 [82,142] 82 3.37 [118,25] 118 3.38 [93,70] 93 3.39 [27,115,24] 27 3.3a [88,61,138] 88 3.3b [85,148] 85 3.3c [2,32] 2 3.3d [20,55,41] 20 3.3e [122,86] 122 3.3f [0,81] 0 3.40 [113,29,18] 113 3.41 [75,52] 75 3.42 [21,150,92] 21 3.43 [16,17,86] 16 3.44 [38,140] 38 3.45 [146,67] 146 3.46 [129,10] 129 3.47 [35,36,37] 35 3.48 [127,65] 127 3.49 [77,108] 77 3.4a [134,50] 134 3.4b [59,9,10] 59 3.4c [7,24,82] 7 3.4d [34,27] 34 3.4e [78,49] 78 3.4f [84,37,110] 84 3.50 [108,60] 108 3.51 [51,0,93] 51 3.52 [82,84] 82 3.53 [138,33] 138 3.54 [102,36] 102 3.55 [55,129,32] 55 3.56 [38,68,106] 38 3.57 [8,49] 8 3.58 [40,110,42] 40 3.59 [144,0,54] 144 3.5a [45,24] 45 3.5b [127,30] 127 3.5c [64,148] 64 3.5d [83,43] 83 3.5e [35,23,108] 35 3.5f [65,4] 65 3.60 [63,61,18] 63 3.61 [122,47,42] 122 3.62 [134,8,1] 134 3.63 [18,49,150] 18 3.64 [37,14] 37 3.65 [110,70] 110 3.66 [74,32,9] 74 3.67 [66,26] 66 3.68 [13,86,115] 13 3.69 [40,150,94] 40 3.6a [86,55] 86 3.6b [14,13,105] 14 3.6c [130,45,5] 130 3.6d [47,41] 47 3.6e [50,34,138] 50 3.6f [13,20] 13 3.70 [31,108,42] 31 3.71 [82,52,110] 82 3.72 [49,54] 49 3.73 [115,10] 115 3.74 [110,88,69] 110 3.75 [96,20] 96 3.76 [16,43,2] 16 3.77 [24,65,7] 24 3.78 [64,129] 64 3.79 [105,53] 105 3.7a [56,144,38] 56 3.7b [148,84] 148 3.7c [38,50] 38 3.7d [25,12] 25 3.7e [16,95] 16 3.7f [85,130] 85 3.80 [78,53] 78 3.81 [38,146,4] 38 3.82 [65,22] 65 3.83 [69,12] 69 3.84 [19,43,4] 19 3.85 [89,122] 89 3.86 [120,28,57] 120 3.87 [150,25,71] 150 3.88 [142,10] 142 3.89 [120,26] 120 3.8a [20,132,11] 20 3.8b [14,102,85] 14 3.8c [16,150,79] 16 3.8d [120,29,54] 120 3.8e [53,138] 53 3.8f [30,54,52] 30 3.90 [87,56,132] 87 3.91 [46,28,6] 46 3.92 [86,84] 86 3.93 [37,124,1] 37 3.94 [7,20] 7 3.95 [130,6,50] 130 3.96 [11,4,142] 11 3.97 [83,14] 83 3.98 [61,50] 61 3.99 [78,81,36] 78 3.9a [2,10,33] 2 3.9b [4,148] 4 3.9c [42,74,32] 42 3.9d [80,148] 80 3.9e [140,73] 140 3.9f [77,105] 77 3.a0 [29,102] 29 3.a1 [24,134] 24 3.a2 [23,102,43] 23 3.a3 [8,25] 8 3.a4 [62,122,60] 62 3.a5 [134,93,7] 134 3.a6 [48,7] 48 3.a7 [47,56] 47 3.a8 [32,46] 32 3.a9 [80,85] 80 3.aa [8,70] 8 3.ab [144,78,63] 144 3.ac [23,1] 23 3.ad [56,58] 56 3.ae [57,22] 57 3.af [70,6] 70 3.b0 [105,63,86] 105 3.b1 [127,87,52] 127 3.b2 [136,67,77] 136 3.b3 [3,2] 3 3.b4 [28,120] 28 3.b5 [48,27] 48 3.b6 [100,77] 100 3.b7 [142,20] 142 3.b8 [92,86] 92 3.b9 [83,136,62] 83 3.ba [37,48,150] 37 3.bb [63,96,72] 63 3.bc [50,26,93] 50 3.bd [150,35] 150 3.be [100,80,22] 100 3.bf [142,3,10] 142 3.c0 [83,44,100] 83 3.c1 [146,26] 146 3.c2 [138,86] 138 3.c3 [122,34] 122 3.c4 [72,54] 72 3.c5 [120,93] 120 3.c6 [60,23,120] 60 3.c7 [105,84,25] 105 3.c8 [69,50,92] 69 3.c9 [61,48] 61 3.ca [92,90] 92 3.cb [102,7] 102 3.cc [65,40] 65 3.cd [108,83] 108 3.ce [136,83,31] 136 3.cf [113,24,38] 113 3.d0 [127,55] 127 3.d1 [80,14,106] 80 3.d2 [150,36,67] 150 3.d3 [127,16] 127 3.d4 [51,120] 51 3.d5 [69,24] 69 3.d6 [56,100] 56 3.d7 [14,142] 14 3.d8 [88,15] 88 3.d9 [15,76,77] 15 3.da [98,70] 98 3.db [122,77,79] 122 3.dc [32,3] 32 3.dd [55,54] 55 3.de [120,47,95] 120 3.df [22,108] 22 3.e0 [70,71] 70 3.e1 [40,10,136] 40 3.e2 [30,9] 30 3.e3 [115,83,14] 115 3.e4 [4,37] 4 3.e5 [1,34] 1 3.e6 [52,86] 52 3.e7 [87,80,14] 87 3.e8 [84,19,43] 84 3.e9 [132,93] 132 3.ea [52,130] 52 3.eb [70,84,53] 70 3.ec [59,0,144] 59 3.ed [26,106,27] 26 3.ee [45,70] 45 3.ef [34,48] 34 3.f0 [4,120] 4 3.f1 [5,75] 5 3.f2 [52,146,51] 52 3.f3 [81,55] 81 3.f4 [31,130] 31 3.f5 [42,66] 42 3.f6 [67,68,61] 67 3.f7 [26,6] 26 3.f8 [35,30] 35 3.f9 [81,66] 81 3.fa [87,41,134] 87 3.fb [18,21] 18 3.fc [118,67] 118 3.fd [66,3] 66 3.fe [73,5] 73 3.ff [17,31] 17 3.100 [19,35,85] 19 3.101 [11,8,150] 11 3.102 [17,120,90] 17 3.103 [37,127,50] 37 3.104 [86,57] 86 3.105 [74,8] 74 3.106 [93,75] 93 3.107 [59,100] 59 3.108 [79,18] 79 3.109 [74,127,88] 74 3.10a [106,73] 106 3.10b [78,44] 78 3.10c [11,85] 11 3.10d [80,23] 80 3.10e [22,7] 22 3.10f [50,113] 50 3.110 [23,36,54] 23 3.111 [47,71,130] 47 3.112 [75,49] 75 3.113 [80,45,124] 80 3.114 [140,34] 140 3.115 [63,122] 63 3.116 [50,15,57] 50 3.117 [94,127] 94 3.118 [91,21] 91 3.119 [113,15] 113 3.11a [79,44] 79 3.11b [22,102,16] 22 3.11c [87,53,76] 87 3.11d [30,88] 30 3.11e [98,20] 98 3.11f [29,136] 29 3.120 [43,3,34] 43 3.121 [44,11,98] 44 3.122 [113,47,6] 113 3.123 [14,51] 14 3.124 [50,148] 50 3.125 [11,68] 11 3.126 [87,116] 87 3.127 [100,25,46] 100 3.128 [56,45] 56 3.129 [68,21] 68 3.12a [51,88] 51 3.12b [150,83,50] 150 3.12c [98,40,2] 98 3.12d [24,29,95] 24 3.12e [144,1] 144 3.12f [3,98] 3 3.130 [150,55] 150 3.131 [62,146,15] 62 3.132 [146,49] 146 3.133 [86,38,53] 86 3.134 [20,26] 20 3.135 [146,77] 146 3.136 [136,53,70] 136 3.137 [17,76,78] 17 3.138 [142,21,71] 142 3.139 [80,148,75] 80 3.13a [122,95,14] 122 3.13b [87,25] 87 3.13c [84,122] 84 3.13d [21,87,129] 21 3.13e [58,45,108] 58 3.13f [21,0,140] 21 3.140 [94,55] 94 3.141 [42,3,5] 42 3.142 [62,120] 62 3.143 [18,9,118] 18 3.144 [35,29] 35 3.145 [105,7,64] 105 3.146 [150,38,88] 150 3.147 [42,70] 42 3.148 [18,56,122] 18 3.149 [58,132,29] 58 3.14a [95,36] 95 3.14b [35,116,72] 35 3.14c [16,77] 16 3.14d [75,47] 75 3.14e [54,13,46] 54 3.14f [102,33,88] 102 3.150 [118,35] 118 3.151 [76,77,38] 76 3.152 [74,124,95] 74 3.153 [110,26] 110 3.154 [18,53,0] 18 3.155 [102,1,32] 102 3.156 [118,57,8] 118 3.157 [12,136] 12 3.158 [87,148,80] 87 3.159 [115,76] 115 3.15a [132,28] 132 3.15b [72,19] 72 3.15c [113,45,87] 113 3.15d [60,69] 60 3.15e [144,30] 144 3.15f [43,38] 43 3.160 [138,62,13] 138 3.161 [65,72] 65 3.162 [30,65] 30 3.163 [26,32,86] 26 3.164 [88,94] 88 3.165 [84,115,31] 84 3.166 [2,41] 2 3.167 [6,118,86] 6 3.168 [89,26] 89 3.169 [66,116] 66 3.16a [41,68,7] 41 3.16b [134,38,77] 134 3.16c [65,86] 65 3.16d [74,91,144] 74 3.16e [58,13,113] 58 3.16f [14,20] 14 3.170 [44,9] 44 3.171 [100,17] 100 3.172 [15,4] 15 3.173 [82,60] 82 3.174 [24,70] 24 3.175 [15,52,89] 15 3.176 [80,2] 80 3.177 [7,78] 7 3.178 [9,136] 9 3.179 [13,59] 13 3.17a [52,63,89] 52 3.17b [85,0] 85 3.17c [138,9] 138 3.17d [31,138] 31 3.17e [98,65] 98 3.17f [53,144] 53 3.180 [48,41] 48 3.181 [23,84,54] 23 3.182 [69,108] 69 3.183 [92,108] 92 3.184 [73,43] 73 3.185 [6,83,70] 6 3.186 [3,108,88] 3 3.187 [98,54,30] 98 3.188 [113,29] 113 3.189 [72,106,85] 72 3.18a [50,21,26] 50 3.18b [90,59] 90 3.18c [89,70,134] 89 3.18d [69,110] 69 3.18e [129,9] 129 3.18f [108,91] 108 3.190 [20,89,86] 20 3.191 [64,120] 64 3.192 [67,68] 67 3.193 [45,98] 45 3.194 [84,94,98] 84 3.195 [70,24,122] 70 3.196 [124,27,30] 124 3.197 [35,150] 35 3.198 [15,50,52] 15 3.199 [124,66,16] 124 3.19a [61,105] 61 3.19b [9,55] 9 3.19c [37,60] 37 3.19d [9,23,100] 9 3.19e [28,74,115] 28 3.19f [67,49] 67 3.1a0 [88,124] 88 3.1a1 [60,106] 60 3.1a2 [28,29] 28 3.1a3 [71,52] 71 3.1a4 [20,57] 20 3.1a5 [40,3,18] 40 3.1a6 [72,100,33] 72 3.1a7 [59,66] 59 3.1a8 [54,28] 54 3.1a9 [140,16] 140 3.1aa [26,94] 26 3.1ab [136,21,12] 136 3.1ac [94,26,96] 94 3.1ad [95,18,118] 95 3.1ae [16,22,85] 16 3.1af [88,120] 88 3.1b0 [31,67] 31 3.1b1 [130,29] 130 3.1b2 [32,29] 32 3.1b3 [140,79,3] 140 3.1b4 [113,88] 113 3.1b5 [64,96] 64 3.1b6 [20,18,98] 20 3.1b7 [85,18] 85 3.1b8 [74,53,100] 74 3.1b9 [20,70] 20 3.1ba [78,13,68] 78 3.1bb [8,106,71] 8 3.1bc [82,87] 82 3.1bd [53,136] 53 3.1be [75,90,100] 75 3.1bf [78,45] 78 3.1c0 [46,21,102] 46 3.1c1 [84,54,37] 84 3.1c2 [36,51] 36 3.1c3 [72,22] 72 3.1c4 [108,69,58] 108 3.1c5 [136,80] 136 3.1c6 [10,23,136] 10 3.1c7 [34,16,53] 34 3.1c8 [67,0] 67 3.1c9 [7,24] 7 3.1ca [35,134] 35 3.1cb [27,73] 27 3.1cc [23,98,86] 23 3.1cd [72,5] 72 3.1ce [80,136,94] 80 3.1cf [35,60] 35 3.1d0 [58,95,102] 58 3.1d1 [47,11,124] 47 3.1d2 [148,18] 148 3.1d3 [53,22] 53 3.1d4 [34,64,82] 34 3.1d5 [40,98] 40 3.1d6 [129,95] 129 3.1d7 [29,92] 29 3.1d8 [140,58] 140 3.1d9 [46,1] 46 3.1da [31,89] 31 3.1db [144,7] 144 3.1dc [19,10,88] 19 3.1dd [52,100] 52 3.1de [3,23,100] 3 3.1df [17,67,70] 17 3.1e0 [45,0] 45 3.1e1 [77,122] 77 3.1e2 [81,54] 81 3.1e3 [140,60] 140 3.1e4 [0,45,132] 0 3.1e5 [25,146] 25 3.1e6 [9,60] 9 3.1e7 [0,108] 0 3.1e8 [40,29] 40 3.1e9 [78,12] 78 3.1ea [82,20] 82 3.1eb [72,82] 72 3.1ec [61,102,64] 61 3.1ed [52,65] 52 3.1ee [105,9] 105 3.1ef [62,11] 62 3.1f0 [72,116] 72 3.1f1 [127,42,68] 127 3.1f2 [134,44,73] 134 3.1f3 [140,4] 140 3.1f4 [124,8,41] 124 3.1f5 [98,22,44] 98 3.1f6 [10,129] 10 3.1f7 [85,42,12] 85 3.1f8 [64,66] 64 3.1f9 [59,150,61] 59 3.1fa [96,45,31] 96 3.1fb [14,132,89] 14 3.1fc [115,41,24] 115 3.1fd [132,51,18] 132 3.1fe [2,35] 2 3.1ff [84,132] 84 3.200 [41,71] 41 3.201 [96,14] 96 3.202 [86,113,81] 86 3.203 [80,27] 80 3.204 [31,11] 31 3.205 [132,72,18] 132 3.206 [54,46,96] 54 3.207 [144,68] 144 3.208 [30,56,10] 30 3.209 [132,2] 132 3.20a [67,80,21] 67 3.20b [73,34] 73 3.20c [30,0] 30 3.20d [148,44] 148 3.20e [29,116,11] 29 3.20f [5,23] 5 3.210 [24,46,120] 24 3.211 [113,55,60] 113 3.212 [30,35] 30 3.213 [52,89,18] 52 3.214 [74,75] 74 3.215 [16,26,98] 16 3.216 [48,47] 48 3.217 [10,98] 10 3.218 [43,75] 43 3.219 [108,48] 108 3.21a [85,31] 85 3.21b [16,38] 16 3.21c [37,34] 37 3.21d [25,4] 25 3.21e [94,12,122] 94 3.21f [36,18] 36 3.220 [108,77] 108 3.221 [38,49,74] 38 3.222 [50,130,61] 50 3.223 [7,148] 7 3.224 [17,82,36] 17 3.225 [129,28] 129 3.226 [0,129] 0 3.227 [127,73,7] 127 3.228 [85,44] 85 3.229 [68,17] 68 3.22a [43,76,142] 43 3.22b [12,34,6] 12 3.22c [53,0] 53 3.22d [105,85] 105 3.22e [10,77] 10 3.22f [65,75,5] 65 3.230 [6,70] 6 3.231 [77,78,0] 77 3.232 [16,79] 16 3.233 [62,89] 62 3.234 [62,140,89] 62 3.235 [11,140,85] 11 3.236 [62,130,33] 62 3.237 [105,3] 105 3.238 [5,30] 5 3.239 [83,72] 83 3.23a [78,115,72] 78 3.23b [41,0] 41 3.23c [58,83,17] 58 3.23d [130,94] 130 3.23e [41,110] 41 3.23f [54,38] 54 3.240 [148,26,69] 148 3.241 [41,60] 41 3.242 [80,132] 80 3.243 [102,95] 102 3.244 [11,108] 11 3.245 [32,45,79] 32 3.246 [68,2] 68 3.247 [58,146,57] 58 3.248 [37,144] 37 3.249 [55,85,140] 55 3.24a [150,15] 150 3.24b [17,48] 17 3.24c [74,33] 74 3.24d [2,22] 2 3.24e [115,86] 115 3.24f [57,127] 57 3.250 [46,12] 46 3.251 [40,11,31] 40 3.252 [130,68] 130 3.253 [122,64,93] 122 3.254 [95,65] 95 3.255 [38,118] 38 3.256 [46,127,37] 46 3.257 [42,58] 42 3.258 [24,124,11] 24 3.259 [124,86] 124 3.25a [19,31,32] 19 3.25b [0,21] 0 3.25c [1,116,6] 1 3.25d [52,91] 52 3.25e [92,79] 92 3.25f [85,27,44] 85 3.260 [61,110,44] 61 3.261 [73,19,127] 73 3.262 [92,37,91] 92 3.263 [71,98,25] 71 3.264 [60,14,7] 60 3.265 [130,3] 130 3.266 [98,35] 98 3.267 [2,36] 2 3.268 [25,106,63] 25 3.269 [44,29] 44 3.26a [148,94] 148 3.26b [92,34,86] 92 3.26c [67,62,52] 67 3.26d [88,42,105] 88 3.26e [115,94,60] 115 3.26f [96,11] 96 3.270 [21,14] 21 3.271 [108,90] 108 3.272 [70,127] 70 3.273 [74,129,32] 74 3.274 [56,81] 56 3.275 [86,72] 86 3.276 [19,22,120] 19 3.277 [3,82] 3 3.278 [38,110,47] 38 3.279 [55,94] 55 3.27a [33,26,8] 33 3.27b [9,30] 9 3.27c [98,57] 98 3.27d [73,52] 73 3.27e [91,57] 91 3.27f [22,37,32] 22 3.280 [55,100] 55 3.281 [27,77] 27 3.282 [136,37,82] 136 3.283 [8,25,91] 8 3.284 [106,67,21] 106 3.285 [116,26,65] 116 3.286 [148,71] 148 3.287 [140,51,13] 140 3.288 [140,47] 140 3.289 [96,8,78] 96 3.28a [116,50] 116 3.28b [58,98] 58 3.28c [100,79,76] 100 3.28d [63,69,124] 63 3.28e [13,150,55] 13 3.28f [44,37] 44 3.290 [136,89,32] 136 3.291 [77,90] 77 3.292 [49,150,46] 49 3.293 [59,33] 59 3.294 [18,71] 18 3.295 [83,1,110] 83 3.296 [110,34,20] 110 3.297 [48,41,106] 48 3.298 [142,64] 142 3.299 [80,54] 80 3.29a [82,55,61] 82 3.29b [21,79,66] 21 3.29c [27,21] 27 3.29d [58,25] 58 3.29e [120,82,87] 120 3.29f [41,84] 41 3.2a0 [92,10,61] 92 3.2a1 [85,110] 85 3.2a2 [13,2,52] 13 3.2a3 [3,122] 3 3.2a4 [88,87,66] 88 3.2a5 [14,88,129] 14 3.2a6 [144,73,2] 144 3.2a7 [33,4,59] 33 3.2a8 [113,44,66] 113 3.2a9 [92,30,31] 92 3.2aa [45,74] 45 3.2ab [132,20,14] 132 3.2ac [7,98,76] 7 3.2ad [18,87,25] 18 3.2ae [77,94] 77 3.2af [24,82] 24 3.2b0 [63,122,70] 63 3.2b1 [22,91] 22 3.2b2 [5,144] 5 3.2b3 [14,32,15] 14 3.2b4 [146,29,78] 146 3.2b5 [51,72] 51 3.2b6 [93,130,10] 93 3.2b7 [64,115,1] 64 3.2b8 [25,66] 25 3.2b9 [34,31,127] 34 3.2ba [43,41] 43 3.2bb [10,144,57] 10 3.2bc [14,23,65] 14 3.2bd [148,94] 148 3.2be [60,110] 60 3.2bf [46,9] 46 3.2c0 [87,64,98] 87 3.2c1 [38,29,146] 38 3.2c2 [62,127,71] 62 3.2c3 [78,75,118] 78 3.2c4 [150,94,93] 150 3.2c5 [74,138,83] 74 3.2c6 [118,94] 118 3.2c7 [10,61] 10 3.2c8 [65,100] 65 3.2c9 [18,57] 18 3.2ca [70,69] 70 3.2cb [31,144,13] 31 3.2cc [33,58] 33 3.2cd [118,41] 118 3.2ce [25,132] 25 3.2cf [102,3,80] 102 3.2d0 [61,106,32] 61 3.2d1 [59,10,4] 59 3.2d2 [74,12,83] 74 3.2d3 [8,25,132] 8 3.2d4 [91,122,6] 91 3.2d5 [132,17,54] 132 3.2d6 [90,77] 90 3.2d7 [49,32] 49 3.2d8 [78,100] 78 3.2d9 [47,124] 47 3.2da [64,25,71] 64 3.2db [15,106] 15 3.2dc [7,74] 7 3.2dd [55,148] 55 3.2de [79,70] 79 3.2df [80,110] 80 3.2e0 [65,118] 65 3.2e1 [47,40,142] 47 3.2e2 [148,11] 148 3.2e3 [132,40,95] 132 3.2e4 [45,51] 45 3.2e5 [69,42] 69 3.2e6 [7,140] 7 3.2e7 [18,61] 18 3.2e8 [76,70,67] 76 3.2e9 [24,130] 24 3.2ea [116,21] 116 3.2eb [42,93,94] 42 3.2ec [12,70] 12 3.2ed [115,69] 115 3.2ee [38,32] 38 3.2ef [56,6,50] 56 3.2f0 [105,93] 105 3.2f1 [61,108] 61 3.2f2 [7,9] 7 3.2f3 [89,30] 89 3.2f4 [36,25] 36 3.2f5 [93,142,8] 93 3.2f6 [21,2,88] 21 3.2f7 [48,6,75] 48 3.2f8 [2,6] 2 3.2f9 [35,106] 35 3.2fa [73,94] 73 3.2fb [48,140,27] 48 3.2fc [52,100,21] 52 3.2fd [66,2] 66 3.2fe [78,95] 78 3.2ff [1,148] 1 3.300 [90,20] 90 3.301 [93,15,20] 93 3.302 [72,118] 72 3.303 [34,3] 34 3.304 [27,91,48] 27 3.305 [52,18] 52 3.306 [16,81] 16 3.307 [83,78,65] 83 3.308 [41,50,132] 41 3.309 [28,122] 28 3.30a [74,146] 74 3.30b [65,51] 65 3.30c [136,52,66] 136 3.30d [20,11] 20 3.30e [82,29,15] 82 3.30f [23,47,12] 23 3.310 [41,12] 41 3.311 [79,148] 79 3.312 [26,49] 26 3.313 [63,57,54] 63 3.314 [43,83] 43 3.315 [95,85,47] 95 3.316 [129,86,84] 129 3.317 [94,40,34] 94 3.318 [105,46] 105 3.319 [140,76] 140 3.31a [36,73] 36 3.31b [75,102] 75 3.31c [64,110] 64 3.31d [1,144,63] 1 3.31e [30,68,134] 30 3.31f [60,58] 60 3.320 [47,61] 47 3.321 [6,72,132] 6 3.322 [110,60] 110 3.323 [18,130] 18 3.324 [85,19,136] 85 3.325 [83,113,92] 83 3.326 [54,53,68] 54 3.327 [26,120] 26 3.328 [6,5,19] 6 3.329 [10,7] 10 3.32a [47,142,88] 47 3.32b [2,60] 2 3.32c [98,70] 98 3.32d [55,40] 55 3.32e [53,58,87] 53 3.32f [30,68,94] 30 3.330 [118,50] 118 3.331 [67,47] 67 3.332 [58,61,127] 58 3.333 [88,13,5] 88 3.334 [113,86] 113 3.335 [18,150] 18 3.336 [23,60] 23 3.337 [57,115,76] 57 3.338 [110,86,51] 110 3.339 [8,70,29] 8 3.33a [38,49] 38 3.33b [23,93] 23 3.33c [21,58,40] 21 3.33d [62,55,116] 62 3.33e [129,64,1] 129 3.33f [48,132,21] 48 3.340 [70,144,33] 70 3.341 [15,68] 15 3.342 [102,48,66] 102 3.343 [122,90,88] 122 3.344 [37,48,118] 37 3.345 [1,40,46] 1 3.346 [6,87] 6 3.347 [132,51,85] 132 3.348 [58,4,38] 58 3.349 [73,1] 73 3.34a [38,31] 38 3.34b [23,142] 23 3.34c [24,91] 24 3.34d [132,15,13] 132 3.34e [44,148] 44 3.34f [102,94,61] 102 3.350 [72,127] 72 3.351 [15,144] 15 3.352 [31,75] 31 3.353 [53,87] 53 3.354 [118,49] 118 3.355 [86,77,144] 86 3.356 [146,15] 146 3.357 [33,132] 33 3.358 [94,33] 94 3.359 [127,63] 127 3.35a [108,7,90] 108 3.35b [105,70] 105 3.35c [84,79,100] 84 3.35d [80,9] 80 3.35e [65,75,1] 65 3.35f [89,102,1] 89 3.360 [130,38,40] 130 3.361 [18,60] 18 3.362 [19,28,17] 19 3.363 [24,63,90] 24 3.364 [130,49] 130 3.365 [81,136] 81 3.366 [8,51] 8 3.367 [106,67] 106 3.368 [47,7] 47 3.369 [36,50] 36 3.36a [16,31,11] 16 3.36b [72,50] 72 3.36c [38,24] 38 3.36d [100,86,51] 100 3.36e [4,130] 4 3.36f [45,136] 45 3.370 [40,105] 40 3.371 [55,70,148] 55 3.372 [44,21,46] 44 3.373 [144,24,21] 144 3.374 [90,49,75] 90 3.375 [62,2] 62 3.376 [29,124] 29 3.377 [10,21] 10 3.378 [24,134,51] 24 3.379 [36,105,81] 36 3.37a [73,74] 73 3.37b [54,79,88] 54 3.37c [29,115,16] 29 3.37d [84,57] 84 3.37e [91,94] 91 3.37f [23,24,129] 23 3.380 [6,48,54] 6 3.381 [129,30] 129 3.382 [110,24] 110 3.383 [45,130] 45 3.384 [96,13,90] 96 3.385 [82,21,124] 82 3.386 [12,50] 12 3.387 [18,42] 18 3.388 [78,132] 78 3.389 [78,115,42] 78 3.38a [52,96] 52 3.38b [43,8] 43 3.38c [14,8] 14 3.38d [136,63,68] 136 3.38e [22,102] 22 3.38f [120,2] 120 3.390 [68,9] 68 3.391 [124,45] 124 3.392 [73,79,8] 73 3.393 [1,65] 1 3.394 [20,45] 20 3.395 [45,47] 45 3.396 [102,80] 102 3.397 [69,51] 69 3.398 [140,95] 140 3.399 [14,116,91] 14 3.39a [2,82] 2 3.39b [13,91,8] 13 3.39c [81,27] 81 3.39d [65,58,118] 65 3.39e [93,74] 93 3.39f [21,83,52] 21 3.3a0 [23,47] 23 3.3a1 [89,118] 89 3.3a2 [92,5] 92 3.3a3 [10,98] 10 3.3a4 [32,130] 32 3.3a5 [79,58] 79 3.3a6 [15,28] 15 3.3a7 [56,33,43] 56 3.3a8 [68,142,2] 68 3.3a9 [76,142,21] 76 3.3aa [63,64] 63 3.3ab [19,118,25] 19 3.3ac [64,38,144] 64 3.3ad [59,142,41] 59 3.3ae [3,58] 3 3.3af [105,27] 105 3.3b0 [15,44] 15 3.3b1 [82,71,118] 82 3.3b2 [49,31,67] 49 3.3b3 [35,76,21] 35 3.3b4 [7,58,76] 7 3.3b5 [102,23] 102 3.3b6 [50,142] 50 3.3b7 [34,122] 34 3.3b8 [138,34,81] 138 3.3b9 [0,142] 0 3.3ba [57,4] 57 3.3bb [122,11,62] 122 3.3bc [120,6] 120 3.3bd [21,59] 21 3.3be [105,1,13] 105 3.3bf [72,77,82] 72 3.3c0 [8,66,120] 8 3.3c1 [43,140,48] 43 3.3c2 [88,31] 88 3.3c3 [94,83,52] 94 3.3c4 [45,130,30] 45 3.3c5 [25,90] 25 3.3c6 [115,53,23] 115 3.3c7 [71,150] 71 3.3c8 [53,1,67] 53 3.3c9 [37,100] 37 3.3ca [52,45,30] 52 3.3cb [68,71,90] 68 3.3cc [92,118,73] 92 3.3cd [5,56,110] 5 3.3ce [15,146] 15 3.3cf [74,44,29] 74 3.3d0 [13,76] 13 3.3d1 [5,38,142] 5 3.3d2 [7,148,93] 7 3.3d3 [35,130] 35 3.3d4 [20,58,91] 20 3.3d5 [62,146,44] 62 3.3d6 [50,118,48] 50 3.3d7 [79,130] 79 3.3d8 [86,24,115] 86 3.3d9 [6,120,2] 6 3.3da [136,63] 136 3.3db [146,7] 146 3.3dc [22,64,25] 22 3.3dd [44,120,42] 44 3.3de [85,124] 85 3.3df [0,132] 0 3.3e0 [78,72] 78 3.3e1 [32,30] 32 3.3e2 [59,61] 59 3.3e3 [74,98,4] 74 3.3e4 [54,83,25] 54 3.3e5 [0,93,5] 0 3.3e6 [87,27] 87 3.3e7 [46,48] 46 3.3e8 [134,77] 134 3.3e9 [27,24] 27 3.3ea [108,27] 108 3.3eb [127,47,19] 127 3.3ec [68,37,102] 68 3.3ed [5,20,59] 5 3.3ee [150,81] 150 3.3ef [22,28] 22 3.3f0 [85,148] 85 3.3f1 [76,94] 76 3.3f2 [28,132,27] 28 3.3f3 [40,118,54] 40 3.3f4 [54,129] 54 3.3f5 [127,14,84] 127 3.3f6 [57,70] 57 3.3f7 [96,35] 96 3.3f8 [74,150,13] 74 3.3f9 [67,90,85] 67 3.3fa [79,3] 79 3.3fb [47,138,57] 47 3.3fc [82,81,0] 82 3.3fd [22,0] 22 3.3fe [59,76] 59 3.3ff [24,10] 24 3.400 [54,28] 54 3.401 [76,113] 76 3.402 [98,46,29] 98 3.403 [49,55,94] 49 3.404 [80,35] 80 3.405 [34,90,108] 34 3.406 [22,46] 22 3.407 [7,56] 7 3.408 [56,82,6] 56 3.409 [89,124] 89 3.40a [20,50,83] 20 3.40b [31,146] 31 3.40c [3,30,144] 3 3.40d [68,96,10] 68 3.40e [94,19,118] 94 3.40f [0,54] 0 3.410 [138,31] 138 3.411 [25,44,140] 25 3.412 [41,113] 41 3.413 [53,51,118] 53 3.414 [37,18,100] 37 3.415 [24,85] 24 3.416 [115,58,55] 115 3.417 [122,78] 122 3.418 [38,62,84] 38 3.419 [5,106,67] 5 3.41a [44,42,108] 44 3.41b [2,1,118] 2 3.41c [22,59] 22 3.41d [148,48] 148 3.41e [47,142] 47 3.41f [142,44,27] 142 3.420 [13,150,82] 13 3.421 [59,9,40] 59 3.422 [80,116,9] 80 3.423 [19,92] 19 3.424 [26,94] 26 3.425 [1,113] 1 3.426 [15,4] 15 3.427 [148,71,80] 148 3.428 [35,53] 35 3.429 [80,6] 80 3.42a [110,54,26] 110 3.42b [82,65,16] 82 3.42c [48,132,6] 48 3.42d [2,138] 2 3.42e [47,95,92] 47 3.42f [134,52,82] 134 3.430 [25,24,22] 25 3.431 [8,10,116] 8 3.432 [52,86,77] 52 3.433 [78,20] 78 3.434 [73,23] 73 3.435 [84,53] 84 3.436 [42,94,68] 42 3.437 [53,72] 53 3.438 [65,134] 65 3.439 [6,86,132] 6 3.43a [144,85,11] 144 3.43b [80,18] 80 3.43c [66,100,21] 66 3.43d [93,86] 93 3.43e [105,30] 105 3.43f [1,44] 1 3.440 [148,37,23] 148 3.441 [146,49] 146 3.442 [79,32,82] 79 3.443 [33,56] 33 3.444 [49,88,146] 49 3.445 [22,85,19] 22 3.446 [146,86] 146 3.447 [92,95,129] 92 3.448 [51,66,113] 51 3.449 [93,87] 93 3.44a [59,1,102] 59 3.44b [63,57,47] 63 3.44c [127,40] 127 3.44d [58,61,72] 58 3.44e [61,79,134] 61 3.44f [23,70] 23 3.450 [81,36,34] 81 3.451 [54,134,15] 54 3.452 [118,64,18] 118 3.453 [129,72] 129 3.454 [75,130,84] 75 3.455 [46,47] 46 3.456 [51,80,18] 51 3.457 [17,98,68] 17 3.458 [37,71] 37 3.459 [27,116,85] 27 3.45a [84,49] 84 3.45b [8,71] 8 3.45c [46,132] 46 3.45d [22,38,76] 22 3.45e [5,13] 5 3.45f [24,129,75] 24 3.460 [98,8] 98 3.461 [60,96] 60 3.462 [0,42] 0 3.463 [130,28,50] 130 3.464 [2,69,118] 2 3.465 [32,130] 32 3.466 [15,68,10] 15 3.467 [34,89] 34 3.468 [105,32,25] 105 3.469 [40,11,57] 40 3.46a [59,41] 59 3.46b [113,51,53] 113 3.46c [9,95] 9 3.46d [22,85,36] 22 3.46e [100,80] 100 3.46f [42,102] 42 3.470 [36,47] 36 3.471 [92,69,136] 92 3.472 [42,60] 42 3.473 [38,33] 38 3.474 [55,14] 55 3.475 [35,17,96] 35 3.476 [11,52,1] 11 3.477 [60,41] 60 3.478 [17,118,42] 17 3.479 [74,11] 74 3.47a [130,95,14] 130 3.47b [19,27] 19 3.47c [50,0] 50 3.47d [74,33,106] 74 3.47e [10,84] 10 3.47f [94,132] 94 3.480 [74,93,113] 74 3.481 [73,38] 73 3.482 [51,68,81] 51 3.483 [1,148,4] 1 3.484 [8,19] 8 3.485 [67,28] 67 3.486 [84,93] 84 3.487 [22,92] 22 3.488 [7,37,116] 7 3.489 [59,45,74] 59 3.48a [19,54] 19 3.48b [144,29,52] 144 3.48c [83,24] 83 3.48d [48,83] 48 3.48e [140,76,61] 140 3.48f [82,118] 82 3.490 [53,72] 53 3.491 [10,73] 10 3.492 [28,13] 28 3.493 [68,110] 68 3.494 [8,63] 8 3.495 [130,50] 130 3.496 [11,12] 11 3.497 [17,86] 17 3.498 [38,124] 38 3.499 [1,77,118] 1 3.49a [17,142,18] 17 3.49b [75,20] 75 3.49c [36,140,6] 36 3.49d [56,30] 56 3.49e [33,92,140] 33 3.49f [6,68] 6 3.4a0 [5,3,108] 5 3.4a1 [129,71] 129 3.4a2 [67,136] 67 3.4a3 [24,94,69] 24 3.4a4 [113,44,49] 113 3.4a5 [28,110,71] 28 3.4a6 [68,74] 68 3.4a7 [20,150] 20 3.4a8 [24,110] 24 3.4a9 [69,102] 69 3.4aa [50,95,144] 50 3.4ab [89,15] 89 3.4ac [91,16,127] 91 3.4ad [41,11] 41 3.4ae [57,70,106] 57 3.4af [4,42,35] 4 3.4b0 [32,14,3] 32 3.4b1 [1,75] 1 3.4b2 [124,29,66] 124 3.4b3 [1,26] 1 3.4b4 [18,51] 18 3.4b5 [146,91,48] 146 3.4b6 [47,138] 47 3.4b7 [8,102] 8 3.4b8 [150,36] 150 3.4b9 [71,127,76] 71 3.4ba [37,47,96] 37 3.4bb [90,40] 90 3.4bc [38,6] 38 3.4bd [46,52,102] 46 3.4be [30,88,29] 30 3.4bf [21,58,108] 21 3.4c0 [17,82] 17 3.4c1 [63,93,76] 63 3.4c2 [29,10,87] 29 3.4c3 [7,18] 7 3.4c4 [84,85] 84 3.4c5 [9,51] 9 3.4c6 [23,9] 23 3.4c7 [146,7,32] 146 3.4c8 [50,142] 50 3.4c9 [124,13] 124 3.4ca [50,32,45] 50 3.4cb [88,65,27] 88 3.4cc [144,26] 144 3.4cd [106,14,60] 106 3.4ce [66,38] 66 3.4cf [37,86,84] 37 3.4d0 [47,138] 47 3.4d1 [42,60] 42 3.4d2 [0,33,98] 0 3.4d3 [46,124] 46 3.4d4 [64,78] 64 3.4d5 [146,16,71] 146 3.4d6 [61,11] 61 3.4d7 [5,63] 5 3.4d8 [20,90] 20 3.4d9 [36,45] 36 3.4da [9,40] 9 3.4db [95,17] 95 3.4dc [0,120,10] 0 3.4dd [17,4,83] 17 3.4de [94,113,7] 94 3.4df [120,62,75] 120 3.4e0 [59,54,142] 59 3.4e1 [93,130] 93 3.4e2 [116,23] 116 3.4e3 [127,91] 127 3.4e4 [90,32,130] 90 3.4e5 [28,71,82] 28 3.4e6 [134,6] 134 3.4e7 [73,122] 73 3.4e8 [96,91,10] 96 3.4e9 [6,43] 6 3.4ea [122,91] 122 3.4eb [58,129,76] 58 3.4ec [94,115,92] 94 3.4ed [144,44,37] 144 3.4ee [124,7] 124 3.4ef [40,87] 40 3.4f0 [13,24,63] 13 3.4f1 [3,118,94] 3 3.4f2 [64,108] 64 3.4f3 [146,93] 146 3.4f4 [142,57] 142 3.4f5 [90,120] 90 3.4f6 [83,105,93] 83 3.4f7 [20,129] 20 3.4f8 [5,11] 5 3.4f9 [115,91] 115 3.4fa [24,110] 24 3.4fb [49,58] 49 3.4fc [7,58] 7 3.4fd [20,124] 20 3.4fe [53,94] 53 3.4ff [24,116] 24 3.500 [68,47] 68 3.501 [118,90] 118 3.502 [44,150] 44 3.503 [57,47,36] 57 3.504 [7,48,148] 7 3.505 [55,110] 55 3.506 [56,138,1] 56 3.507 [16,73] 16 3.508 [86,17,91] 86 3.509 [35,6] 35 3.50a [1,129,87] 1 3.50b [1,41] 1 3.50c [59,17] 59 3.50d [86,25] 86 3.50e [96,1] 96 3.50f [87,17] 87 3.510 [27,30] 27 3.511 [33,105,72] 33 3.512 [4,95,57] 4 3.513 [64,23,5] 64 3.514 [122,52] 122 3.515 [92,53] 92 3.516 [115,6,56] 115 3.517 [63,142] 63 3.518 [102,30,20] 102 3.519 [16,41,138] 16 3.51a [64,13] 64 3.51b [87,17,18] 87 3.51c [34,41,52] 34 3.51d [17,75] 17 3.51e [136,95,12] 136 3.51f [49,78,102] 49 3.520 [75,144,70] 75 3.521 [27,17] 27 3.522 [8,43,110] 8 3.523 [37,67] 37 3.524 [46,20,62] 46 3.525 [49,95] 49 3.526 [49,116,78] 49 3.527 [138,12] 138 3.528 [87,65,144] 87 3.529 [78,46,124] 78 3.52a [40,134,47] 40 3.52b [12,35,146] 12 3.52c [43,67] 43 3.52d [89,19] 89 3.52e [90,69] 90 3.52f [110,71,61] 110 3.530 [1,77,108] 1 3.531 [91,68,57] 91 3.532 [6,136,83] 6 3.533 [118,0] 118 3.534 [89,2] 89 3.535 [13,62] 13 3.536 [9,140] 9 3.537 [12,34,136] 12 3.538 [6,110,15] 6 3.539 [71,36] 71 3.53a [54,83] 54 3.53b [26,113,78] 26 3.53c [71,129,50] 71 3.53d [5,3,19] 5 3.53e [29,67,47] 29 3.53f [9,100,67] 9 3.540 [70,36,61] 70 3.541 [10,9] 10 3.542 [26,142] 26 3.543 [113,73] 113 3.544 [18,49] 18 3.545 [136,95] 136 3.546 [113,0] 113 3.547 [67,72] 67 3.548 [82,34] 82 3.549 [57,122] 57 3.54a [83,8,89] 83 3.54b [94,120] 94 3.54c [82,57] 82 3.54d [106,90] 106 3.54e [25,48,129] 25 3.54f [18,4] 18 3.550 [86,20] 86 3.551 [37,86,16] 37 3.552 [28,95,13] 28 3.553 [84,45] 84 3.554 [34,20] 34 3.555 [8,42,50] 8 3.556 [66,34] 66 3.557 [36,77] 36 3.558 [9,63,80] 9 3.559 [98,81] 98 3.55a [76,43,57] 76 3.55b [26,73] 26 3.55c [25,8,142] 25 3.55d [26,86] 26 3.55e [89,113] 89 3.55f [63,86] 63 3.560 [130,73,11] 130 3.561 [53,90] 53 3.562 [71,115,43] 71 3.563 [17,115,8] 17 3.564 [17,62,146] 17 3.565 [127,32] 127 3.566 [59,44] 59 3.567 [95,4,118] 95 3.568 [150,77] 150 3.569 [76,67] 76 3.56a [26,62] 26 3.56b [6,74,0] 6 3.56c [136,7,78] 136 3.56d [1,124] 1 3.56e [129,20] 129 3.56f [98,12,95] 98 3.570 [105,13,14] 105 3.571 [56,35] 56 3.572 [55,0,124] 55 3.573 [113,50,60] 113 3.574 [8,77,132] 8 3.575 [75,10,81] 75 3.576 [94,60,63] 94 3.577 [32,19] 32 3.578 [108,32,21] 108 3.579 [64,83,93] 64 3.57a [86,132] 86 3.57b [146,92,86] 146 3.57c [24,50,122] 24 3.57d [27,24,136] 27 3.57e [91,136,73] 91 3.57f [28,83] 28 3.580 [23,54,33] 23 3.581 [1,93,130] 1 3.582 [31,46,150] 31 3.583 [26,132,0] 26 3.584 [72,108,11] 72 3.585 [18,98] 18 3.586 [96,94] 96 3.587 [93,82] 93 3.588 [124,86,45] 124 3.589 [40,66,13] 40 3.58a [89,30] 89 3.58b [73,87] 73 3.58c [140,4,19] 140 3.58d [13,66] 13 3.58e [72,89] 72 3.58f [96,42] 96 3.590 [142,14] 142 3.591 [11,29] 11 3.592 [106,9,47] 106 3.593 [47,44] 47 3.594 [64,5,67] 64 3.595 [82,83,57] 82 3.596 [130,38] 130 3.597 [62,91,4] 62 3.598 [79,35] 79 3.599 [5,53,67] 5 3.59a [95,61,72] 95 3.59b [108,89] 108 3.59c [130,20] 130 3.59d [91,62,37] 91 3.59e [15,102,10] 15 3.59f [26,22] 26 3.5a0 [14,130,41] 14 3.5a1 [31,142,15] 31 3.5a2 [57,10] 57 3.5a3 [77,102] 77 3.5a4 [59,33,102] 59 3.5a5 [88,2,102] 88 3.5a6 [65,16,11] 65 3.5a7 [116,49,55] 116 3.5a8 [132,6,34] 132 3.5a9 [67,88,58] 67 3.5aa [38,85] 38 3.5ab [2,29,1] 2 3.5ac [0,93] 0 3.5ad [134,24] 134 3.5ae [49,96,64] 49 3.5af [95,66,6] 95 3.5b0 [122,11,56] 122 3.5b1 [122,56,11] 122 3.5b2 [75,36,74] 75 3.5b3 [52,21] 52 3.5b4 [105,38] 105 3.5b5 [129,94] 129 3.5b6 [102,86] 102 3.5b7 [138,25,1] 138 3.5b8 [79,31] 79 3.5b9 [11,62] 11 3.5ba [14,130,60] 14 3.5bb [110,60] 110 3.5bc [76,18,37] 76 3.5bd [56,27] 56 3.5be [130,16,75] 130 3.5bf [105,72,77] 105 3.5c0 [11,93,58] 11 3.5c1 [90,55,105] 90 3.5c2 [49,98,50] 49 3.5c3 [28,79,132] 28 3.5c4 [45,76] 45 3.5c5 [105,58] 105 3.5c6 [58,100] 58 3.5c7 [27,73] 27 3.5c8 [6,46] 6 3.5c9 [13,0,59] 13 3.5ca [118,95] 118 3.5cb [11,37,64] 11 3.5cc [57,4] 57 3.5cd [40,47] 40 3.5ce [31,24] 31 3.5cf [1,8,25] 1 3.5d0 [116,31] 116 3.5d1 [83,74,129] 83 3.5d2 [31,32] 31 3.5d3 [69,70] 69 3.5d4 [8,15] 8 3.5d5 [17,31] 17 3.5d6 [61,16] 61 3.5d7 [10,132] 10 3.5d8 [43,136] 43 3.5d9 [34,106,77] 34 3.5da [3,20,58] 3 3.5db [9,12] 9 3.5dc [129,29] 129 3.5dd [85,96] 85 3.5de [95,1,25] 95 3.5df [124,11,37] 124 3.5e0 [24,34] 24 3.5e1 [77,11] 77 3.5e2 [100,83,69] 100 3.5e3 [127,48,58] 127 3.5e4 [86,60,124] 86 3.5e5 [41,74] 41 3.5e6 [30,40,93] 30 3.5e7 [94,17,84] 94 3.5e8 [23,1,37] 23 3.5e9 [54,60] 54 3.5ea [124,0] 124 3.5eb [142,24,1] 142 3.5ec [17,134,79] 17 3.5ed [74,148,52] 74 3.5ee [38,36,9] 38 3.5ef [82,17] 82 3.5f0 [43,105] 43 3.5f1 [68,82] 68 3.5f2 [96,63,54] 96 3.5f3 [83,144,18] 83 3.5f4 [41,56,30] 41 3.5f5 [58,71,29] 58 3.5f6 [59,105] 59 3.5f7 [73,134] 73 3.5f8 [52,18,21] 52 3.5f9 [18,64,118] 18 3.5fa [129,46] 129 3.5fb [92,31] 92 3.5fc [52,86] 52 3.5fd [2,43,60] 2 3.5fe [3,42] 3 3.5ff [40,148,82] 40 3.600 [82,110] 82 3.601 [95,1] 95 3.602 [68,43,69] 68 3.603 [24,83,90] 24 3.604 [56,146,67] 56 3.605 [69,148] 69 3.606 [7,92] 7 3.607 [42,13] 42 3.608 [91,94] 91 3.609 [116,49] 116 3.60a [71,27,102] 71 3.60b [31,4,41] 31 3.60c [38,52] 38 3.60d [102,76,77] 102 3.60e [3,95,144] 3 3.60f [113,92,19] 113 3.610 [25,66,23] 25 3.611 [57,142] 57 3.612 [38,68] 38 3.613 [21,67] 21 3.614 [50,48,124] 50 3.615 [53,132] 53 3.616 [60,59,122] 60 3.617 [14,108,7] 14 3.618 [13,34,129] 13 3.619 [47,142,29] 47 3.61a [122,90] 122 3.61b [150,31,44] 150 3.61c [47,87,92] 47 3.61d [90,60,53] 90 3.61e [2,134] 2 3.61f [47,127,15] 47 3.620 [29,59] 29 3.621 [110,11] 110 3.622 [57,18,102] 57 3.623 [80,45] 80 3.624 [55,80] 55 3.625 [16,78,55] 16 3.626 [15,33] 15 3.627 [6,79] 6 3.628 [92,53] 92 3.629 [7,43] 7 3.62a [48,6] 48 3.62b [110,6] 110 3.62c [25,120,86] 25 3.62d [130,70,42] 130 3.62e [37,43,110] 37 3.62f [28,93] 28 3.630 [132,40,55] 132 3.631 [49,38] 49 3.632 [146,64] 146 3.633 [23,33] 23 3.634 [71,77,70] 71 3.635 [25,70] 25 3.636 [36,14] 36 3.637 [93,0] 93 3.638 [27,88] 27 3.639 [87,45] 87 3.63a [79,22,140] 79 3.63b [32,93,55] 32 3.63c [136,93,66] 136 3.63d [78,26] 78 3.63e [81,120,23] 81 3.63f [66,49,4] 66 3.640 [25,64,18] 25 3.641 [79,52,89] 79 3.642 [44,82] 44 3.643 [7,44,35] 7 3.644 [30,106,6] 30 3.645 [94,20] 94 3.646 [87,106,80] 87 3.647 [11,108,17] 11 3.648 [16,27] 16 3.649 [140,91] 140 3.64a [118,69] 118 3.64b [10,63] 10 3.64c [74,72,63] 74 3.64d [140,70,23] 140 3.64e [51,24,74] 51 3.64f [78,40] 78 3.650 [62,21,91] 62 3.651 [100,23] 100 3.652 [75,100,41] 75 3.653 [57,91,122] 57 3.654 [106,62] 106 3.655 [42,77,32] 42 3.656 [9,113,10] 9 3.657 [10,2,110] 10 3.658 [136,79] 136 3.659 [82,129,59] 82 3.65a [69,44] 69 3.65b [130,56] 130 3.65c [4,96,38] 4 3.65d [7,25] 7 3.65e [89,68,1] 89 3.65f [26,74,136] 26 3.660 [36,69] 36 3.661 [100,74] 100 3.662 [36,89] 36 3.663 [9,59,102] 9 3.664 [22,19] 22 3.665 [70,150] 70 3.666 [105,57,26] 105 3.667 [77,1,68] 77 3.668 [16,144] 16 3.669 [130,34,8] 130 3.66a [81,14] 81 3.66b [50,76] 50 3.66c [113,58] 113 3.66d [116,60,38] 116 3.66e [78,79] 78 3.66f [5,28,15] 5 3.670 [89,30,31] 89 3.671 [52,122] 52 3.672 [132,53,28] 132 3.673 [90,77,108] 90 3.674 [122,23,62] 122 3.675 [48,54,37] 48 3.676 [146,29] 146 3.677 [72,19] 72 3.678 [18,146,15] 18 3.679 [82,116,80] 82 3.67a [75,64,57] 75 3.67b [9,82] 9 3.67c [46,66,138] 46 3.67d [2,73,98] 2 3.67e [25,116,95] 25 3.67f [76,30] 76 3.680 [150,61,27] 150 3.681 [37,110,11] 37 3.682 [7,124] 7 3.683 [74,142] 74 3.684 [17,95] 17 3.685 [72,86,136] 72 3.686 [108,77,74] 108 3.687 [23,105] 23 3.688 [51,124] 51 3.689 [106,83,56] 106 3.68a [56,69] 56 3.68b [58,105] 58 3.68c [8,41] 8 3.68d [94,67] 94 3.68e [49,63] 49 3.68f [144,42] 144 3.690 [96,1,24] 96 3.691 [68,108,2] 68 3.692 [110,68] 110 3.693 [40,66] 40 3.694 [0,69,46] 0 3.695 [71,136,13] 71 3.696 [54,83] 54 3.697 [25,100] 25 3.698 [87,142] 87 3.699 [45,94] 45 3.69a [127,4,55] 127 3.69b [48,1,26] 48 3.69c [43,124,59] 43 3.69d [76,2,124] 76 3.69e [100,48] 100 3.69f [94,3] 94 3.6a0 [41,84] 41 3.6a1 [11,88] 11 3.6a2 [122,43] 122 3.6a3 [1,130,2] 1 3.6a4 [15,80,50] 15 3.6a5 [9,83] 9 3.6a6 [5,44,134] 5 3.6a7 [22,28,69] 22 3.6a8 [13,122,62] 13 3.6a9 [54,24] 54 3.6aa [13,79] 13 3.6ab [102,3,26] 102 3.6ac [34,100] 34 3.6ad [86,68,138] 86 3.6ae [96,62] 96 3.6af [75,56] 75 3.6b0 [102,60] 102 3.6b1 [43,89,32] 43 3.6b2 [124,30] 124 3.6b3 [74,3] 74 3.6b4 [102,51,60] 102 3.6b5 [136,16] 136 3.6b6 [30,35] 30 3.6b7 [20,127,51] 20 3.6b8 [30,4,54] 30 3.6b9 [127,36] 127 3.6ba [56,106] 56 3.6bb [92,67] 92 3.6bc [7,124] 7 3.6bd [47,148,77] 47 3.6be [86,80,129] 86 3.6bf [88,69] 88 3.6c0 [80,89] 80 3.6c1 [49,36,134] 49 3.6c2 [105,71] 105 3.6c3 [63,50] 63 3.6c4 [2,61,28] 2 3.6c5 [12,90] 12 3.6c6 [138,23] 138 3.6c7 [17,59] 17 3.6c8 [34,100,27] 34 3.6c9 [2,102] 2 3.6ca [130,71,25] 130 3.6cb [20,116] 20 3.6cc [73,8] 73 3.6cd [5,53,88] 5 3.6ce [84,86] 84 3.6cf [35,57] 35 3.6d0 [108,92,43] 108 3.6d1 [83,132] 83 3.6d2 [41,1,2] 41 3.6d3 [98,26] 98 3.6d4 [140,32] 140 3.6d5 [129,94] 129 3.6d6 [18,84,41] 18 3.6d7 [118,29,62] 118 3.6d8 [14,118] 14 3.6d9 [53,115] 53 3.6da [140,8] 140 3.6db [116,81] 116 3.6dc [81,83,132] 81 3.6dd [49,63,127] 49 3.6de [13,120,7] 13 3.6df [51,21,28] 51 3.6e0 [52,50] 52 3.6e1 [50,49,60] 50 3.6e2 [62,7] 62 3.6e3 [48,49,118] 48 3.6e4 [91,43] 91 3.6e5 [10,15] 10 3.6e6 [45,1] 45 3.6e7 [59,35,49] 59 3.6e8 [69,27] 69 3.6e9 [122,66] 122 3.6ea [48,25] 48 3.6eb [80,10] 80 3.6ec [8,49,148] 8 3.6ed [43,24] 43 3.6ee [130,81,36] 130 3.6ef [4,23] 4 3.6f0 [41,7] 41 3.6f1 [42,52,89] 42 3.6f2 [148,29] 148 3.6f3 [146,50] 146 3.6f4 [82,34,64] 82 3.6f5 [124,1] 124 3.6f6 [7,5] 7 3.6f7 [102,55] 102 3.6f8 [115,92] 115 3.6f9 [14,13] 14 3.6fa [136,23] 136 3.6fb [92,26,33] 92 3.6fc [140,6] 140 3.6fd [18,72] 18 3.6fe [102,51,93] 102 3.6ff [60,130,82] 60 3.700 [91,17] 91 3.701 [41,56] 41 3.702 [62,64,148] 62 3.703 [118,2,85] 118 3.704 [3,62] 3 3.705 [150,69,31] 150 3.706 [16,95] 16 3.707 [127,94,28] 127 3.708 [18,110] 18 3.709 [19,16,90] 19 3.70a [124,10] 124 3.70b [76,61] 76 3.70c [130,5] 130 3.70d [79,8] 79 3.70e [8,25,130] 8 3.70f [5,96] 5 3.710 [20,82,75] 20 3.711 [146,77,47] 146 3.712 [13,58] 13 3.713 [45,2] 45 3.714 [48,46,124] 48 3.715 [120,63] 120 3.716 [31,68] 31 3.717 [130,80,79] 130 3.718 [61,56,78] 61 3.719 [88,122,50] 88 3.71a [68,142] 68 3.71b [55,150] 55 3.71c [146,26,80] 146 3.71d [110,45,20] 110 3.71e [6,51,92] 6 3.71f [144,46,56] 144 3.720 [78,46,64] 78 3.721 [74,75,28] 74 3.722 [4,41] 4 3.723 [56,110,74] 56 3.724 [96,73,11] 96 3.725 [30,9,50] 30 3.726 [52,1,85] 52 3.727 [45,87,54] 45 3.728 [43,57] 43 3.729 [67,96,43] 67 3.72a [65,102,26] 65 3.72b [63,132,35] 63 3.72c [25,59,68] 25 3.72d [110,94,51] 110 3.72e [118,81,14] 118 3.72f [77,43] 77 3.730 [63,20] 63 3.731 [90,44,75] 90 3.732 [89,72] 89 3.733 [127,12,35] 127 3.734 [108,54] 108 3.735 [86,37,75] 86 3.736 [38,8,142] 38 3.737 [11,62,24] 11 3.738 [88,86,19] 88 3.739 [7,41,54] 7 3.73a [15,88,124] 15 3.73b [55,102] 55 3.73c [64,53,30] 64 3.73d [64,127] 64 3.73e [110,73,26] 110 3.73f [87,94] 87 3.740 [28,90] 28 3.741 [118,43] 118 3.742 [86,83] 86 3.743 [94,100,8] 94 3.744 [17,95] 17 3.745 [78,93] 78 3.746 [96,62] 96 3.747 [61,118] 61 3.748 [84,30] 84 3.749 [124,14] 124 3.74a [45,86] 45 3.74b [67,76,120] 67 3.74c [110,0,42] 110 3.74d [146,11] 146 3.74e [21,118] 21 3.74f [46,9] 46 3.750 [80,146] 80 3.751 [85,24,79] 85 3.752 [51,14,61] 51 3.753 [51,115] 51 3.754 [3,90] 3 3.755 [67,86,89] 67 3.756 [36,50,9] 36 3.757 [37,90] 37 3.758 [55,17] 55 3.759 [144,37] 144 3.75a [12,25] 12 3.75b [73,71] 73 3.75c [54,71,140] 54 3.75d [140,2] 140 3.75e [80,27] 80 3.75f [110,21] 110 3.760 [110,18] 110 3.761 [25,127,62] 25 3.762 [43,98] 43 3.763 [77,40] 77 3.764 [108,22] 108 3.765 [50,148] 50 3.766 [56,74,116] 56 3.767 [132,92] 132 3.768 [24,26] 24 3.769 [102,69,66] 102 3.76a [19,127,4] 19 3.76b [95,94,129] 95 3.76c [54,85,150] 54 3.76d [106,1,71] 106 3.76e [106,46,20] 106 3.76f [89,138,74] 89 3.770 [142,25] 142 3.771 [58,77,142] 58 3.772 [4,14] 4 3.773 [82,4,67] 82 3.774 [11,85] 11 3.775 [91,100,21] 91 3.776 [28,65] 28 3.777 [21,50] 21 3.778 [68,132] 68 3.779 [148,36] 148 3.77a [49,30,94] 49 3.77b [54,88,73] 54 3.77c [91,105,76] 91 3.77d [56,87] 56 3.77e [61,31,16] 61 3.77f [93,59] 93 3.780 [84,113] 84 3.781 [79,9] 79 3.782 [91,80] 91 3.783 [93,15,144] 93 3.784 [25,2] 25 3.785 [84,98,29] 84 3.786 [120,82,6] 120 3.787 [140,79] 140 3.788 [102,65,32] 102 3.789 [86,56,34] 86 3.78a [81,1,12] 81 3.78b [15,49] 15 3.78c [150,72] 150 3.78d [102,80,26] 102 3.78e [33,32] 33 3.78f [138,29,38] 138 3.790 [30,36] 30 3.791 [120,36] 120 3.792 [49,88] 49 3.793 [113,31,48] 113 3.794 [47,21,38] 47 3.795 [90,42] 90 3.796 [75,49] 75 3.797 [7,70] 7 3.798 [65,19] 65 3.799 [65,52] 65 3.79a [17,146] 17 3.79b [86,26] 86 3.79c [138,33] 138 3.79d [44,116] 44 3.79e [68,10,75] 68 3.79f [72,26,120] 72 3.7a0 [46,52,105] 46 3.7a1 [47,89,12] 47 3.7a2 [4,15,74] 4 3.7a3 [86,134] 86 3.7a4 [106,46,57] 106 3.7a5 [75,89] 75 3.7a6 [21,138] 21 3.7a7 [58,88] 58 3.7a8 [11,113] 11 3.7a9 [42,49] 42 3.7aa [150,78,4] 150 3.7ab [69,36,71] 69 3.7ac [106,53,28] 106 3.7ad [92,110,15] 92 3.7ae [48,57,106] 48 3.7af [95,96,45] 95 3.7b0 [93,23,127] 93 3.7b1 [8,108] 8 3.7b2 [124,79] 124 3.7b3 [22,130,48] 22 3.7b4 [129,8,5] 129 3.7b5 [29,43] 29 3.7b6 [22,21,28] 22 3.7b7 [69,67,70] 69 3.7b8 [56,27] 56 3.7b9 [92,150,57] 92 3.7ba [36,18,73] 36 3.7bb [44,50] 44 3.7bc [90,144,83] 90 3.7bd [63,148] 63 3.7be [9,132,94] 9 3.7bf [31,34,37] 31 3.7c0 [84,98,27] 84 3.7c1 [146,75] 146 3.7c2 [49,68,14] 49 3.7c3 [138,69] 138 3.7c4 [124,74,4] 124 3.7c5 [67,29] 67 3.7c6 [86,148,84] 86 3.7c7 [134,94] 134 3.7c8 [0,94,130] 0 3.7c9 [108,1] 108 3.7ca [29,150] 29 3.7cb [102,20] 102 3.7cc [53,110,10] 53 3.7cd [55,115] 55 3.7ce [0,2] 0 3.7cf [22,127] 22 3.7d0 [19,27,6] 19 3.7d1 [87,134,86] 87 3.7d2 [62,6] 62 3.7d3 [116,33] 116 3.7d4 [19,85] 19 3.7d5 [27,120,91] 27 3.7d6 [25,50,71] 25 3.7d7 [1,63] 1 3.7d8 [84,45,59] 84 3.7d9 [4,86] 4 3.7da [26,0] 26 3.7db [3,70] 3 3.7dc [14,124] 14 3.7dd [59,66] 59 3.7de [81,76] 81 3.7df [46,8] 46 3.7e0 [106,37] 106 3.7e1 [52,79,41] 52 3.7e2 [47,69] 47 3.7e3 [92,74,63] 92 3.7e4 [19,73,62] 19 3.7e5 [90,89,115] 90 3.7e6 [87,102] 87 3.7e7 [15,25,76] 15 3.7e8 [55,80,89] 55 3.7e9 [144,81] 144 3.7ea [29,124] 29 3.7eb [132,11,68] 132 3.7ec [82,80] 82 3.7ed [83,25] 83 3.7ee [61,62] 61 3.7ef [4,57] 4 3.7f0 [144,70,28] 144 3.7f1 [7,52] 7 3.7f2 [12,29] 12 3.7f3 [57,72] 57 3.7f4 [43,56,113] 43 3.7f5 [71,17,110] 71 3.7f6 [144,12] 144 3.7f7 [124,72] 124 3.7f8 [150,81,54] 150 3.7f9 [74,136,57] 74 3.7fa [118,90] 118 3.7fb [13,66] 13 3.7fc [89,144,24] 89 3.7fd [6,98] 6 3.7fe [34,22,49] 34 3.7ff [47,38] 47 #osd count first primary c wt wt osd.0 43 16 16 3.62999 1 osd.1 57 21 21 3.62999 1 osd.2 44 18 18 3.62999 1 osd.3 34 15 15 3.62999 1 osd.4 43 14 14 3.62999 1 osd.5 33 19 19 3.62999 1 osd.6 48 19 19 3.62999 1 osd.7 48 26 26 3.62999 1 osd.8 48 22 22 3.62999 1 osd.9 40 18 18 3.62999 1 osd.10 43 16 16 3.62999 1 osd.11 49 17 17 3.62999 1 osd.12 28 9 9 3.62999 1 osd.13 40 18 18 3.62999 1 osd.14 42 18 18 3.62999 1 osd.15 41 19 19 3.62999 1 osd.16 34 18 18 3.62999 1 osd.17 41 21 21 3.62999 1 osd.18 49 22 22 3.62999 1 osd.19 34 15 15 3.62999 1 osd.20 46 18 18 3.62999 1 osd.21 45 15 15 3.62999 1 osd.22 34 20 20 3.62999 1 osd.23 41 18 18 3.62999 1 osd.24 52 23 23 3.62999 1 osd.25 51 22 22 3.62999 1 osd.26 46 17 17 3.62999 1 osd.27 38 13 13 3.62999 1 osd.28 35 14 14 3.62999 1 osd.29 44 12 12 3.62999 1 osd.30 43 16 16 3.62999 1 osd.31 41 17 17 3.62999 1 osd.32 38 11 11 3.62999 1 osd.33 29 8 8 3.62999 1 osd.34 39 16 16 3.62999 1 osd.35 33 16 16 3.62999 1 osd.36 38 15 15 3.62999 1 osd.37 41 18 18 3.62999 1 osd.38 47 21 21 3.62999 1 osd.40 35 16 16 3.62999 1 osd.41 41 16 16 3.62999 1 osd.42 36 15 15 3.62999 1 osd.43 42 17 17 3.62999 1 osd.44 37 12 12 3.62999 1 osd.45 39 17 17 3.62999 1 osd.46 38 16 16 3.62999 1 osd.47 47 23 23 3.62999 1 osd.48 40 18 18 3.62999 1 osd.49 46 19 19 3.62999 1 osd.50 51 18 18 3.62999 1 osd.51 35 12 12 3.62999 1 osd.52 46 20 20 3.62999 1 osd.53 39 16 16 3.62999 1 osd.54 45 17 17 3.62999 1 osd.55 39 17 17 3.62999 1 osd.56 39 19 19 3.62999 1 osd.57 45 14 14 3.62999 1 osd.58 43 17 17 3.62999 1 osd.59 38 21 21 3.62999 1 osd.60 41 10 10 3.62999 1 osd.61 38 16 16 3.62999 1 osd.62 42 17 17 3.62999 1 osd.63 39 15 15 3.62999 1 osd.64 38 17 17 3.62999 1 osd.65 32 16 16 3.62999 1 osd.66 34 9 9 3.62999 1 osd.67 42 17 17 3.62999 1 osd.68 47 18 18 3.62999 1 osd.69 38 14 14 3.62999 1 osd.70 44 10 10 3.62999 1 osd.71 39 11 11 3.62999 1 osd.72 44 17 17 3.62999 1 osd.73 37 16 16 3.62999 1 osd.74 48 25 25 3.62999 1 osd.75 39 17 17 3.62999 1 osd.76 38 12 12 3.62999 1 osd.77 42 11 11 3.62999 1 osd.78 37 21 21 3.62999 1 osd.79 37 14 14 3.62999 1 osd.80 44 22 22 3.62999 1 osd.81 28 11 11 3.62999 1 osd.82 48 24 24 3.62999 1 osd.83 44 15 15 3.62999 1 osd.84 39 20 20 3.62999 1 osd.85 39 15 15 3.62999 1 osd.86 61 21 21 3.62999 1 osd.87 38 20 20 3.62999 1 osd.88 39 14 14 3.62999 1 osd.89 41 18 18 3.62999 1 osd.90 41 15 15 3.62999 1 osd.91 38 14 14 3.62999 1 osd.92 37 20 20 3.62999 1 osd.93 44 15 15 3.62999 1 osd.94 57 18 18 3.62999 1 osd.95 40 12 12 3.62999 1 osd.96 32 17 17 3.62999 1 osd.97 0 0 0 3.62999 1 osd.98 38 14 14 3.62999 1 osd.99 0 0 0 3.62999 1 osd.100 37 11 11 3.62999 1 osd.101 0 0 0 3.62999 1 osd.102 52 22 22 3.62999 1 osd.103 0 0 0 3.62999 1 osd.104 0 0 0 3.62999 1 osd.105 36 20 20 3.62999 1 osd.106 32 13 13 3.62999 1 osd.107 0 0 0 3.62999 1 osd.108 41 17 17 3.62999 1 osd.109 0 0 0 3.62999 1 osd.110 48 19 19 3.62999 1 osd.111 0 0 0 3.62999 1 osd.112 0 0 0 3.62999 1 osd.113 34 18 18 3.62999 1 osd.114 0 0 0 3.62999 1 osd.115 29 12 12 3.62999 1 osd.116 29 10 10 3.62999 1 osd.117 0 0 0 3.62999 1 osd.118 48 18 18 3.62999 1 osd.119 0 0 0 3.62999 1 osd.120 36 12 12 3.62999 1 osd.121 0 0 0 3.62999 1 osd.122 42 20 20 3.62999 1 osd.123 0 0 0 3.62999 1 osd.124 49 18 18 3.62999 1 osd.125 0 0 0 3.62999 1 osd.126 0 0 0 3.62999 1 osd.127 39 18 18 3.62999 1 osd.128 0 0 0 3.62999 1 osd.129 38 17 17 3.62999 1 osd.130 49 22 22 3.62999 1 osd.131 0 0 0 3.62999 1 osd.132 47 15 15 3.62999 1 osd.133 0 0 0 3.62999 1 osd.134 31 12 12 3.62999 1 osd.135 0 0 0 3.62999 1 osd.136 40 18 18 3.62999 1 osd.137 0 0 0 3.62999 1 osd.138 31 15 15 3.62999 1 osd.139 0 0 0 3.62999 1 osd.140 34 20 20 3.62999 1 osd.141 0 0 0 3.62999 1 osd.142 40 10 10 3.62999 1 osd.143 0 0 0 3.62999 1 osd.144 44 19 19 3.62999 1 osd.145 0 0 0 3.62999 1 osd.146 38 21 21 3.62999 1 osd.147 0 0 0 3.62999 1 osd.148 40 14 14 3.62999 1 osd.149 0 0 0 3.62999 1 osd.150 41 18 18 3.62999 1 osd.151 0 0 0 3.62999 1 in 151 avg 33 stddev 16.7417 (0.507324x) (expected 5.741 0.17397x)) min osd.12 28 max osd.86 61 size 0 0 size 1 0 size 2 1134 size 3 914 Matt Conner Keeper Technology On Tue, Dec 8, 2015 at 5:35 AM, Ilya Dryomov <idryomov@xxxxxxxxx> wrote: > > On Tue, Dec 8, 2015 at 10:57 AM, Tom Christensen <pavera@xxxxxxxxx> wrote: > > We aren't running NFS, but regularly use the kernel driver to map RBDs and > > mount filesystems in same. We see very similar behavior across nearly all > > kernel versions we've tried. In my experience only very few versions of the > > kernel driver survive any sort of crush map change/update while something is > > mapped. In fact in the last 2 years I think I've only seen this work on 1 > > kernel version unfortunately its badly out of date and we can't run it in > > our environment anymore, I think it was a 3.0 kernel version running on > > ubuntu 12.04. We have just recently started trying to find a kernel that > > will survive OSD outages or changes to the cluster. We're on ubuntu 14.04, > > and have tried 3.16, 3.19.0-25, 4.3, and 4.2 without success in the last > > week. We only map 1-3 RBDs per client machine at a time but we regularly > > will get processes stuck in D state which are accessing the filesystem > > inside the RBD and will have to hard reboot the RBD client machine. This is > > always associated with a cluster change in some way, reweighting OSDs, > > rebooting an OSD host, restarting an individual OSD, adding OSDs, and > > removing OSDs all cause the kernel client to hang. If no change is made to > > the cluster, the kernel client will be happy for weeks. > > There are a couple of known bugs in the remap/resubmit area, but those > are supposedly corner cases (like *all* the OSDs going down and then > back up, etc). I had no idea it was that severe and goes that back. > Apparently triggering it requires a heavier load, as we've never seen > anything like that in our tests. > > For unrelated reasons, remap/resubmit code is getting entirely > rewritten for kernel 4.5, so, if you've been dealing with this issue > for the last two years (I don't remember seeing any tickets listing > that many kernel versions and not mentioning NFS), I'm afraid the best > course of action for you would be to wait for 4.5 to come out and try > it. If you'd be willing to test out an early version on one of more of > your client boxes, I can ping you when it's ready. > > I'll take a look at 3.0 vs 3.16 with an eye on remap code. Did you > happen to try 3.10? > > It sounds like you can reproduce this pretty easily. Can you get it to > lock up and do: > > # cat /sys/kernel/debug/ceph/*/osdmap > # cat /sys/kernel/debug/ceph/*/osdc > $ ceph status > > and bunch of times? I have a hunch that kernel client simply fails to > request enough of new osdmaps after the cluster topology changes under > load. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list > ceph-users@xxxxxxxxxxxxxx > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |