Sorry, typo in topic, this is with 12.2.2. 12.2.1 was ok until update last friday to 12.2.2.
Regards, Tobi
On 12/13/2017 11:45 AM, Tobias Prousa wrote:
Hi there,
sorry to disturb you again but I'm still not there. After restoring my CephFS to a working state (with a lot of help from Yan, Zheng, thank
you so much), I got my CephFS back working by restarting MDSs and ramping up first clients to use it. Everything looked promising for about 1
hour. Then suddenly MDSs started failing, standbys dropped in until all standbys were used up and last MDSs failed.
They now fail with an assertion like that:
-99> 2017-12-13 11:00:30.043646 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1080 0x55b9f10c9900
client_request(client.59196208:200244 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002, caller_gid=1002{}) v2
-98> 2017-12-13 11:00:30.043675 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1080 ====
client_request(client.59196208:200244 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1717554603 0 0) 0x55b9f10c9900 con 0x55b9f3adc800
-97> 2017-12-13 11:00:30.043691 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200244 setfilelockrule 1,
type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002,
caller_gid=1002{}) v2
-96> 2017-12-13 11:00:30.043736 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200244 = 0
(0) Success safe) v1 -- 0x55b9f13719c0 con 0
-95> 2017-12-13 11:00:30.044261 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1081 0x55b9f10c9000
client_request(client.59196208:200245 setfilelockrule 1, type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002, caller_gid=1002{}) v2
-94> 2017-12-13 11:00:30.044331 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1081 ====
client_request(client.59196208:200245 setfilelockrule 1, type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2047937405 0 0) 0x55b9f10c9000 con 0x55b9f3adc800
-93> 2017-12-13 11:00:30.044365 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200245 setfilelockrule 1,
type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.038330 caller_uid=1002,
caller_gid=1002{}) v2
-92> 2017-12-13 11:00:30.044445 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200245 = 0
(0) Success safe) v1 -- 0x55b9f13703c0 con 0
-91> 2017-12-13 11:00:30.045543 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1082 0x55b9f10c9600
client_request(client.59196208:200246 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2
-90> 2017-12-13 11:00:30.045598 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1082 ====
client_request(client.59196208:200246 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2923553120 0 0) 0x55b9f10c9600 con 0x55b9f3adc800
-89> 2017-12-13 11:00:30.045618 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200246 setfilelockrule 1,
type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002,
caller_gid=1002{}) v2
-88> 2017-12-13 11:00:30.045679 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200246 = 0
(0) Success safe) v1 -- 0x55b9f1370c00 con 0
-87> 2017-12-13 11:00:30.048175 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1083 0x55b9f10c6000
client_request(client.59196208:200247 setfilelockrule 1, type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2
-86> 2017-12-13 11:00:30.048229 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1083 ====
client_request(client.59196208:200247 setfilelockrule 1, type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2987337654 0 0) 0x55b9f10c6000 con 0x55b9f3adc800
-85> 2017-12-13 11:00:30.048253 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200247 setfilelockrule 1,
type 1, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002,
caller_gid=1002{}) v2
-84> 2017-12-13 11:00:30.048305 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200247 = 0
(0) Success safe) v1 -- 0x55b9f136f340 con 0
-83> 2017-12-13 11:00:30.048790 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1084 0x55b9f10c7b00
client_request(client.59196208:200248 setfilelockrule 1, type 2, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2
-82> 2017-12-13 11:00:30.048900 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1084 ====
client_request(client.59196208:200248 setfilelockrule 1, type 2, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2428075346 0 0) 0x55b9f10c7b00 con 0x55b9f3adc800
-81> 2017-12-13 11:00:30.048919 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200248 setfilelockrule 1,
type 2, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.042330 caller_uid=1002,
caller_gid=1002{}) v2
-80> 2017-12-13 11:00:30.048963 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200248 = 0
(0) Success safe) v1 -- 0x55b9f136eb00 con 0
-79> 2017-12-13 11:00:30.069855 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1085 0x55b9f10c6c00
client_request(client.59196208:200249 setfilelockrule 1, type 2, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-78> 2017-12-13 11:00:30.069893 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1085 ====
client_request(client.59196208:200249 setfilelockrule 1, type 2, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1839967222 0 0) 0x55b9f10c6c00 con 0x55b9f3adc800
-77> 2017-12-13 11:00:30.069910 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200249 setfilelockrule 1,
type 2, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-76> 2017-12-13 11:00:30.069956 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200249 = 0
(0) Success safe) v1 -- 0x55b9f136f080 con 0
-75> 2017-12-13 11:00:30.069975 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1086 0x55b9f10c9300
client_request(client.59196208:200250 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-74> 2017-12-13 11:00:30.070003 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1086 ====
client_request(client.59196208:200250 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1409035243 0 0) 0x55b9f10c9300 con 0x55b9f3adc800
-73> 2017-12-13 11:00:30.070016 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200250 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-72> 2017-12-13 11:00:30.070051 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200250 = 0
(0) Success safe) v1 -- 0x55b9f3013180 con 0
-71> 2017-12-13 11:00:30.070323 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1087 0x55b9f10c7e00
client_request(client.59196208:200251 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-70> 2017-12-13 11:00:30.070349 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1087 ====
client_request(client.59196208:200251 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2256862525 0 0) 0x55b9f10c7e00 con 0x55b9f3adc800
-69> 2017-12-13 11:00:30.070365 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200251 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-68> 2017-12-13 11:00:30.070404 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200251 = 0
(0) Success safe) v1 -- 0x55b9f312b9c0 con 0
-67> 2017-12-13 11:00:30.070759 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1088 0x55b9f10c8a00
client_request(client.59196208:200252 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-66> 2017-12-13 11:00:30.070799 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1088 ====
client_request(client.59196208:200252 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1337126205 0 0) 0x55b9f10c8a00 con 0x55b9f3adc800
-65> 2017-12-13 11:00:30.070815 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200252 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-64> 2017-12-13 11:00:30.070827 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1089 0x55b9f1407e00
client_request(client.59196208:200253 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-63> 2017-12-13 11:00:30.070854 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200252 = 0
(0) Success safe) v1 -- 0x55b9f312aec0 con 0
-62> 2017-12-13 11:00:30.070897 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1089 ====
client_request(client.59196208:200253 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2596944875 0 0) 0x55b9f1407e00 con 0x55b9f3adc800
-61> 2017-12-13 11:00:30.070910 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200253 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-60> 2017-12-13 11:00:30.070947 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200253 = 0
(0) Success safe) v1 -- 0x55b9f2ef63c0 con 0
-59> 2017-12-13 11:00:30.071345 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1090 0x55b9f1409300
client_request(client.59196208:200254 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2
-58> 2017-12-13 11:00:30.071370 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1090 ====
client_request(client.59196208:200254 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2256862525 0 0) 0x55b9f1409300 con 0x55b9f3adc800
-57> 2017-12-13 11:00:30.071384 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200254 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.066330 caller_uid=1002,
caller_gid=1002{}) v2
-56> 2017-12-13 11:00:30.071423 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200254 = 0
(0) Success safe) v1 -- 0x55b9f2f48000 con 0
-55> 2017-12-13 11:00:30.083353 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1091 0x55b9f1408400
client_request(client.59196208:200255 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002, caller_gid=1002{}) v2
-54> 2017-12-13 11:00:30.083485 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1091 ====
client_request(client.59196208:200255 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2051456776 0 0) 0x55b9f1408400 con 0x55b9f3adc800
-53> 2017-12-13 11:00:30.083537 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200255 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002,
caller_gid=1002{}) v2
-52> 2017-12-13 11:00:30.083589 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200255 = 0
(0) Success safe) v1 -- 0x55b9f354e940 con 0
-51> 2017-12-13 11:00:30.084028 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1092 0x55b9f1408100
client_request(client.59196208:200256 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002, caller_gid=1002{}) v2
-50> 2017-12-13 11:00:30.084074 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1092 ====
client_request(client.59196208:200256 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1711831518 0 0) 0x55b9f1408100 con 0x55b9f3adc800
-49> 2017-12-13 11:00:30.084090 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200256 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.078330 caller_uid=1002,
caller_gid=1002{}) v2
-48> 2017-12-13 11:00:30.084135 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200256 = 0
(0) Success safe) v1 -- 0x55b9f3b9d600 con 0
-47> 2017-12-13 11:00:30.085563 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1093 0x55b9f1407b00
client_request(client.59196208:200257 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2
-46> 2017-12-13 11:00:30.085661 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1093 ====
client_request(client.59196208:200257 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2049276777 0 0) 0x55b9f1407b00 con 0x55b9f3adc800
-45> 2017-12-13 11:00:30.085691 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200257 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002,
caller_gid=1002{}) v2
-44> 2017-12-13 11:00:30.085744 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200257 = 0
(0) Success safe) v1 -- 0x55b9efd52dc0 con 0
-43> 2017-12-13 11:00:30.086319 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1094 0x55b9f1407800
client_request(client.59196208:200258 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2
-42> 2017-12-13 11:00:30.086347 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1094 ====
client_request(client.59196208:200258 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1718336959 0 0) 0x55b9f1407800 con 0x55b9f3adc800
-41> 2017-12-13 11:00:30.086365 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200258 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002,
caller_gid=1002{}) v2
-40> 2017-12-13 11:00:30.086420 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200258 = 0
(0) Success safe) v1 -- 0x55b9efd53e40 con 0
-39> 2017-12-13 11:00:30.088484 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1095 0x55b9f1407500
client_request(client.59196208:200259 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2
-38> 2017-12-13 11:00:30.088590 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1095 ====
client_request(client.59196208:200259 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2049276777 0 0) 0x55b9f1407500 con 0x55b9f3adc800
-37> 2017-12-13 11:00:30.088613 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200259 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002,
caller_gid=1002{}) v2
-36> 2017-12-13 11:00:30.088660 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200259 = 0
(0) Success safe) v1 -- 0x55b9efd543c0 con 0
-35> 2017-12-13 11:00:30.089174 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1096 0x55b9f1407200
client_request(client.59196208:200260 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2
-34> 2017-12-13 11:00:30.089207 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1096 ====
client_request(client.59196208:200260 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1718336959 0 0) 0x55b9f1407200 con 0x55b9f3adc800
-33> 2017-12-13 11:00:30.089222 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200260 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1880 2017-12-13 11:00:30.082330 caller_uid=1002,
caller_gid=1002{}) v2
-32> 2017-12-13 11:00:30.089269 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200260 = 0
(0) Success safe) v1 -- 0x55b9efd55700 con 0
-31> 2017-12-13 11:00:30.092777 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1097 0x55b9f1406f00
client_request(client.59196208:200261 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.086331 caller_uid=1002, caller_gid=1002{}) v2
-30> 2017-12-13 11:00:30.092831 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1098 0x55b9f1406c00
client_request(client.59196208:200262 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.086331 caller_uid=1002, caller_gid=1002{}) v2
-29> 2017-12-13 11:00:30.092877 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1097 ====
client_request(client.59196208:200261 setfilelockrule 1, type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.086331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (290870685 0 0) 0x55b9f1406f00 con 0x55b9f3adc800
-28> 2017-12-13 11:00:30.092904 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200261 setfilelockrule 1,
type 1, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.086331 caller_uid=1002,
caller_gid=1002{}) v2
-27> 2017-12-13 11:00:30.092954 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200261 = 0
(0) Success safe) v1 -- 0x55b9f3bc6c00 con 0
-26> 2017-12-13 11:00:30.093006 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1098 ====
client_request(client.59196208:200262 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.086331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (67147728 0 0) 0x55b9f1406c00 con 0x55b9f3adc800
-25> 2017-12-13 11:00:30.093020 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200262 setfilelockrule 1,
type 4, owner 9597116132936561456, pid 5469, start 120, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.086331 caller_uid=1002,
caller_gid=1002{}) v2
-24> 2017-12-13 11:00:30.093064 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200262 = 0
(0) Success safe) v1 -- 0x55b9f3bc6ec0 con 0
-23> 2017-12-13 11:00:30.093322 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1099 0x55b9f1406600
client_request(client.59196208:200263 setfilelockrule 1, type 2, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2
-22> 2017-12-13 11:00:30.093350 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1099 ====
client_request(client.59196208:200263 setfilelockrule 1, type 2, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1529401609 0 0) 0x55b9f1406600 con 0x55b9f3adc800
-21> 2017-12-13 11:00:30.093366 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200263 setfilelockrule 1,
type 2, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002,
caller_gid=1002{}) v2
-20> 2017-12-13 11:00:30.093402 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200263 = 0
(0) Success safe) v1 -- 0x55b9f3bc6940 con 0
-19> 2017-12-13 11:00:30.093571 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1100 0x55b9f1408700
client_request(client.59196208:200264 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2
-18> 2017-12-13 11:00:30.093616 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1100 ====
client_request(client.59196208:200264 setfilelockrule 1, type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0
#0x1000ca0a5a4 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (3105850201 0 0) 0x55b9f1408700 con 0x55b9f3adc800
-17> 2017-12-13 11:00:30.093631 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200264 setfilelockrule 1,
type 4, owner 9597116132936561456, pid 5469, start 124, length 1, wait 0 #0x1000ca0a5a4 2017-12-13 11:00:30.090331 caller_uid=1002,
caller_gid=1002{}) v2
-16> 2017-12-13 11:00:30.093673 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200264 = 0
(0) Success safe) v1 -- 0x55b9f2f2a100 con 0
-15> 2017-12-13 11:00:30.095641 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1101 0x55b9f1408a00
client_request(client.59196208:200265 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2
-14> 2017-12-13 11:00:30.095737 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1101 ====
client_request(client.59196208:200265 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (2953283522 0 0) 0x55b9f1408a00 con 0x55b9f3adc800
-13> 2017-12-13 11:00:30.095767 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200265 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 120, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002,
caller_gid=1002{}) v2
-12> 2017-12-13 11:00:30.095820 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200265 = 0
(0) Success safe) v1 -- 0x55b9f2f29e40 con 0
-11> 2017-12-13 11:00:30.096168 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1102 0x55b9f1408d00
client_request(client.59196208:200266 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2
-10> 2017-12-13 11:00:30.096196 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1102 ====
client_request(client.59196208:200266 setfilelockrule 1, type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0
#0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002, caller_gid=1002{}) v2 ==== 122+0+0 (1708150075 0 0) 0x55b9f1408d00 con 0x55b9f3adc800
-9> 2017-12-13 11:00:30.096212 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200266 setfilelockrule 1,
type 4, owner 9597116132980272432, pid 5408, start 124, length 1, wait 0 #0x1000d3b1916 2017-12-13 11:00:30.090331 caller_uid=1002,
caller_gid=1002{}) v2
-8> 2017-12-13 11:00:30.096254 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200266 = 0
(0) Success safe) v1 -- 0x55b9f2f2a680 con 0
-7> 2017-12-13 11:00:30.245683 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1103 0x55b9f1409000
client_request(client.59196208:200267 lookup #0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
-6> 2017-12-13 11:00:30.245750 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1103 ====
client_request(client.59196208:200267 lookup #0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
==== 136+0+0 (3801526274 0 0) 0x55b9f1409000 con 0x55b9f3adc800
-5> 2017-12-13 11:00:30.245762 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200267 lookup
#0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
-4> 2017-12-13 11:00:30.245793 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 --> 172.16.1.6:0/1672703426 -- client_reply(???:200267 = -2
(2) No such file or directory) v1 -- 0x55b9f2f298c0 con 0
-3> 2017-12-13 11:00:30.246451 7fe6f7ef8700 5 -- 172.16.0.2:6800/4136361304 >> 172.16.1.6:0/1672703426 conn(0x55b9f3adc800 :6800
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=780 cs=1 l=0). rx client.59196208 seq 1104 0x55b9f1406900
client_request(client.59196208:200268 create #0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
-2> 2017-12-13 11:00:30.246488 7fe6f5758700 1 -- 172.16.0.2:6800/4136361304 <== client.59196208 172.16.1.6:0/1672703426 1104 ====
client_request(client.59196208:200268 create #0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
==== 136+0+0 (923668220 0 0) 0x55b9f1406900 con 0x55b9f3adc800
-1> 2017-12-13 11:00:30.246499 7fe6f5758700 4 mds.0.server handle_client_request client_request(client.59196208:200268 create
#0x100095414b7/prefs.xml.save 2017-12-13 11:00:30.242333 caller_uid=1002, caller_gid=1002{}) v2
0> 2017-12-13 11:00:30.249799 7fe6f5758700 -1 /build/ceph-12.2.2/src/mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)'
thread 7fe6f5758700 time 2017-12-13 11:00:30.246563
/build/ceph-12.2.2/src/mds/MDCache.cc: 258: FAILED assert(inode_map.count(in->vino()) == 0)
ceph version 12.2.2 (cf0baeeeeba3b47f9427c6c97e2144b094b7e5ba) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x55b9e585baf2]
2: (MDCache::add_inode(CInode*)+0x285) [0x55b9e5601495]
3: (Server::prepare_new_inode(boost::intrusive_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t*)+0x1089) [0x55b9e55ac169]
4: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0xf86) [0x55b9e55ae1c6]
5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xcd0) [0x55b9e55bfc50]
6: (Server::handle_client_request(MClientRequest*)+0x2b6) [0x55b9e55bff86]
7: (Server::dispatch(Message*)+0x37b) [0x55b9e55c474b]
8: (MDSRank::handle_deferrable_message(Message*)+0x7fc) [0x55b9e553a15c]
9: (MDSRank::_dispatch(Message*, bool)+0x1db) [0x55b9e554756b]
10: (MDSRankDispatcher::ms_dispatch(Message*)+0x15) [0x55b9e5548335]
11: (MDSDaemon::ms_dispatch(Message*)+0xf3) [0x55b9e5531b73]
12: (DispatchQueue::entry()+0x7ca) [0x55b9e5b58a9a]
13: (DispatchQueue::DispatchThread::entry()+0xd) [0x55b9e58e05fd]
14: (()+0x8064) [0x7fe6fa563064]
15: (clone()+0x6d) [0x7fe6f963362d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
I now reduced load by removing most active clients again, only 3 mostly idle clients remaining and MDS keeps up and running again. As soon as
I start adding clients trouble returns.
As long as MDS is up data consistency looks good, there simply seems to be something in the objects that upsets MDS to assertion.
Any idea anyone?
Again, this all started to happen after I "trivially" upgraded my long-lived cluster (deployes back in bobtail age) from 12.2.1 to 12.2.2 last
friday. Btw. the cluster went through all versions from bobtail up to luminous whenever a new version was officially released.
Thank you so much for any pointer!
Best regards,
Tobi
--
-----------------------------------------------------------
Dipl.-Inf. (FH) Tobias Prousa
Leiter Entwicklung Datenlogger
CAETEC GmbH
Industriestr. 1
D-82140 Olching
www.caetec.de
Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Olching
Handelsregister: Amtsgericht München, HRB 183929
Geschäftsführung: Stephan Bacher, Andreas Wocke
Tel.: +49 (0)8142 / 50 13 60
Fax.: +49 (0)8142 / 50 13 69
eMail: tobias.prousa@xxxxxxxxx
Web: http://www.caetec.de
------------------------------------------------------------
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
