Hello, Got this kind of logs in two node of 3 node cluster both node has 2 OSD, only affected 2 OSD on two separate node thats why i dont understand the situation. There wasnt any extra io on the system at the given time. Using radosgw with s3 api to store objects under ceph average ops around 20-150 and bw usage 100-2000kb read / sec and only 50-1000kb / sec written. A few lines from the log (this is OSD.5 and more log entries on OSD.2) 2014-05-22 18:41:03.725702 7ff85cc07700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 30.093011 secs 2014-05-22 18:41:03.725721 7ff85cc07700 0 log [WRN] : slow request 30.093011 seconds old, received at 2014-05-22 18:40:33.628042: osd_op(client.7821.0:67251068 default.4181.1_products/800x600/537e28022fdcc.jpg [cmpxattr user.rgw.idtag (22) op 1 mode 1,setxattr user.rgw.idtag (33),call refcount.put] 11.fe53a6fb e590) v4 currently waiting for subops from [2] 2014-05-22 18:41:33.730590 7ff85cc07700 0 log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 60.102500 secs 2014-05-22 18:41:33.730602 7ff85cc07700 0 log [WRN] : slow request 60.102500 seconds old, received at 2014-05-22 18:40:33.628042: osd_op(client.7821.0:67251068 default.4181.1_products/800x600/537e28022fdcc.jpg [cmpxattr user.rgw.idtag (22) op 1 mode 1,setxattr user.rgw.idtag (33),call refcount.put] 11.fe53a6fb e590) v4 currently waiting for subops from [2] 2014-05-22 18:41:34.730785 7ff85cc07700 0 log [WRN] : 2 slow requests, 1 included below; oldest blocked for > 61.102703 secs 2014-05-22 18:41:34.730805 7ff85cc07700 0 log [WRN] : slow request 30.113226 seconds old, received at 2014-05-22 18:41:04.617519: osd_op(client.7821.0:67251426 default.4181.1_products/800x600/537e28022fdcc.jpg [getxattrs,stat] 11.fe53a6fb e590) v4 currently waiting for rw locks 2014-05-22 18:42:04.735887 7ff85cc07700 0 log [WRN] : 2 slow requests, 1 included below; oldest blocked for > 91.107830 secs 2014-05-22 18:42:04.735890 7ff85cc07700 0 log [WRN] : slow request 60.118353 seconds old, received at 2014-05-22 18:41:04.617519: osd_op(client.7821.0:67251426 default.4181.1_products/800x600/537e28022fdcc.jpg [getxattrs,stat] 11.fe53a6fb e590) v4 currently waiting for rw locks 2014-05-22 18:42:06.736279 7ff85cc07700 0 log [WRN] : 3 slow requests, 1 included below; oldest blocked for > 93.108188 secs 2014-05-22 18:42:06.736298 7ff85cc07700 0 log [WRN] : slow request 30.085101 seconds old, received at 2014-05-22 18:41:36.651129: osd_op(client.7821.0:67251757 default.4181.1_products/800x600/537e28022fdcc.jpg [getxattrs,stat] 11.fe53a6fb e590) v4 currently waiting for rw locks ... some more on OSD.5 -- Gy?rv?ri G?bor - Scr34m scr34m at frontember.hu
 |