I'm having some issues downloading a big file (60G+).
After some investigation it seems to be very similar to http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001272.html, however I'm currently running Hammer 0.94.3. However the files were uploaded when the cluster was running Firefly (IIRC 0.80.7).
From the radosgw --debug-ms=1 --debug-rgw=20
[...]
2015-09-28 16:08:14.596299 7f6361f93700 1 -- 10.248.33.13:0/1004902 --> 10.249.33.12:6814/15506 -- osd_op(client.129331531.0:817 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_2 [read 0~4194304] 9.7a8bb21d ack+read+known_if_redirected e59831) v5 -- ?+0 0x7f63e507aca0 con 0x7f63e4d2ef50
2015-09-28 16:08:14.596361 7f6361f93700 20 rados->aio_operate r=0 bl.length=0
2015-09-28 16:08:14.596371 7f6361f93700 20 rados->get_obj_iterate_cb oid=default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_3 obj-ofs=1795162112 read_ofs=0 len=4194304
2015-09-28 16:08:14.596385 7f6361f93700 1 -- 10.248.33.13:0/1004902 --> 10.249.33.14:6825/14173 -- osd_op(client.129331531.0:818 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_3 [read 0~4194304] 9.6407682a ack+read+known_if_redirected e59831) v5 -- ?+0 0x7f63e4e77370 con 0x7f63e4d8aa60
2015-09-28 16:08:14.596421 7f6361f93700 20 rados->aio_operate r=0 bl.length=0
2015-09-28 16:08:14.596445 7f6361f93700 20 rados->get_obj_iterate_cb oid=default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_4 obj-ofs=1799356416 read_ofs=0 len=4194304
2015-09-28 16:08:14.596474 7f6361f93700 1 -- 10.248.33.13:0/1004902 --> 10.249.33.39:6805/28145 -- osd_op(client.129331531.0:819 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_4 [read 0~4194304] 9.ceaa43a4 ack+read+known_if_redirected e59831) v5 -- ?+0 0x7f63e4e77370 con 0x7f63e4f94220
2015-09-28 16:08:14.596502 7f6361f93700 20 rados->aio_operate r=0 bl.length=0
2015-09-28 16:08:14.597073 7f6337d30700 1 -- 10.248.33.13:0/1004902 <== osd.48 10.248.33.12:6821/27677 9 ==== osd_op_reply(816 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_1 [read 0~4194304] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 260+0+0 (3636490520 0 0) 0x7f6414011d30 con 0x7f63e4d9dc10
2015-09-28 16:08:14.597152 7f640bfff700 20 get_obj_aio_completion_cb: io completion ofs=1786773504 len=4194304
2015-09-28 16:08:14.597159 7f640bfff700 0 ERROR: got unexpected error when trying to read object: -2
2015-09-28 16:08:14.597184 7f6361f93700 20 get_obj_data::cancel_all_io()
2015-09-28 16:08:14.597234 7f6339641700 1 -- 10.248.33.13:0/1004902 <== osd.42 10.249.33.12:6814/15506 8 ==== osd_op_reply(817 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_2 [read 0~4194304] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 260+0+0 (4010761036 0 0) 0x7f63fc011820 con 0x7f63e4d2ef50
2015-09-28 16:08:14.597257 7f640bfff700 20 get_obj_aio_completion_cb: io completion ofs=1790967808 len=4194304
2015-09-28 16:08:14.597263 7f640bfff700 0 ERROR: got unexpected error when trying to read object: -2
2015-09-28 16:08:14.597298 7f6336320700 1 -- 10.248.33.13:0/1004902 <== osd.23 10.249.33.14:6825/14173 8 ==== osd_op_reply(818 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_3 [read 0~4194304] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 260+0+0 (8601650 0 0) 0x7f63ec015050 con 0x7f63e4d8aa60
2015-09-28 16:08:14.597323 7f640bfff700 20 get_obj_aio_completion_cb: io completion ofs=1795162112 len=4194304
2015-09-28 16:08:14.597326 7f640bfff700 0 ERROR: got unexpected error when trying to read object: -2
2015-09-28 16:08:14.597177 7f6336d26700 1 -- 10.248.33.13:0/1004902 <== osd.66 10.249.33.39:6805/28145 9 ==== osd_op_reply(819 default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_4 [read 0~4194304] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 260+0+0 (3449160520 0 0) 0x7f63d80173e0 con 0x7f63e4f94220
2015-09-28 16:08:14.597338 7f640bfff700 20 get_obj_aio_completion_cb: io completion ofs=1799356416 len=4194304
2015-09-28 16:08:14.597339 7f640bfff700 0 ERROR: got unexpected error when trying to read object: -2
2015-09-28 16:08:14.597507 7f6361f93700 2 req 1:32.448590:s3:GET /Exchange%20Stores/Directors.edb:get_obj:http status=404
2015-09-28 16:08:14.597515 7f6361f93700 1 ====== req done req=0x7f6420019d70 http_status=404 ======
doing a rados ls -p .rgw.buckets and searching for the ".35_" parts I only get these...
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_11
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_10
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_12
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_8
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_7
default.26848827.2__shadow_Exchange Stores/Directors.edb.WA70vanNXi08cmwrdmURjXFIUCgaIVv.35_9
Does this mean that the file is incomplete? What could've happened with it? I've had a few dead disks on this cluster, could the missing files be removed because of it?
thanks,
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
