That did the trick, we had it set to 0 just on the swift rgw definitions although it was set on other rgw services, I'm guessing someone must have thought there was a different precedence in play in the past.
On Tue, 2018-12-11 at 11:41 -0500, Casey Bodley wrote:
Hi Leon,Are you running with a non-default value of rgw_gc_max_objs? I was ableto reproduce this exact stack trace by setting rgw_gc_max_objs = 0; Ican't think of any other way to get a 'Floating point exception' here.On 12/11/18 10:31 AM, Leon Robinson wrote:Hello, I have found a surefire way to bring down our swift gateways.First, upload a bunch of large files and split it in to segments, e.g.for i in {1..100}; do swift upload test_container -S 10485760CentOS-7-x86_64-GenericCloud.qcow2 --object-nameCentOS-7-x86_64-GenericCloud.qcow2-$i; doneThis creates 100 objects in test_container and 1000 or so objects intest_container_segmentsThen, Delete them. Preferably in a ludicrous manner.for i in $(swift list test_container); do swift delete test_container$i; doneWhat results is:-13> 2018-12-11 15:17:57.627655 7fc128b49700 1 --172.28.196.121:0/464072497 <== osd.480 172.26.212.6:6802/2058882 1==== osd_op_reply(11 .dir.default.1083413551.2.7 [call,call]v1423252'7548804 uv7548804 _ondisk_ = 0) v8 ==== 213+0+0 (3895049453 00) 0x55c98f45e9c0 con 0x55c98f4d7800-12> 2018-12-11 15:17:57.627827 7fc0e3ffe700 1 --172.28.196.121:0/464072497 --> 172.26.221.7:6816/2366816 --osd_op(unknown.0.0:12 14.110b14:d08c26b8:::default.1083413551.2_CentOS-7-x86_64-GenericCloud.qcow2-10%2f1532606905.440697%2f938016768%2f10485760%2f00000037:head[cmpxattr user.rgw.idtag (25) op 1 mode 1,call rgw.obj_remove] snapc0=[] ondisk+write+known_if_redirected e1423252) v8 -- 0x55c98f4603c0 con 0-11> 2018-12-11 15:17:57.628582 7fc128348700 5 --172.28.196.121:0/157062182 >> 172.26.225.9:6828/2257653conn(0x55c98f0eb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCHpgs=540 cs=1 l=1). rx osd.87 seq 2 0x55c98f4603c0 osd_op_reply(340obj_delete_at_hint.0000000055 [call] v1423252'9217746 uv9217746 ondisk= 0) v8-10> 2018-12-11 15:17:57.628604 7fc128348700 1 --172.28.196.121:0/157062182 <== osd.87 172.26.225.9:6828/2257653 2 ====osd_op_reply(340 obj_delete_at_hint.0000000055 [call] v1423252'9217746uv9217746 _ondisk_ = 0) v8 ==== 173+0+0 (3971813511 0 0) 0x55c98f4603c0con 0x55c98f0eb000-9> 2018-12-11 15:17:57.628760 7fc1017f9700 1 --172.28.196.121:0/157062182 --> 172.26.225.9:6828/2257653 --osd_op(unknown.0.0:341 13.4f13:f3db1134:::obj_delete_at_hint.0000000055:head [call timeindex.list]snapc 0=[] ondisk+read+known_if_redirected e1423252) v8 --0x55c98f45fa00 con 0-8> 2018-12-11 15:17:57.629306 7fc128348700 5 --172.28.196.121:0/157062182 >> 172.26.225.9:6828/2257653conn(0x55c98f0eb000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCHpgs=540 cs=1 l=1). rx osd.87 seq 3 0x55c98f45fa00 osd_op_reply(341obj_delete_at_hint.0000000055 [call] v0'0 uv9217746 _ondisk_ = 0) v8-7> 2018-12-11 15:17:57.629326 7fc128348700 1 --172.28.196.121:0/157062182 <== osd.87 172.26.225.9:6828/2257653 3 ====osd_op_reply(341 obj_delete_at_hint.0000000055 [call] v0'0 uv9217746_ondisk_ = 0) v8 ==== 173+0+15 (3272189389 0 2149983739) 0x55c98f45fa00con 0x55c98f0eb000-6> 2018-12-11 15:17:57.629398 7fc128348700 5 --172.28.196.121:0/464072497 >> 172.26.221.7:6816/2366816conn(0x55c98f4d6000 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCHpgs=181 cs=1 l=1). rx osd.58 seq 2 0x55c98f45fa00 osd_op_reply(12default.1083413551.2_CentOS-7-x86_64-GenericCloud.qcow2-10/1532606905.440697/938016768/10485760/00000037[cmpxattr (25) op 1 mode 1,call] v1423252'743755 uv743755 _ondisk_ = 0) v8-5> 2018-12-11 15:17:57.629418 7fc128348700 1 --172.28.196.121:0/464072497 <== osd.58 172.26.221.7:6816/2366816 2 ====osd_op_reply(12default.1083413551.2_CentOS-7-x86_64-GenericCloud.qcow2-10/1532606905.440697/938016768/10485760/00000037[cmpxattr (25) op 1 mode 1,call] v1423252'743755 uv743755 _ondisk_ = 0)v8 ==== 290+0+0 (3763879162 0 0) 0x55c98f45fa00 con 0x55c98f4d6000-4> 2018-12-11 15:17:57.629458 7fc1017f9700 1 --172.28.196.121:0/157062182 --> 172.26.225.9:6828/2257653 --osd_op(unknown.0.0:342 13.4f13:f3db1134:::obj_delete_at_hint.0000000055:head [call lock.unlock]snapc 0=[] ondisk+write+known_if_redirected e1423252) v8 --0x55c98f45fd40 con 0-3> 2018-12-11 15:17:57.629603 7fc0e3ffe700 1 --172.28.196.121:0/464072497 --> 172.26.212.6:6802/2058882 --osd_op(unknown.0.0:13 15.1e015:079bdcbb:::.dir.default.1083413551.2.7:head [callrgw.guard_bucket_resharding,call rgw.bucket_complete_op] snapc 0=[]ondisk+write+known_if_redirected e1423252) v8 -- 0x55c98f460700 con 0-2> 2018-12-11 15:17:57.631312 7fc128b49700 5 --172.28.196.121:0/464072497 >> 172.26.212.6:6802/2058882conn(0x55c98f4d7800 :-1 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCHpgs=202 cs=1 l=1). rx osd.480 seq 2 0x55c98f460700 osd_op_reply(13.dir.default.1083413551.2.7 [call,call] v1423252'7548805 uv7548805_ondisk_ = 0) v8-1> 2018-12-11 15:17:57.631329 7fc128b49700 1 --172.28.196.121:0/464072497 <== osd.480 172.26.212.6:6802/2058882 2==== osd_op_reply(13 .dir.default.1083413551.2.7 [call,call]v1423252'7548805 uv7548805 _ondisk_ = 0) v8 ==== 213+0+0 (4216487267 00) 0x55c98f460700 con 0x55c98f4d78000> 2018-12-11 15:17:57.631834 7fc0e3ffe700 -1 *** Caught signal(Floating point exception) **in thread 7fc0e3ffe700 thread_name:civetweb-workerceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94)luminous (stable)1: (()+0x200024) [0x55c98cc95024]2: (()+0x11390) [0x7fc13e474390]3: (RGWGC::tag_index(std::__cxx11::basic_string<char,std::char_traits<char>, std::allocator<char> > const&)+0x56)[0x55c98cf78cc6]4: (RGWGC::send_chain(cls_rgw_obj_chain&,std::__cxx11::basic_string<char, std::char_traits<char>,std::allocator<char> > const&, bool)+0x6a) [0x55c98cf7b06a]5: (RGWRados::Object::complete_atomic_modification()+0xd3)[0x55c98cdbfb63]6: (RGWRados::Object::Delete::delete_obj()+0xa22) [0x55c98cdf4142]7: (RGWDeleteObj::execute()+0x46c) [0x55c98cd8802c]8: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*,req_state*, bool)+0x165) [0x55c98cdb01c5]9: (process_request(RGWRados*, RGWREST*, RGWRequest*,std::__cxx11::basic_string<char, std::char_traits<char>,std::allocator<char> > const&, rgw::auth::StrategyRegistry const&,RGWRestfulIO*, OpsLogSocket*, int*)+0x1dbc) [0x55c98cdb234c]10: (RGWCivetWebFrontend::process(mg_connection*)+0x38f) [0x55c98cc4aacf]11: (()+0x1f05d9) [0x55c98cc855d9]12: (()+0x1f1fa9) [0x55c98cc86fa9]13: (()+0x76ba) [0x7fc13e46a6ba]14: (clone()+0x6d) [0x7fc133b5941d]NOTE: a copy of the executable, or `objdump -rdS <executable>` isneeded to interpret this.--- logging levels ---0/ 5 none0/ 1 lockdep0/ 1 context1/ 1 crush1/ 5 mds1/ 5 mds_balancer1/ 5 mds_locker1/ 5 mds_log1/ 5 mds_log_expire1/ 5 mds_migrator0/ 1 buffer0/ 1 timer0/ 1 filer0/ 1 striper0/ 1 objecter0/ 5 rados0/ 5 rbd0/ 5 rbd_mirror0/ 5 rbd_replay0/ 5 journaler0/ 5 objectcacher0/ 5 client1/ 5 osd0/ 5 optracker0/ 5 objclass1/ 3 filestore1/ 3 journal0/ 5 ms1/ 5 mon0/10 monc0/ 0 paxos0/ 5 tp1/ 5 auth1/ 5 crypto1/ 1 finisher1/ 1 reserver1/ 5 heartbeatmap1/ 5 perfcounter1/ 5 rgw1/10 civetweb1/ 5 javaclient1/ 5 asok1/ 1 throttle0/ 0 refs1/ 5 xio1/ 5 compressor1/ 5 bluestore1/ 5 bluefs1/ 3 bdev1/ 5 kstore4/ 5 rocksdb4/ 5 leveldb4/ 5 memdb1/ 5 kinetic1/ 5 fuse1/ 5 mgr1/ 5 mgrc1/ 5 dpdk1/ 5 eventtrace-2/-2 (syslog threshold)-1/-1 (stderr threshold)max_recent 10000max_new 1000log_file /var/log/ceph/radosgw_swift.logWhich isn't great. We can restart the radosgw but then anyone else whofancies deleting a large segmented object can kill our service.Any ideas?--Leon L. Robinson <leon.robinson@xxxxxxxxxxxx<mailto:leon.robinson@xxxxxxxxxxxx>>------------------------------------------------------------------------NOTICE AND DISCLAIMERThis e-mail (including any attachments) is intended for theabove-named person(s). If you are not the intended recipient, notifythe sender immediately, delete this email from your system and do notdisclose or use for any purpose. We may monitor all incoming andoutgoing emails in line with current legislation. We have taken stepsto ensure that this email and attachments are free from any virus, butit remains your responsibility to ensure that viruses do not adverselyaffect you_______________________________________________ceph-users mailing listceph-users@xxxxxxxxxxxxxxhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com_______________________________________________ceph-users mailing listceph-users@xxxxxxxxxxxxxxhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com NOTICE AND DISCLAIMER This e-mail (including any attachments) is intended for the above-named person(s). If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose. We may monitor all incoming and outgoing emails in line with current legislation. We have taken steps to ensure that this email and attachments are free from any virus, but it remains your responsibility to ensure that viruses do not adversely affect you |
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com