On Wed, Nov 12, 2014 at 9:43 PM, Yehuda Sadeh <yehuda@xxxxxxxxxx> wrote: > On Tue, Nov 11, 2014 at 5:19 AM, Mustafa Muhammad > <mustafaa.alhamdaani@xxxxxxxxx> wrote: >> On Tue, Nov 11, 2014 at 3:44 PM, pushpesh sharma <pushpesh.eck@xxxxxxxxx> wrote: >>> Mustafa, >>> >>> You can get rid of these messages by setting your rgw_obj_chunk_size >= >>> 'Object size your are testing with'. It will also increase the performance. >> >> Thank you for answering, I am using multiple objects (ranging from 500 >> MBs to 2 GBs), so I put "rgw obj chunk size = 4G" in >> /etc/ceph/ceph.conf? 4G is OK right? and what is the upper limit? > > No, you shouldn't put that much for the chunk size. This will > effectively disable striping, and cause a significant memory > consumption per thread. Ok, thank you. One more think, I am trying to set "rgw thread pool size" more than 1024, at 1024 it works, but anything more than that it doesn't (even 1025). I asked in #ceph and #ceph-devel and no answer. Also were can I find civetweb log and configuration? P.S. ulimit for apache is very high, so it is not a problem. Thanks Mustafa > > Yehuda > >> >>> >>> For CivetWeb you just need to set 'rgw_frontends="civetweb port=8080" , you >>> can tune some of rgw_ config with it. I find the most useful one with >>> civetweb is 'rgw_thread_pool_size' which maps to 'num_op_thread' in civetweb >>> configs. I find value of '128' good enough, but you can play around. >> It worked, thank you, but it is very slow comared to apache (but >> lighter) and nginx, after changing object chunk size, it improved alot >> (from about 100 MB/s to about 100 MB/s, still slower that nginx (about >> 150~200 MB/s). >> >>> >>> Yehuda, >>> I didnt find any way to disable access logs in CivetWeb. I set all the >>> *enable_logs parameter to false. >>> I am not able to properly setup fcgi multiple instance on same host, any >>> information would be useful. >>> >>> >>> On Tue, Nov 11, 2014 at 4:40 PM, Mustafa Muhammad >>> <mustafaa.alhamdaani@xxxxxxxxx> wrote: >>>> >>>> On Tue, Nov 11, 2014 at 1:49 AM, Yehuda Sadeh <yehuda@xxxxxxxxxx> wrote: >>>> > On Mon, Nov 10, 2014 at 12:45 PM, Mustafa Muhammad >>>> > <mustafaa.alhamdaani@xxxxxxxxx> wrote: >>>> >> Hi, >>>> >> I am using radosgw to connect to my ceph cluster, I am testing it and >>>> >> with large number of requests, I get: >>>> >> *** buffer overflow detected ***: /bin/radosgw terminated >>>> >> in the syslog. >>>> >> I use CentOS 7, and this is the some of the last lines of the log: >>>> >> >>>> >> ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6) >>>> >> 1: /bin/radosgw() [0x5daaf6] >>>> >> 2: (()+0xf130) [0x7f177cd4e130] >>>> >> 3: (gsignal()+0x39) [0x7f177bf905c9] >>>> >> 4: (abort()+0x148) [0x7f177bf91cd8] >>>> >> 5: (()+0x75dd7) [0x7f177bfd0dd7] >>>> >> 6: (__fortify_fail()+0x37) [0x7f177c0688f7] >>>> >> 7: (()+0x10bac0) [0x7f177c066ac0] >>>> >> 8: (()+0x10d867) [0x7f177c068867] >>>> >> 9: (OS_Accept()+0xc1) [0x7f177d4a18b1] >>>> >> 10: (FCGX_Accept_r()+0x9c) [0x7f177d49f91c] >>>> >> 11: (RGWFCGXProcess::run()+0x1c8) [0x4c9318] >>>> >> 12: (RGWProcessControlThread::entry()+0xe) [0x4cc25e] >>>> >> 13: (()+0x7df3) [0x7f177cd46df3] >>>> >> 14: (clone()+0x6d) [0x7f177c05101d] >>>> >> NOTE: a copy of the executable, or `objdump -rdS <executable>` is >>>> >> needed to interpret this. >>>> >> >>>> >> --- logging levels --- >>>> >> 0/ 5 none >>>> >> 0/ 1 lockdep >>>> >> 0/ 1 context >>>> >> 1/ 1 crush >>>> >> 1/ 5 mds >>>> >> 1/ 5 mds_balancer >>>> >> 1/ 5 mds_locker >>>> >> 1/ 5 mds_log >>>> >> 1/ 5 mds_log_expire >>>> >> 1/ 5 mds_migrator >>>> >> 0/ 1 buffer >>>> >> 0/ 1 timer >>>> >> 0/ 1 filer >>>> >> 0/ 1 striper >>>> >> 0/ 1 objecter >>>> >> 0/ 5 rados >>>> >> 0/ 5 rbd >>>> >> 0/ 5 journaler >>>> >> 0/ 5 objectcacher >>>> >> 0/ 5 client >>>> >> 0/ 5 osd >>>> >> 0/ 5 optracker >>>> >> 0/ 5 objclass >>>> >> 1/ 3 filestore >>>> >> 1/ 3 keyvaluestore >>>> >> 1/ 3 journal >>>> >> 0/ 5 ms >>>> >> 1/ 5 mon >>>> >> 0/10 monc >>>> >> 1/ 5 paxos >>>> >> 0/ 5 tp >>>> >> 1/ 5 auth >>>> >> 1/ 5 crypto >>>> >> 1/ 1 finisher >>>> >> 1/ 5 heartbeatmap >>>> >> 1/ 5 perfcounter >>>> >> 1/ 5 rgw >>>> >> 1/ 5 javaclient >>>> >> 1/ 5 asok >>>> >> 1/ 1 throttle >>>> >> -2/-2 (syslog threshold) >>>> >> -1/-1 (stderr threshold) >>>> >> max_recent 10000 >>>> >> max_new 1000 >>>> >> log_file /var/log/ceph/radosgw.log >>>> >> --- end dump of recent events --- >>>> > >>>> > This might be an issue with the fastcgi library that radosgw uses (not >>>> > sure which one and what version is used in centos 7). How many >>>> > concurrent requests does it handle when it fails? You can try testing >>>> > it with the standalone web server (civetweb), see how it behaves. >>>> I think I am using fcgi 2.4.0, I use nginx with "fastcgi_buffering >>>> off;" so it doesn't touch the disks. >>>> Somtimes it handles 4000 connections, sometimes 1000. >>>> I want to test civetweb but couldn't find any info about how to do so, >>>> can you please give me a link to docs or something. >>>> >>>> Thank you. >>>> >>>> Mustafa >>>> >> >>>> >> P.S. I get lots of errors like: >>>> >> RGWObjManifest::operator++(): result: ofs=20971520 stripe_ofs=20971520 >>>> >> part_ofs=0 rule->part_size=104857600 >>>> > >>>> > This is just a too verbose log message, not necessarily pointing at >>>> > anything wrong. >>>> > >>>> > Thanks, >>>> > Yehuda >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> >>> >>> >>> >>> -- >>> -Pushpesh >>> >> >> >> >> Now about the buffer overflow, should I file a bug? >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@xxxxxxxxxxxxxxx >> More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
