To the client they were showing up as a 500 error. Ty, do you know of any client-side issues that could have come up during the test run? And there was only a single GET happening at a time, right? On 10/11/17, 9:27 AM, "ceph-users on behalf of Casey Bodley" <ceph-users-bounces@xxxxxxxxxxxxxx on behalf of cbodley@xxxxxxxxxx> wrote: >Hi Travis, > >This is reporting an error when sending data back to the client. >Generally it means that the client timed out and closed the connection. >Are you also seeing failures on the client side? > >Casey > > >On 10/10/2017 06:45 PM, Travis Nielsen wrote: >> In Luminous 12.2.1, when running a GET on a large (1GB file) repeatedly >> for an hour from RGW, the following error was hit intermittently a >>number >> of times. The first error was hit after 45 minutes and then the error >> happened frequently for the remainder of the test. >> >> ERROR: flush_read_list(): d->client_cb->handle_data() returned -5 >> >> Here is some more context from the rgw log around one of the failures. >> >> 2017-10-10 18:20:32.321681 I | rgw: 2017-10-10 18:20:32.321643 >> 7f8929f41700 1 civetweb: 0x55bd25899000: 10.32.0.1 - - >> [10/Oct/2017:18:19:07 +0000] "GET /bucket100/testfile.tst HTTP/1.1" 1 0 >>- >> aws-sdk-java/1.9.0 Linux/4.4.0-93-generic >> OpenJDK_64-Bit_Server_VM/25.131-b11/1.8.0_131 >> 2017-10-10 18:20:32.383855 I | rgw: 2017-10-10 18:20:32.383786 >> 7f8924736700 1 ====== starting new request req=0x7f892472f140 ===== >> 2017-10-10 18:20:46.605668 I | rgw: 2017-10-10 18:20:46.605576 >> 7f894af83700 0 ERROR: flush_read_list(): d->client_cb->handle_data() >> returned -5 >> 2017-10-10 18:20:46.605934 I | rgw: 2017-10-10 18:20:46.605914 >> 7f894af83700 1 ====== req done req=0x7f894af7c140 op status=-5 >> http_status=200 ====== >> 2017-10-10 18:20:46.606249 I | rgw: 2017-10-10 18:20:46.606225 >> 7f8924736700 0 ERROR: flush_read_list(): d->client_cb->handle_data() >> returned -5 >> >> I don't see anything else standing out in the log. The object store was >> configured with an erasure-coded data pool with k=2 and m=1. >> >> There are a number of threads around this, but I don't see a resolution. >> Is there a tracking issue for this? >> >>https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.cep >>h.com%2Fpipermail%2Fceph-users-ceph.com%2F2016-February%2F007756.ht&data= >>02%7C01%7CTravis.Nielsen%40quantum.com%7C5ba068e75938455da6a408d510c50ddd >>%7C322a135f14fb4d72aede122272134ae0%7C1%7C0%7C636433360880521631&sdata=Wb >>EdpMEB%2BvjZS%2BxclppC3%2BHALu6iayzwjTQeFK3qMp8%3D&reserved=0 >> ml >> >>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spin >>ics.net%2Flists%2Fceph-users%2Fmsg16117.html&data=02%7C01%7CTravis.Nielse >>n%40quantum.com%7C5ba068e75938455da6a408d510c50ddd%7C322a135f14fb4d72aede >>122272134ae0%7C1%7C0%7C636433360880521631&sdata=5PSDwmEnZB7g9atCeRlZvTPUX >>wHB3c1bFjiDt7VfkKI%3D&reserved=0 >> >>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.spin >>ics.net%2Flists%2Fceph-devel%2Fmsg37657.html&data=02%7C01%7CTravis.Nielse >>n%40quantum.com%7C5ba068e75938455da6a408d510c50ddd%7C322a135f14fb4d72aede >>122272134ae0%7C1%7C0%7C636433360880521631&sdata=4oEXugLrXjmIPRnz4LAiavbMF >>kgUnEj5jBw%2F%2Bk9BYJE%3D&reserved=0 >> >> >> Here's our tracking Rook issue. >> >>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.c >>om%2Frook%2Frook%2Fissues%2F1067&data=02%7C01%7CTravis.Nielsen%40quantum. >>com%7C5ba068e75938455da6a408d510c50ddd%7C322a135f14fb4d72aede122272134ae0 >>%7C1%7C0%7C636433360880521631&sdata=pYi3%2FZupNoy7Act1bQomEee6seO%2BKDt%2 >>BzkgzmcYeJV4%3D&reserved=0 >> >> >> Thanks, >> Travis >> _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |