Re: Repeated failures in RGW in Ceph 12.1.4

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



The daily log rotation. 

-Ben

On Wed, Aug 30, 2017 at 3:09 PM, Bryan Banister <bbanister@xxxxxxxxxxxxxxx> wrote:

Looking at the systemd service it does show that twice, at roughly the same time and one day apart, the service did receive a HUP signal:

Aug 29 16:31:02 carf-ceph-osd02 radosgw[130050]: 2017-08-29 16:31:02.528559 7fffc641c700 -1 received  signal: Hangup from  PID: 73176 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0

Aug 30 16:32:02 carf-ceph-osd02 radosgw[130050]: 2017-08-30 16:32:02.529825 7fffc641c700 -1 received  signal: Hangup from  PID: 48062 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0

 

Any idea what would do this?

 

I'll be updating the version to 12.2.0 shortly,

-Bryan

 

-----Original Message-----
From: ceph-users [mailto:ceph-users-bounces@lists.ceph.com] On Behalf Of Bryan Banister

Sent: Wednesday, August 30, 2017 3:42 PM
To: Yehuda Sadeh-Weinraub <yehuda@xxxxxxxxxx>
Cc: ceph-users@xxxxxxxxxxxxxx
Subject: Re: Repeated failures in RGW in Ceph 12.1.4

 

Note: External Email

-------------------------------------------------

 

We are not sending a HUP signal that we know about.  We were not modifying our configuration.  However all user accounts in the RGW were lost!

-Bryan

 

-----Original Message-----

From: Yehuda Sadeh-Weinraub [mailto:yehuda@xxxxxxxxxx]

Sent: Wednesday, August 30, 2017 3:30 PM

To: Bryan Banister <bbanister@xxxxxxxxxxxxxxx>

Cc: ceph-users@xxxxxxxxxxxxxx

Subject: Re: Repeated failures in RGW in Ceph 12.1.4

 

Note: External Email

-------------------------------------------------

 

On Wed, Aug 30, 2017 at 5:44 PM, Bryan Banister

<bbanister@xxxxxxxxxxxxxxx> wrote:

> Not sure what’s happening but we started to but a decent load on the RGWs we

> have setup and we were seeing failures with the following kind of

> fingerprint:

> 

> 

> 

> 2017-08-29 17:06:22.072361 7ffdc501a700  1 rgw realm reloader: Frontends

> paused

> 

 

Are you modifying configuration? Could be that something is sending

HUP singal to the radosgw process. We disabled this behavior (process

dynamic reconfig after HUP) in 12.2.0.

 

Yehuda

 

> 2017-08-29 17:06:22.072359 7fffacbe9700  1 civetweb: 0x555556add000:

> 7.128.12.19 - - [29/Aug/2017:16:47:36 -0500] "PUT

> /blah?partNumber=8&uploadId=2~L9MEmUUmZKb2y8JCotxo62yzdMbHmye HTTP/1.1" 1 0

> - Minio (linux; amd64) minio-go/3.0.0

> 

> 2017-08-29 17:06:22.072438 7fffcb426700  0 ERROR: failed to clone shard,

> completion_mgr.get_next() returned ret=-125

> 

> 2017-08-29 17:06:23.689610 7ffdc501a700  1 rgw realm reloader: Store closed

> 

> 2017-08-29 17:06:24.117630 7ffdc501a700  1 failed to decode the mdlog

> history: buffer::end_of_buffer

> 

> 2017-08-29 17:06:24.117635 7ffdc501a700  1 failed to read mdlog history: (5)

> Input/output error

> 

> 2017-08-29 17:06:24.118711 7ffdc501a700  1 rgw realm reloader: Creating new

> store

> 

> 2017-08-29 17:06:24.118901 7ffdc501a700  1 mgrc service_daemon_register

> rgw.carf-ceph-osd01 metadata {arch=x86_64,ceph_version=ceph version 12.1.4

> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc),cpu=Intel(R)

> Xeon(R) CPU E5-2680 v4 @ 2.40GHz,distro=rhel,distro_description=Red Hat

> Enterprise Linux Server 7.3

> (Maipo),distro_version=7.3,frontend_config#0=civetweb port=80

> num_threads=1024,frontend_type#0=civetweb,hos

> 

> tname=carf-ceph-osd01,kernel_description=#1 SMP Tue Apr 4 04:49:42 CDT

> 2017,kernel_version=3.10.0-514.6.1.el7.jump3.x86_64,mem_swap_kb=0,mem_total_kb=263842036,num_handles=1,os=Linux,pid=14723,zone_id=b0634f34-67e2-4b44-ab00-5282f1e2cd83,zone_name=carf01,zonegroup_id=8207fcf5-7bd3-43df-ab5a-ea17e5949eec,zonegroup_name=us}

> 

> 2017-08-29 17:06:24.118925 7ffdc501a700  1 rgw realm reloader: Finishing

> initialization of new store

> 

> 2017-08-29 17:06:24.118927 7ffdc501a700  1 rgw realm reloader:  - REST

> subsystem init

> 

> 2017-08-29 17:06:24.118943 7ffdc501a700  1 rgw realm reloader:  - user

> subsystem init

> 

> 2017-08-29 17:06:24.118947 7ffdc501a700  1 rgw realm reloader:  - user

> subsystem init

> 

> 2017-08-29 17:06:24.118950 7ffdc501a700  1 rgw realm reloader:  - usage

> subsystem init

> 

> 2017-08-29 17:06:24.118985 7ffdc501a700  1 rgw realm reloader: Resuming

> frontends with new realm configuration.

> 

> 2017-08-29 17:06:24.119018 7fffad3ea700  1 ====== starting new request

> req=0x7fffad3e4190 =====

> 

> 2017-08-29 17:06:24.119039 7fffacbe9700  1 ====== starting new request

> req=0x7fffacbe3190 =====

> 

> 2017-08-29 17:06:24.120163 7fffacbe9700  1 ====== req done

> req=0x7fffacbe3190 op status=0 http_status=403 ======

> 

> 2017-08-29 17:06:24.120200 7fffad3ea700  1 ====== req done

> req=0x7fffad3e4190 op status=0 http_status=403 ======

> 

> 

> 

> Any help understanding how to fix this would be greatly appreciated!

> 

> -Bryan

> 

> 

> ________________________________

> 

> Note: This email is for the confidential use of the named addressee(s) only

> and may contain proprietary, confidential or privileged information. If you

> are not the intended recipient, you are hereby notified that any review,

> dissemination or copying of this email is strictly prohibited, and to please

> notify the sender immediately and destroy this email and any attachments.

> Email transmission cannot be guaranteed to be secure or error-free. The

> Company, therefore, does not make any guarantees as to the completeness or

> accuracy of this email or any attachments. This email is for informational

> purposes only and does not constitute a recommendation, offer, request or

> solicitation of any kind to buy, sell, subscribe, redeem or perform any type

> of transaction of a financial product.

> 

> _______________________________________________

> ceph-users mailing list

> ceph-users@xxxxxxxxxxxxxx

> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

> 

 

________________________________

 

Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




Note: This email is for the confidential use of the named addressee(s) only and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you are hereby notified that any review, dissemination or copying of this email is strictly prohibited, and to please notify the sender immediately and destroy this email and any attachments. Email transmission cannot be guaranteed to be secure or error-free. The Company, therefore, does not make any guarantees as to the completeness or accuracy of this email or any attachments. This email is for informational purposes only and does not constitute a recommendation, offer, request or solicitation of any kind to buy, sell, subscribe, redeem or perform any type of transaction of a financial product.

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux