Hi Daniel,
Thanks for you're reply. I've checked the package versions on that server and all ceph related packages on that server are from 15.2.8 version:
ii librados2 15.2.8-1focal amd64 RADOS distributed object store client library
ii libradosstriper1 15.2.8-1focal amd64 RADOS striping interface
ii python3-rados 15.2.8-1focal amd64 Python 3 libraries for the Ceph librados library
ii radosgw 15.2.8-1focal amd64 REST gateway for RADOS distributed object store
ii librbd1 15.2.8-1focal amd64 RADOS block device client library
ii python3-rbd 15.2.8-1focal amd64 Python 3 libraries for the Ceph librbd library
ii ceph 15.2.8-1focal amd64 distributed storage and file system
ii ceph-base 15.2.8-1focal amd64 common ceph daemon libraries and management tools
ii ceph-common 15.2.8-1focal amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fuse 15.2.8-1focal amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 15.2.8-1focal amd64 metadata server for the ceph distributed file system
ii ceph-mgr 15.2.8-1focal amd64 manager for the ceph distributed storage system
ii ceph-mgr-cephadm 15.2.8-1focal all cephadm orchestrator module for ceph-mgr
ii ceph-mgr-dashboard 15.2.8-1focal all dashboard module for ceph-mgr
ii ceph-mgr-diskprediction-cloud 15.2.8-1focal all diskprediction-cloud module for ceph-mgr
ii ceph-mgr-diskprediction-local 15.2.8-1focal all diskprediction-local module for ceph-mgr
ii ceph-mgr-k8sevents 15.2.8-1focal all kubernetes events module for ceph-mgr
ii ceph-mgr-modules-core 15.2.8-1focal all ceph manager modules which are always enabled
ii ceph-mgr-rook 15.2.8-1focal all rook module for ceph-mgr
ii ceph-mon 15.2.8-1focal amd64 monitor server for the ceph storage system
ii ceph-osd 15.2.8-1focal amd64 OSD server for the ceph storage system
ii cephadm 15.2.8-1focal amd64 cephadm utility to bootstrap ceph daemons with systemd and containers
ii libcephfs2 15.2.8-1focal amd64 Ceph distributed file system client library
ii python3-ceph 15.2.8-1focal amd64 Meta-package for python libraries for the Ceph libraries
ii python3-ceph-argparse 15.2.8-1focal all Python 3 utility libraries for Ceph CLI
ii python3-ceph-common 15.2.8-1focal all Python 3 utility libraries for Ceph
ii python3-cephfs 15.2.8-1focal amd64 Python 3 libraries for the Ceph libcephfs library
As this is a brand new 20.04 server I do not see how the older version could have got onto it.
Andrei
----- Original Message -----
> From: "Daniel Gryniewicz" <dang@xxxxxxxxxx>
> To: "ceph-users" <ceph-users@xxxxxxx>
> Sent: Thursday, 28 January, 2021 14:06:16
> Subject: Re: radosgw process crashes multiple times an hour
> It looks like your radosgw is using a different version of librados. In
> the backtrace, the top useful line begins:
>
> librados::v14_2_0
>
> when it should be v15.2.0, like the ceph::buffer in the same line.
>
> Is there an old librados lying around that didn't get cleaned up somehow?
>
> Daniel
>
>
>
> On 1/28/21 7:27 AM, Andrei Mikhailovsky wrote:
>> Hello,
>>
>> I am experiencing very frequent crashes of the radosgw service. It happens
>> multiple times every hour. As an example, over the last 12 hours we've had 35
>> crashes. Has anyone experienced similar behaviour of the radosgw octopus
>> release service? More info below:
>>
>> Radosgw service is running on two Ubuntu servers. I have tried upgrading OS on
>> one of the servers to Ubuntu 20.04 with latest updates. The second server is
>> still running Ubuntu 18.04. Both services crash occasionally, but the service
>> which is running on Ubuntu 20.04 crashes far more often it seems. The ceph
>> cluster itself is pretty old and was initially setup around 2013. The cluster
>> was updated pretty regularly with every major release. Currently, I've got
>> Octopus 15.2.8 running on all osd, mon, mgr and radosgw servers.
>>
>> Crash Backtrace:
>>
>> ceph crash info 2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8
>> |less
>> {
>> "backtrace": [
>> "(()+0x46210) [0x7f815a49a210]",
>> "(gsignal()+0xcb) [0x7f815a49a18b]",
>> "(abort()+0x12b) [0x7f815a479859]",
>> "(()+0x9e951) [0x7f8150ee9951]",
>> "(()+0xaa47c) [0x7f8150ef547c]",
>> "(()+0xaa4e7) [0x7f8150ef54e7]",
>> "(()+0xaa799) [0x7f8150ef5799]",
>> "(()+0x344ba) [0x7f815a1404ba]",
>> "(()+0x71e04) [0x7f815a17de04]",
>> "(librados::v14_2_0::IoCtx::nobjects_begin(librados::v14_2_0::ObjectCursor
>> const&, ceph::buffer::v15_2_0::list const&)+0x5d) [0x7f815a18c7bd]",
>> "(RGWSI_RADOS::Pool::List::init(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > const&,
>> RGWAccessListFilter*)+0x115) [0x7f815b0d9935]",
>> "(RGWSI_SysObj_Core::pool_list_objects_init(rgw_pool const&,
>> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
>> const&, std::__cxx11::basic_string<char, std::char_traits<char>,
>> std::allocator<char> > const&, RGWSI_SysObj::Pool::ListCtx*)+0x255)
>> [0x7f815abd7035]",
>> "(RGWSI_MetaBackend_SObj::list_init(RGWSI_MetaBackend::Context*,
>> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
>> const&)+0x206) [0x7f815b0ccfe6]",
>> "(RGWMetadataHandler_GenericMetaBE::list_keys_init(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > const&, void**)+0x41)
>> [0x7f815ad23201]",
>> "(RGWMetadataManager::list_keys_init(std::__cxx11::basic_string<char,
>> std::char_traits<char>, std::allocator<char> > const&,
>> std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >
>> const&, void**)+0x71) [0x7f815ad254d1]",
>> "(AsyncMetadataList::_send_request()+0x9b) [0x7f815b13c70b]",
>> "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x25)
>> [0x7f815ae60f25]",
>> "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*,
>> ThreadPool::TPHandle&)+0x11) [0x7f815ae69401]",
>> "(ThreadPool::worker(ThreadPool::WorkThread*)+0x5bb) [0x7f81517b072b]",
>> "(ThreadPool::WorkThread::entry()+0x15) [0x7f81517b17f5]",
>> "(()+0x9609) [0x7f815130d609]",
>> "(clone()+0x43) [0x7f815a576293]"
>> ],
>> "ceph_version": "15.2.8",
>> "crash_id": "2021-01-28T11:36:48.912771Z_08f80efd-c0ad-4551-88ce-905ca9cd3aa8",
>> "entity_name": "client.radosgw1.gateway",
>> "os_id": "ubuntu",
>> "os_name": "Ubuntu",
>> "os_version": "20.04.1 LTS (Focal Fossa)",
>> "os_version_id": "20.04",
>> "process_name": "radosgw",
>> "stack_sig": "347474f09a756104ac2bb99d80e0c1fba3e9dc6f26e4ef68fe55946c103b274a",
>> "timestamp": "2021-01-28T11:36:48.912771Z",
>> "utsname_hostname": "arh-ibstorage1-ib",
>> "utsname_machine": "x86_64",
>> "utsname_release": "5.4.0-64-generic",
>> "utsname_sysname": "Linux",
>> "utsname_version": "#72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021"
>> }
>>
>>
>>
>>
>>
>> radosgw.log file (file names were redacted):
>>
>>
>> -25> 2021-01-28T11:36:48.794+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010:
>> 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT /<file_name>-u115134.JPG
>> HTTP/1.1" 400 460 - -
>> -24> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== starting new request
>> req=0x7f80437f5780 =====
>> -23> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s initializing for
>> trans_id = tx000000000000000001431-006012a1d0-31197b5c-default
>> -22> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s getting op 1
>> -21> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj
>> verifying requester
>> -20> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj
>> normalizing buckets and tenants
>> -19> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj init
>> permissions
>> -18> 2021-01-28T11:36:48.814+0000 7f80437fe700 0 req 5169 0s NOTICE: invalid
>> dest placement: default-placement/REDUCED_REDUNDANCY
>> -17> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 op->ERRORHANDLER: err_no=-22
>> new_err_no=-22
>> -16> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj op
>> status=0
>> -15> 2021-01-28T11:36:48.814+0000 7f80437fe700 2 req 5169 0s s3:put_obj http
>> status=400
>> -14> 2021-01-28T11:36:48.814+0000 7f80437fe700 1 ====== req done
>> req=0x7f80437f5780 op status=0 http_status=400 latency=0s ======
>> -13> 2021-01-28T11:36:48.822+0000 7f80437fe700 1 civetweb: 0x7f814c0cf9e8:
>> 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT
>> /<file_name>-d20201223-u115132.JPG HTTP/1.1" 400 460 - -
>> -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request
>> req=0x7f8043ff6780 =====
>> -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for
>> trans_id = tx000000000000000001432-006012a1d0-31197b5c-default
>> -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1
>> -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying
>> requester
>> -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj
>> normalizing buckets and tenants
>> -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request
>> req=0x7f8043ff6780 =====
>> -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for
>> trans_id = tx000000000000000001432-006012a1d0-31197b5c-default
>> -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1
>> -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying
>> requester
>> -12> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== starting new request
>> req=0x7f8043ff6780 =====
>> -11> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s initializing for
>> trans_id = tx000000000000000001432-006012a1d0-31197b5c-default
>> -10> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s getting op 1
>> -9> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj verifying
>> requester
>> -8> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj
>> normalizing buckets and tenants
>> -7> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj init
>> permissions
>> -6> 2021-01-28T11:36:48.878+0000 7f8043fff700 0 req 5170 0s NOTICE: invalid dest
>> placement: default-placement/REDUCED_REDUNDANCY
>> -5> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 op->ERRORHANDLER: err_no=-22
>> new_err_no=-22
>> -4> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj op
>> status=0
>> -3> 2021-01-28T11:36:48.878+0000 7f8043fff700 2 req 5170 0s s3:put_obj http
>> status=400
>> -2> 2021-01-28T11:36:48.878+0000 7f8043fff700 1 ====== req done
>> req=0x7f8043ff6780 op status=0 http_status=400 latency=0s ======
>>
>> -1> 2021-01-28T11:36:48.886+0000 7f8043fff700 1 civetweb: 0x7f814c0cf010:
>> 176.35.173.88 - - [28/Jan/2021:11:36:48 +0000] "PUT
>> /<file_name>-223-u115136.JPG HTTP/1.1" 400 460 - -
>> 0> 2021-01-28T11:36:48.910+0000 7f8128ff9700 -1 *** Caught signal (Aborted) **
>> 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 deferred set uid:gid to 64045:64045
>> (ceph:ceph)
>> 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 ceph version 15.2.8
>> (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable), process radosgw,
>> pid 30417
>> 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework: civetweb
>> 2021-01-28T11:36:49.810+0000 7f76032db9c0 0 framework conf key: port, val: 443s
>>
>>
>> Could someone help me troubleshoot and fix the issue?
>>
>> Thanks
>> Andrei
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
