This is caused by an object that does not yet have a bucket associated
with it. It doesn't happen in S3, because S3 doesn't set_atomic() that
early, and it's fixed on main by the objctx removal (which is too
complicated for backport). Can you open a tracker for this, so that we
can get a fix into quincy (and older versions, which probably have this
issue too)?
Daniel
On 5/19/22 20:47, zhou-jielei@xxxxxx wrote:
Hi
My ceph version is 17.2.0 also rgw is the same version.
When I use swift api .if I set a X-Container-Meta-Web-Index: on a container. And access it. It will case rgw crash.
This is the log :
#swift post -m 'web-index:index.html' okamura-static-web
# curl http:// *********/swift/v1/AUTH_9c4a3213fc124afcb369c3b8246cc543/okamura-static-web/ -v
* Trying *********:80...
* TCP_NODELAY set
* Connected to dev-swift-vip.i1.dev.v6.internal-gmo (*********) port 80 (#0)
GET /swift/v1/AUTH_9c4a3213fc124afcb369c3b8246cc543/okamura-static-web/ HTTP/1.1
Host: dev-swift-vip.i1.dev.v6.internal-gmo
User-Agent: curl/7.68.0
Accept: */*
* Empty reply from server
* Connection #0 to host dev-swift-vip.i1.dev.v6.internal-gmo left intact
curl: (52) Empty reply from server
first time it will return Empty reply
next request will connect refused
#curl http://*********** -v
* Trying ***********:80...
* TCP_NODELAY set
* connect to 172.22.35.94 port 80 failed: Connection refused
* Failed to connect to dev-swift-vip.i1.dev.v6.internal-gmo port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to dev-swift-vip.i1.dev.v6.internal-gmo port 80: Connection refused
in ceph rgw log. We will see thread is crashed
-60> 2022-05-20T00:21:19.739+0000 7fe264a1c700 5 lifecycle: schedule life cycle next start time: Sat May 21 00:00:00 2022
-59> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 HTTP_ACCEPT=*/*
-58> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 HTTP_HOST=***********
-57> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 HTTP_USER_AGENT=curl/7.68.0
-56> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 HTTP_VERSION=1.1
-55> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 REMOTE_ADDR=***********
-54> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 REQUEST_METHOD=GET
-53> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 REQUEST_URI=/swift/v1/AUTH_9c4a3213fc124afcb369c3b8246cc543/okamura-static-web/
-52> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 SCRIPT_URI=/swift/v1/AUTH_9c4a3213fc124afcb369c3b8246cc543/okamura-static-web/
-51> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 SERVER_PORT=80
-50> 2022-05-20T00:21:20.294+0000 7fe2471e1700 1 ====== starting new request req=0x7fe2983db650 =====
-49> 2022-05-20T00:21:20.294+0000 7fe2471e1700 2 req 14058840826806468654 0.000000000s initializing for trans_id = tx00000c31b093ebb8f002e-006286df00-d605-dev-c3j1
-48> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s rgw api priority: s3=8 s3website=7
-47> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s host=dev-swift-vip.i1.dev.v6.internal-gmo
-46> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
-45> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_9c4a3213fc124afcb369c3b8246cc543/okamura-static-web/
-44> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s ver=v1 first=okamura-static-web req=
-43> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s handler=28RGWHandler_REST_Bucket_SWIFT
-42> 2022-05-20T00:21:20.294+0000 7fe2471e1700 2 req 14058840826806468654 0.000000000s getting op 0
-41> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s get_system_obj_state: rctx=0x7fe2983da680 obj=dev-c3j1.rgw.log:script.prerequest. state=0x560db724e9a0 s->prefetch_data=0
-40> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s cache get: name=dev-c3j1.rgw.log++script.prerequest. : hit (negative entry)
-39> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s swift:list_bucket scheduling with throttler client=3 cost=1
-38> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s swift:list_bucket op=28RGWListBucket_ObjStore_SWIFT
-37> 2022-05-20T00:21:20.294+0000 7fe2471e1700 2 req 14058840826806468654 0.000000000s swift:list_bucket verifying requester
-36> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine
-35> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::TempURLEngine denied with reason=-13
-34> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine
-33> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::SignedTokenEngine denied with reason=-1
-32> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::keystone::TokenEngine
-31> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::keystone::TokenEngine denied with reason=-13
-30> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SwiftAnonymousEngine
-29> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket rgw::auth::swift::SwiftAnonymousEngine granted access
-28> 2022-05-20T00:21:20.294+0000 7fe2471e1700 2 req 14058840826806468654 0.000000000s swift:list_bucket normalizing buckets and tenants
-27> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s s->object=<NULL> s->bucket=9c4a3213fc124afcb369c3b8246cc543/okamura-static-web
-26> 2022-05-20T00:21:20.294+0000 7fe2471e1700 2 req 14058840826806468654 0.000000000s swift:list_bucket init permissions
-25> 2022-05-20T00:21:20.294+0000 7fe2471e1700 20 req 14058840826806468654 0.000000000s swift:list_bucket get_system_obj_state: rctx=0x7fe2983da060 obj=dev-c3j1.rgw.meta:root:9c4a3213fc124afcb369c3b8246cc543/okamura-static-web state=0x560db724ee20 s->prefetch_data=0
-24> 2022-05-20T00:21:20.294+0000 7fe2471e1700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache get: name=dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web : miss
-23> 2022-05-20T00:21:20.294+0000 7fe2411d5700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache put: name=dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web info.flags=0x16
-22> 2022-05-20T00:21:20.294+0000 7fe2411d5700 10 req 14058840826806468654 0.000000000s swift:list_bucket adding dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web to cache LRU end
-21> 2022-05-20T00:21:20.294+0000 7fe2411d5700 10 req 14058840826806468654 0.000000000s swift:list_bucket updating xattr: name=ceph.objclass.version bl.length()=42
-20> 2022-05-20T00:21:20.294+0000 7fe2411d5700 20 req 14058840826806468654 0.000000000s swift:list_bucket get_system_obj_state: s->obj_tag was set empty
-19> 2022-05-20T00:21:20.294+0000 7fe2411d5700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache get: name=dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web : type miss (requested=0x11, cached=0x16)
-18> 2022-05-20T00:21:20.294+0000 7fe2411d5700 20 req 14058840826806468654 0.000000000s swift:list_bucket rados->read ofs=0 len=0
-17> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s swift:list_bucket rados_obj.operate() r=0 bl.length=302
-16> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache put: name=dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web info.flags=0x11
-15> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s swift:list_bucket moving dev-c3j1.rgw.meta+root+9c4a3213fc124afcb369c3b8246cc543/okamura-static-web to cache LRU end
-14> 2022-05-20T00:21:20.294+0000 7fe2409d4700 15 req 14058840826806468654 0.000000000s swift:list_bucket decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner><ID>9c4a3213fc124afcb369c3b8246cc543$9c4a3213fc124afcb369c3b8246cc543</ID><DisplayName>demo</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser"><ID>9c4a3213fc124afcb369c3b8246cc543$9c4a3213fc124afcb369c3b8246cc543</ID><DisplayName>demo</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
-13> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s swift:list_bucket get_system_obj_state: rctx=0x7fe2983d9ac8 obj=dev-c3j1.rgw.meta:users.uid:9c4a3213fc124afcb369c3b8246cc543$9c4a3213fc124afcb369c3b8246cc543 state=0x560db724f060 s->prefetch_data=0
-12> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache get: name=dev-c3j1.rgw.meta+users.uid+9c4a3213fc124afcb369c3b8246cc543$9c4a3213fc124afcb369c3b8246cc543 : hit (requested=0x16, cached=0x17)
-11> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s swift:list_bucket get_system_obj_state: s->obj_tag was set empty
-10> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s swift:list_bucket Read xattr: user.rgw.idtag
-9> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s swift:list_bucket cache get: name=dev-c3j1.rgw.meta+users.uid+9c4a3213fc124afcb369c3b8246cc543$9c4a3213fc124afcb369c3b8246cc543 : hit (requested=0x13, cached=0x17)
-8> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s swift:list_bucket RGWSI_User_RADOS::read_user_info(): anonymous user
-7> 2022-05-20T00:21:20.294+0000 7fe2409d4700 2 req 14058840826806468654 0.000000000s swift:list_bucket recalculating target
-6> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s Starting retarget
-5> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s get_obj_state: rctx=0x7fe2983da9e0 obj=okamura-static-web:index.html state=0x560db6f02de8 s->prefetch_data=1
-4> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s WARNING: blocking librados call
-3> 2022-05-20T00:21:20.294+0000 7fe2409d4700 10 req 14058840826806468654 0.000000000s manifest: total_size = 10
-2> 2022-05-20T00:21:20.294+0000 7fe2409d4700 20 req 14058840826806468654 0.000000000s get_obj_state: setting s->obj_tag to 807b4a00-6279-4f24-8971-b1175e2f49c1.54526.10995817613273309006
-1> 2022-05-20T00:21:20.294+0000 7fe2409d4700 2 req 14058840826806468654 0.000000000s swift:get_obj reading permissions
0> 2022-05-20T00:21:20.298+0000 7fe2409d4700 -1 *** Caught signal (Segmentation fault) **
in thread 7fe2409d4700 thread_name:radosgw
ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)
1: /lib64/libpthread.so.0(+0x12ce0) [0x7fe294232ce0]
2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7fe296f6f4a3]
3: (rgw::sal::RadosObject::set_atomic(RGWObjectCtx*) const+0x55) [0x7fe297473f25]
4: (rgw_build_object_policies(DoutPrefixProvider const*, rgw::sal::Store*, req_state*, bool, optional_yield)+0x1cb) [0x7fe2972e317b]
5: (RGWHandler::do_read_permissions(RGWOp*, bool, optional_yield)+0x56) [0x7fe29730f0c6]
6: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Store*, bool)+0x3c4) [0x7fe296f49fb4]
7: (process_request(rgw::sal::Store*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSink*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, std::shared_ptr<RateLimiter>, int*)+0x2616) [0x7fe296f4d696]
8: /lib64/libradosgw.so.2(+0x65a00a) [0x7fe296eba00a]
9: /lib64/libradosgw.so.2(+0x65b6b1) [0x7fe296ebb6b1]
10: /lib64/libradosgw.so.2(+0x65b82c) [0x7fe296ebb82c]
11: make_fcontext()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 rbd_pwl
0/ 5 journaler
0/ 5 objectcacher
0/ 5 immutable_obj_cache
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 0 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
20/20 rgw
1/ 5 rgw_sync
1/ 5 rgw_datacache
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 fuse
2/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
1/ 5 prioritycache
0/ 5 test
0/ 5 cephfs_mirror
0/ 5 cephsqlite
0/ 5 seastore
0/ 5 seastore_onode
0/ 5 seastore_odata
0/ 5 seastore_omap
0/ 5 seastore_tm
0/ 5 seastore_cleaner
0/ 5 seastore_lba
0/ 5 seastore_cache
0/ 5 seastore_journal
0/ 5 seastore_device
0/ 5 alienstore
1/ 5 mclock
-2/-2 (syslog threshold)
99/99 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
7fe159005700 / safe_timer
7fe15a007700 / ms_dispatch
7fe15a808700 / ceph_timer
7fe15b80a700 / io_context_pool
7fe2409d4700 / radosgw
7fe2411d5700 / radosgw
7fe2471e1700 / radosgw
7fe24b1e9700 / radosgw
7fe2511f5700 / radosgw
7fe2519f6700 / radosgw
7fe2521f7700 / radosgw
7fe2529f8700 / radosgw
7fe2531f9700 / radosgw
7fe2539fa700 / radosgw
7fe2541fb700 / radosgw
7fe2549fc700 / radosgw
7fe2551fd700 / radosgw
7fe2559fe700 / radosgw
7fe2561ff700 / radosgw
7fe256a00700 / radosgw
7fe257201700 / radosgw
7fe259205700 / radosgw
7fe25e20f700 / notif-worker0
7fe25f211700 / rgw_reshard
7fe25fa12700 / rgw_user_st_syn
7fe260213700 / rgw_buck_st_syn
7fe260a14700 / lifecycle_thr_2
7fe262a18700 / lifecycle_thr_1
7fe264a1c700 / lifecycle_thr_0
7fe267221700 / http_manager
7fe268223700 / sync-log-trim
7fe268a24700 / http_manager
7fe26ba2a700 / http_manager
7fe26ca2c700 / rgw_obj_expirer
7fe26d22d700 / rgw_gc
7fe26e22f700 / safe_timer
7fe26f231700 / ms_dispatch
7fe270a34700 / io_context_pool
7fe271235700 / rgw_dt_lg_renew
7fe282257700 / safe_timer
7fe283259700 / ms_dispatch
7fe283a5a700 / ceph_timer
7fe284a5c700 / io_context_pool
7fe28625f700 / kmip worker
7fe286a60700 / http_manager
7fe288263700 / admin_socket
7fe288a64700 / service
7fe289265700 / msgr-worker-2
7fe289a66700 / msgr-worker-1
7fe28a267700 / msgr-worker-0
7fe2984265c0 / radosgw
max_recent 10000
max_new 10000
log_file /var/log/ceph/ceph-client.rgw.*izjtte.log
--- end dump of recent events ---
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
