rgw connection resets

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello, I’ve noticed what I think may be strange behavior from some rgw transfers.  It only seems to be with larger multipart mp4 files.  Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”.  The client then reconnects and the transfer seems to be ok.  This doesn’t appear to happen just downloading a regular file.  This is happening to both my production and test cluster.  I’m not seeing errors anywhere else, the network looks good and the clusters are healthy.  I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring.

We run haproxy in front of the rgw.  For troubleshooting purposes we have a basic haproxy config to just one rgw.  We have tested with multiple browsers.

Ceph 17.2.5 - cluster built and managed with cephadm
Ubuntu 20.04.6 LTS

Log:
                                                                                                                                                  
2024-01-03T17:40:23.916+0000 7f646033d700  1 ====== starting new request req=0x7f640ca95730 =====                                                                                                             
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default                                               
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7                                                                                           
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net                                                                                                     
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0                                                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n
athan:tshoot/testfile.mp4                                                                                                                                                                                      
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3                                                                                
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3                                                                                            
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s getting op 0                                                                                                                 
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0     
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry)                                                  
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3                                                                                        
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester                                                                                               
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy                            
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine                                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access                                                                     
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants                                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot                                                                               
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj init permissions                                                                                                 
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch
_data=0                                                                                                                                                                                                        
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17)                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17)                           
2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner>
<ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser"><ID>nathan$testuser
</ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>                                                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user                                                               
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target                                                                                             
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions                                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1                
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314                                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos                                             
2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/";><Owner>
<ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="Group"><URI>http://acs.amazonaws.c
om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa
me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj init op                                                                                                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p
refetch_data=0                                                                                                                                                                                                 
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17)                    
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag                                                                                        
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17)                    
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask                                                                                                 
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7                                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions                                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49                                                                    
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n
ame=, subuser=, perm_mask=15, is_admin=0) mask=49                                                                                                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous                                                                          
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found                                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1                                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700  5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, 
acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1                                                                                                                                 
2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj  identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask
=15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1                                                                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params                                                                                               
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing                                                                                                     
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting                                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700  2 req 4522146966294331133 0.000000000s s3:get_obj executing                                                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1                
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl                                                                               
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type                                                                      
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag                                                                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest                                                                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver                                                                            
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone                                                                       
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag                                                                           
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256                                                               
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date                                                                         
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs                                                            
2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode:                                                                                                             
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1                
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4
134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304                                                                                                                                                        
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2                                             
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640          
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134
dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2                                             
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640                              
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640          
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134
dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304                                                                                                                                                   
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2                                             
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640                             
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640        
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134
dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728                                                                                                                                                  
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640
2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4
134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304
2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2
2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640
2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640
2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134
dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304
2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2
2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640
2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640
2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134
dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0
2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1
2024-01-03T17:40:24.068+0000 7f642a2d1700  4 write_data failed: Connection reset by peer
2024-01-03T17:40:24.068+0000 7f642a2d1700  0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104
2024-01-03T17:40:24.156+0000 7f642eada700  2 req 4522146966294331133 0.239999235s s3:get_obj completing 
2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data
=0
2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry)
2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0
2024-01-03T17:40:24.156+0000 7f642eada700  2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104
2024-01-03T17:40:24.156+0000 7f642eada700  2 req 4522146966294331133 0.239999235s s3:get_obj http status=200
2024-01-03T17:40:24.156+0000 7f642eada700  1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ======
2024-01-03T17:40:24.156+0000 7f642eada700  1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel 
Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s
2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/*
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0-
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120"
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS"
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443
2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https
2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196
2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET
2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4
2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4
2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000
2024-01-03T17:40:24.160+0000 7f6503483700  1 ====== starting new request req=0x7f640ca95730 =====


Thanks,
Nathan
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux