When checking the RGW logs I can confirm that it is in fact the same issue as the one in the issue. 2023-09-20T12:52:06.670+0000 7f216d702700 1 -- xxx.xxx.58.15:0/758879303 --> [v2:xxx.xxx.58.2:6816/8556,v1:xxx.xxx.58.2:6817/8556] -- osd_op(unknown.0.0:238 18.651 18:8a75a7b2:::39078a70-7768-48c8-96a5-1e13ced83b5b.58017020.1_videos%2f7.mp4:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60419) v8 -- 0x7f21dc00a420 con 0x7f21dc007820 You can find the OSD part of the log here - https://pastebin.com/nGQw4ugd For the record the version of the cluster when I’m able to replicate this is ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Regards, Ondrej > On 20. 9. 2023, at 11:25, Ondřej Kukla <ondrej@xxxxxxx> wrote: > > I was checking the tracker again and I found already fixed issue that seems to be connected with this issue. > > https://tracker.ceph.com/issues/44508 > > Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 > > What I’m still not understanding is why this is only happening when using s3website api. > > Is there someone who could shed some light on this? > > Regards, > > Ondrej > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx > On 19. 9. 2023, at 10:49, Ondřej Kukla <ondrej@xxxxxxx> wrote: > > Hello, > > In our deployment we are using the mix of s3 and s3website RGW. I’ve noticed strange behaviour when sending range requests to the s3website RGWs that I’m not able to replicate on the s3 ones. > > I’ve created a simple wrk LUA script to test sending range requests on tiny ranges so the issue is easily seen. > > When sending these requests against s3 RGW I can see that the amount of data read from Ceph is ± equivalent to what the RGW sends to the client. This change very dramatically when I’m doing the same test against s3website RGW. The read from Ceph is huge (3Gb/s compared to ~22Mb/s on s3 RGW) I seems to me like the RGW is reading the whole files and then sending just the range which is different then what s3 does. > > I do not understand why would s3website need to read that much from Ceph and I believe this is a bug - I was looking through the tracker and wasn’t able to find anything related to s3website and range requests. > > Did anyone else noticed this issue? > > You can replicate it by running this wrk command wrk -t56 -c500 -d5m http://${rgwipaddress}:8080/${bucket}/videos/ -s wrk-range-small.lua > > wrk script > > -- Initialize the pseudo random number generator > math.randomseed( os.time()) > math.random(); math.random(); math.random() > > i = 1 > > function request() > if i == 8 > then > i = 1 > end > > local nrangefrom = math.random() > local nrangeto = math.random(100) > local path = wrk.path > url = path..i..".mp4" > wrk.headers["Range"] = nrangefrom.."-"..nrangeto > i = i+1 > return wrk.format(nil, url) > end > > Kind regards, > > Ondrej > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxx > To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |