Daniel; As I said, I don't actually KNOW most of this. As such, what I laid out was conceptual. Ceph would need to be implemented to perform these operations in parallel, or not. Conceptually, those areas where operations can be parallelized, making them parallel would improve wall clock performance in 80% - 90% of cases, thus making this configurable wouldn't make sense. That said, I don't know which route the developers went. All I know is that the client transfers each chunk to the master for its PG, and the master sends it on to the replicas. I suspect that replicas must acknowledge the chunk before the master finishes the synchronous operation. I suspect that all replicas are transferred (from the master) in parallel. Given the maturity of Ceph, I suspect this has already been done, unless the developers ran into a significant issue, but I don't know. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International, Inc. DHilsbos@xxxxxxxxxxxxxx www.PerformAir.com -----Original Message----- From: Daniel Mezentsev [mailto:dan@xxxxxxxxxx] Sent: Friday, July 17, 2020 4:14 PM To: Dominic Hilsbos Cc: ceph-users@xxxxxxx Subject: Re: ceph/rados performace sync vs async Hi Dominic, Pool configured as replica, replica size is 2 (i know it's not recommended, but it's test environment, and i ca't afford more nodes running at home). What you are saying - makes sense for me. Yoo mentioned that even for sync IO some jobs can be done in parallel. That is above my current level of ceph knowledge, but sounds like very interesting part to take a look at. Can you give me initial direction where to start, and is that options are configurable ? > Daniel; > > How is your pool configured? Replica, or Erasure-Coded? > > I don't actually know any of this, but... > > I would expect that a synchronous call to a replica pool (R=3) would > look something like this: > Client --> PG Master Host (data) > PG Master Host --> Local Disk (data) > PG Master Host --> PG Replica Host 1 (data) > PG Master Host --> PG Replica Host 2 (data) > PG Replica Host 1 --> Disk (data) > PG Replica Host 2 --> Disk (data) > PG Replica Host 1 --> PG Master Host (ack) > PG Replica Host 2 --> PG Master Host (ack) > PG Master Host --> Client (ack) > > Some of that can happen in parallel, for instance the master could > be transferring the file to both replica hosts, while also writing > it to disk. > > You can imagine why a synchronous call could be significantly slower > though, yes? > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International, Inc. > DHilsbos@xxxxxxxxxxxxxx > www.PerformAir.com[1] > > -----Original Message----- > From: Daniel Mezentsev [mailto:dan@xxxxxxxxxx] > Sent: Friday, July 17, 2020 3:05 PM > To: ceph-users@xxxxxxx > Subject: ceph/rados performace sync vs async > > Hi All, > > I started a small project related to metrics collection and > processing, Ceph was chosen as a storage backend. Decided to use rados > directly, to avoid any additional layers. I got a very simple client - > it works fine, but performance is very low. Can't get more than > 30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be > mentioned about the client - I'm using sbcl (yep lisp). Call to rados > API is just cffi call. > > Did try in async mode. Wow ! Saturated network bandwidth for large > objects (4Mb and bigger), for small objects - saturated OSD IOPS > ~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty > reasonable. Bottom line - issue not with the lisp client - im getting > close to C performance, difference is sync vs async IO. > > Why it's so big - sync operations are approx 2-3 times slower then async . > Daniel Mezentsev, founder > (+1) 604 313 8592. > Soleks Data Group. > Shaping the clouds. > _______________________________________________ > ceph-users mailing list -- ceph-users@xxxxxxxxx unsubscribe send an > email to ceph-users-leave@xxxxxxx Links: ------ [1] http://www.PerformAir.com Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds. _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |