Hi Dominic,
Pool configured as replica, replica size is 2 (i know it's not
recommended, but it's test environment, and i ca't afford more nodes
running at home). What you are saying - makes sense for me.
Yoo mentioned that even for sync IO some jobs can be done in parallel.
That is above my current level of ceph knowledge, but sounds like very
interesting part to take a look at. Can you give me initial direction
where to start, and is that options are configurable ?
Daniel;
How is your pool configured? Replica, or Erasure-Coded?
I don't actually know any of this, but...
I would expect that a synchronous call to a replica pool (R=3) would
look something like this:
Client --> PG Master Host (data)
PG Master Host --> Local Disk (data)
PG Master Host --> PG Replica Host 1 (data)
PG Master Host --> PG Replica Host 2 (data)
PG Replica Host 1 --> Disk (data)
PG Replica Host 2 --> Disk (data)
PG Replica Host 1 --> PG Master Host (ack)
PG Replica Host 2 --> PG Master Host (ack)
PG Master Host --> Client (ack)
Some of that can happen in parallel, for instance the master could
be transferring the file to both replica hosts, while also writing
it to disk.
You can imagine why a synchronous call could be significantly slower
though, yes?
Thank you,
Dominic L. Hilsbos, MBA
Director – Information Technology
Perform Air International, Inc.
DHilsbos@xxxxxxxxxxxxxx
www.PerformAir.com[1]
-----Original Message-----
From: Daniel Mezentsev [mailto:dan@xxxxxxxxxx]
Sent: Friday, July 17, 2020 3:05 PM
To: ceph-users@xxxxxxx
Subject: ceph/rados performace sync vs async
Hi All,
I started a small project related to metrics collection and
processing, Ceph was chosen as a storage backend. Decided to use rados
directly, to avoid any additional layers. I got a very simple client -
it works fine, but performance is very low. Can't get more than
30-35MBsec. Rados bench shows 200MBsec for my test pool. Should be
mentioned about the client - I'm using sbcl (yep lisp). Call to rados
API is just cffi call.
Did try in async mode. Wow ! Saturated network bandwidth for large
objects (4Mb and bigger), for small objects - saturated OSD IOPS
~2.4KIOPS for 8 SAS disks, so ~300 IOPS per disk - that sounds pretty
reasonable. Bottom line - issue not with the lisp client - im getting
close to C performance, difference is sync vs async IO.
Why it's so big - sync operations are approx 2-3 times slower then async .
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxxxx unsubscribe send an
email to ceph-users-leave@xxxxxxx
Links:
------
[1] http://www.PerformAir.com
Daniel Mezentsev, founder
(+1) 604 313 8592.
Soleks Data Group.
Shaping the clouds.
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
