Hi Robert,
It definitely is exciting I think. Keep up the good work! :)
Mark
On 11/05/2015 09:14 AM, Robert LeBlanc wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Thanks Gregory,
People are most likely busy and haven't had time to digest this and I
may be expecting more excitement from it (I'm excited due to the
results and probably also that such a large change still works). I'll
keep working towards a PR, this was mostly proof of concept, now that
there is some data I'll clean up the code.
I was thinking that a config option to choose the scheduler would be a
good idea. In terms of the project what is the better approach: create
a new template and each place the template class is instantiated
select the queue, or perform the queue selection in the same template
class, or something else I haven't thought of.
Are there public teuthology-openstack systems that could be used for
testing? I don't remember, I'll have to search back through the
mailing list archives.
I appreciate all the direction as I've tried to figure this out.
Thanks,
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Nov 4, 2015 at 8:20 PM, Gregory Farnum wrote:
On Wed, Nov 4, 2015 at 7:00 PM, Robert LeBlanc wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Thanks for your help on IRC Samuel. I think I found where I made a
mistake. I'll do some more testing. So far with max_backfills=1 on
spindles, the impact of setting an OSD out and in on a saturated
cluster seems to be minimal. On my I/O graphs it is hard to tell where
the OSD was out and in recovering. If I/O becomes blocked, it seems
that they don't linger around long. All of the clients report getting
about the same amount of work done with little variance so no one
client is getting indefinitely blocked (or blocked for really long
times) causing the results between clients to be skewed like before.
So far this queue seems to be very positive. I'd hate to put a lot of
working getting this ready to merge if there is little interest in it
(a lot of things to do at work and some other things I'd like to track
down in the Ceph code as well). What are some of the next steps for
something like this, meaning a pretty significant change to core code?
Well, step one is to convince people it's worthwhile. Your performance
information and anecdotal evidence of client impact is a pretty good
start. For it to get merged:
1) People will need to review it and verify it's not breaking anything
they can identify from code. Things are a bit constricted right now,
but this is pretty small and of high interest so I make no promises
for the core team but submitting a PR will be the way to start.
Getting positive buy-in from other contributors who are interested in
performance will also push it up the queue.
2) There will need to be a lot of testing on something like this.
Everything has to pass a run of the RADOS suite. Unfortunately this is
a bad month for that as the lab is getting physically shipped around
in a few weeks, so if you can afford to make it happen with the
teuthology-openstack stuff that will accelerate the timeline a lot (we
will still need to run it ourselves but once it's passed externally we
can put it in a lot more test runs we expect to pass, instead of in a
bucket with others that will all get blocked on any one failure).
3) For a new queuing system I suspect that rather than a direct merge
to default master, Sam will want to keep both in the code for a while
with a config value and run a lot of the nightlies on this one to
tease out any subtle races and bugs.
4) Eventually we become confident that it's in good shape and it
replaces the old queue.
Obviously those are the optimistic steps. ;)
-Greg
Thank you to all who took time to help point me in the right direction.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Wed, Nov 4, 2015 at 12:49 PM, Samuel Just wrote:
I didn't look into it closely, but that almost certainly means that
your queue is reordering primary->replica replicated write messages.
-Sam
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.2.3
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJWOsY1CRDmVDuy+mK58QAA9LIQALIUgbS4BuDS704HPOpA
XwvGxspelMCaBkLHLgiHU4T/Jc8JaXhgdRMwMiKeLI246Z7hRngSGlIDYc4+
nP4kWZIkwbJeTa/Z6bM6C3itFtJmQpkPvdjI+GiME5ZdYvFgCZQyDD71rqja
H14m0+JsEaIHQF0JZz6OyNxbyRWsM+M68nOvpAx8/fOGHBC/0VwPbLrOUP9O
3J3NvbhN9xlYJeivXSAyzxmHQDD8mO1c1AUTrHgnTViD2k3fmcH0mOHIJ+jn
ARZbeLN3hlXG0i9PHpnHzBVNSxsfb5VPxX970R3gvRWIt40QV/QL7q2SajWP
ofxgEpkaO48ANQSYDlqSNcM+w46TtgcJljtX0vbrHIW3Skyaz4UZQ/dzX4lX
a5Zzk01oFwXfMd10KgVbJf78qVYHy2r5aq46iFnrFLU43iy+Qve7Kex4XZFi
vPFFVea89Of838NqTxW21+3oJthrz1g7RKHghZAbXaj3WKchuEU+uVG4XTo1
0PU4a5ZYVTH6zYHpwJo2/89OzdkBe9S6s00+4JmfVWWEhb6+QwUjBQp1TJbB
TnMzSKfzgRyi/wHThv2XcZN12tttZMM2L4Ea3mHG+cxOTTZ1opv8/H2mprm8
7UuO4vk5K0c4IwPVmt9m5DTVhyn4hZ/QJmc+NARD3zc1u3qWFLkH2WaRMpBb
mRWA
=kgAl
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.2.3
Comment: https://www.mailvelope.com
wsFcBAEBCAAQBQJWO3JgCRDmVDuy+mK58QAAeCcP/jHnG3r257cdcRYZzg9o
iMOxnuKAXNnwscYzJysCHsoQ2S3dB9SCxt8r+QvDo09IkXzarFaW647nzG6H
zeCtbhx2NFU/jOqPip/8XDUaDYlDrjHuskDJwz+jzoaZfWPjLfPkmETU/8vh
mrGZH+kjYuu1WhmM8cGJZJLrKA7C2OPTAU5PRmx5enClHXhdxyskZ7BUxcXp
uPJJg7pemT/qaJPrO7e7wwhYw43GaeSULp8QGFsqireCwbv9mndB7bbOa40U
ElHmgWgcG1UkkydW/U9DaJHM52ZbrAuG7XkZRsmB1oTmVriEoOFYSiGv5F+R
Mjxe9OlqiL9Fd/AQXunAAMdwIU5T3mlkrxMvhroRkW2+EerrRVW3JbJ8gmQ9
lXPRw9RxcQY5m8S+8+CWikBHvsRBCXEGA8tXUYqLuDJKpRHeCo7PpONS3III
QB+tgWaMteoeJGZ7nGLFcaKxTGa1tNKju4M2845/L8Fawy8jdYYcLqOTUs80
M1gpQ0UHzTXdQEdQnufxgaCFfwblF5vIlr6qd89rR5m0eJipElQLi2Uh0Zd3
0t0i0xtFdprkxDmzX/bzbARAnlS1cz/yoB85r3JxeNPev671mocQc0uyFkt7
P04ogGWzLBN5B4nWNWDznOZS52G+vhkFxryUyl9+LDafAKiTTPmhB/LXPMs+
7ny7
=Xg0t
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
