Greetings all and thank you for your contribution to this discussion.
Amar and Sankarshan, regarding your comments:* First of all, when we looked at log very early in project's life, our idea was based mostly on kernel logs (/var/log/messages). We decided, as a file-system, because it is very active with I/Os and should run for years together without failing, there should be NO log messages when the system is healthy, which should be 99%+ time.* Now, if there are no logs when everything is healthy, and most of the things are healthy 99% of the time, naturally the focus was not 'performance' of logging infra, but the correctness. This is where, the strict ordering through locks to preserve the timestamps of logs, and have it organized came by.
I believe this can be said about pretty much every software - nothing is designed to break (except maybe piñatas?).
I am curious though - when the project had begun, there were no INFO levels logs? These logs are logged even when everything is healthy.
In any case, we can set the history aside and focus on the present, and I believe many components were upgraded and improved over time and it looks like the logger was left behind, and is seems there are consequences for this now.
That is interesting observation. For this alone, can we have an option to disable all logging during regression? That would fasten up things for normal runs immediately.
We had an idea that could have improved CI time, but because the test module is built in such a way that each test starts gluster by itself, our idea was not feasible as it would require modification to all the tests (we wanted to run tests with no logging, and do log only when a test fails, on the re-run).
Anyway, IMO, turning off logging does not help us at all handle the core problem, and I also believe it may send a wrong message to our users ("our logging is not good enough so we decide to get rid of it instead of improving it").
And I fully agree with Sankarshan's comment:
If having quicker regression runs is the objective, then perhaps we should not look at turning off logging to accomplish them. Instead, there ought to be additional aspects which can be reviewed with turning off logging being the last available option.
Please make sure, we have minimum dependency on this project, and also the install steps are easy for your project. One of the reasons Gluster succeeded is because it is very easy to install (and uninstall), without much dependencies. I personally would like to keep that behavior same.
I did not mean for it to be a dependency for the project, I suggested that as an alternative logging solution that would be fully integrated into gluster.
I am in agreement with Sankarshan, who responded in another thread about tools like EFK/ELK stack for centralized logging. By moving to centralized logging, and having proper structure for logging, the initial goals we had wouldn't be broken, and we will have better performance even when we are logging in DEBUG mode.
I do not know enough about EFK/ELK to base an opinion, but I understand from your comments that this is considered standard components for logging etc'.
I'm not against this at all but I do need to investigate more before I could share my thoughts on the technical aspects.
[Amar] This, in my opinion is a good reason to undertake this activity. Mainly because we should be having our logging infra similar with other tools, and one shouldn't be having a learning curve to understand gluster's logging.
[Sankarshan] The original proposal from Barak has merit for planning towards an alpha form to be available.
I'm very happy to hear this, as it gives me confidence in my actions, thank you.
Unfortunately, I've been informed that I'm not to pursue this initiative at the current time, as there are other, higher, priorities that requires resources and it was decided not to dedicate them for this initiative.
I do hope that when the time is right, we could revisit this discussion and see how we can advance on that front.
If anyone has insight he'd like to share, I'd be happy to discuss it.
Thank you all very much,
On Wed, Nov 27, 2019 at 4:33 AM Sankarshan Mukhopadhyay <sankarshan.mukhopadhyay@xxxxxxxxx> wrote:
_______________________________________________On Wed, Nov 27, 2019 at 7:44 AM Amar Tumballi <amarts@xxxxxxxxx> wrote:Hi Barak,My replies inline.On Thu, Nov 21, 2019 at 6:34 PM Barak Sason Rofman <bsasonro@xxxxxxxxxx> wrote:
[snip]Initial tests, done by removing logging from the regression testing, shows an improvement of about 20% in run time. This indicates we’re taking a pretty heavy performance hit just because of the logging activity.
That is interesting observation. For this alone, can we have an option to disable all logging during regression? That would fasten up things for normal runs immediately.If having quicker regression runs is the objective, then perhaps we should not look at turning off logging to accomplish them. Instead, there ought to be additional aspects which can be reviewed with turning off logging being the last available option.
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968
Gluster-devel mailing list
Gluster-devel@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-devel
--
Barak Sason Rofman
Gluster Storage Development
_______________________________________________ Community Meeting Calendar: APAC Schedule - Every 2nd and 4th Tuesday at 11:30 AM IST Bridge: https://bluejeans.com/441850968 NA/EMEA Schedule - Every 1st and 3rd Tuesday at 01:00 PM EDT Bridge: https://bluejeans.com/441850968 Gluster-devel mailing list Gluster-devel@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-devel
