It could be interesting to support snapshots of attribute trees here. We could do something with a concurrent b+tree within the entry where we have: attribute tree: pinned-initial-state: write-state: That way we only copy-on-write what we need into the write state, and we have an easy revert path (remove the write state), less mallocs etc. This would save a fair bit of copying I think, and would allow us to have those changes in the entry as we go. It'd especially pay off on large attributes/entries with huge amounts of attributes or many values in a single attribute. But I think yeah, some more thought needed here .... but good research :) > On 8 May 2020, at 05:31, Mark Reynolds <mreynolds@xxxxxxxxxx> wrote: > > As we've been discussing performance recently, I did a little checking on how many times we duplicate a slapi_entry when doing a single modification. I updated an entry like this (which is actually pretty small): > > dn: uid=mareynol,ou=people,dc=example,dc=com > uid: mareynol > objectClass: top > objectClass: person > objectClass: organizationalperson > objectClass: inetOrgPerson > objectClass: ntUser > ntUserDeleteAccount: true > sn: Reynolds > givenName: Mark > initials: T > cn: Mark T. Reynolds > ntUserCodePage: 0 > ntUserAcctExpires: 9223372036854775807 > ntUserDomainId: Mareynol > ntUniqueId: 76ae38ecf89b884cbfe2b713082f16a3 > > > So every time the server duplicates this particular entry it triggers 159 mallocs and 64 frees. The more attributes the entry has the more more mallocs would be required to copy it. I found that when I just change a single attribute in this entry that the server calls slapi_entry_dup() 8 times! (that's over 1200 mallocs and 500 frees) The entry is duplicated two times by the MEP plugin for just being enabled out of the box. We also call backentry_dup() two times which is even more expensive, but I don't think we can avoid that one as it's needed for backend plugin transactions. Below are the stack traces from each "duplication" in order. Surely I would hope we can reduce this number of entry dups for a single modify operation ... > > > #0 slapi_entry_dup (e=0x7fc881864240) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b062528b in internal_plugin_search_entry_callback (e=0x7fc881864240, callback_data=0x7fc8855fcc20) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531 > #2 0x00007fc8b0624549 in internal_srch_entry_callback (be=0x7fc8aee457e0, conn=0x0, op=0x7fc88263b400, e=0x7fc881864240) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99 > #3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc8825fe120, e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0, > send_result=0, nentries=0, urls=0x0) at ../389-ds-base/ldap/servers/slapd/result.c:1507 > #4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc8825fe120, e=0x7fc881864240, ectrls=0x0, attrs=0x0, attrsonly=0) > at ../389-ds-base/ldap/servers/slapd/result.c:1050 > #5 0x00007fc8b060ded1 in send_entry (pb=0x7fc8825fe120, e=0x7fc881864240, operation=0x7fc88263b400, attrs=0x0, attrsonly=0, > pnentries=0x7fc8855f855c) at ../389-ds-base/ldap/servers/slapd/opshared.c:1132 > #6 0x00007fc8b060e497 in iterate (pb=0x7fc8825fe120, be=0x7fc8aee457e0, send_result=1, pnentries=0x7fc8855f855c, pagesize=-1, > pr_statp=0x7fc8855f854c) at ../389-ds-base/ldap/servers/slapd/opshared.c:1463 > #7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc8825fe120, send_result=1, nentries=0x7fc8855f855c, pagesize=-1, > pr_stat=0x7fc8855f854c) at ../389-ds-base/ldap/servers/slapd/opshared.c:1680 > #8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc8825fe120, send_result=1) at ../389-ds-base/ldap/servers/slapd/opshared.c:875 > #9 0x00007fc8b06258c8 in search_internal_callback_pb (pb=0x7fc8825fe120, callback_data=0x7fc8855fcc20, > prc=0x7fc8b062534c <internal_plugin_result_callback>, psec=0x7fc8b0625257 <internal_plugin_search_entry_callback>, > prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730 > #10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc8825fe120) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594 > #11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc8825fe120) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504 > #12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry (dn=0x7fc8855fccf0, attrs=0x0, ret_entry=0x7fc8855fccd8, > component_identity=0x7fc8ac713320) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864 > #13 0x00007fc8b0632eea in get_entry (pb=0x7fc8825fe000, dn=0x7fc8abe82fa0 "uid=mareynol,ou=people,dc=example,dc=com") > at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210 > #14 0x00007fc8b060405f in op_shared_modify (pb=0x7fc8825fe000, pw_change=0, old_pw=0x0) > at ../389-ds-base/ldap/servers/slapd/modify.c:726 > #15 0x00007fc8b06033bb in do_modify (pb=0x7fc8825fe000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #16 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acf6c0, op=0x7fc8ac85a000, pb=0x7fc8825fe000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #17 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #18 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #19 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #20 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68000) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b0625d24 in slapi_search_internal_get_entry (dn=0x7fc8825facf0, attrs=0x0, ret_entry=0x7fc8825facd8, component_identity=0x7fc8ac713320) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873 > #2 0x00007fc8b0632eea in get_entry (pb=0x7fc880bff000, dn=0x7fc880c2c320 "uid=mareynol,ou=people,dc=example,dc=com") > at ../389-ds-base/ldap/servers/slapd/pw_retry.c:210 > #3 0x00007fc8b060405f in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:726 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at ../389-ds-base/ldap/servers/slapd/entry.c:2057 > #1 0x00007fc8ac53b501 in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:614 > #2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #4 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #5 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc88267ebc0) at ../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86 > #2 0x00007fc8ac53b572 in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:622 > #3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68180) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac4fe456 in backentry_dup (e=0x7fc880c7d000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/backentry.c:86 > #2 0x00007fc8ac53b846 in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:680 > #3 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #4 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #5 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #6 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #7 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #8 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #9 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc882667180) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b062528b in internal_plugin_search_entry_callback (e=0x7fc882667180, callback_data=0x7fc8825fa9b0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:531 > #2 0x00007fc8b0624549 in internal_srch_entry_callback (be=0x7fc8aee457e0, conn=0x0, op=0x7fc880c3c400, e=0x7fc882667180) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:99 > #3 0x00007fc8b063a053 in send_ldap_search_entry_ext (pb=0x7fc880bff120, e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0, send_result=0, nentries=0, > urls=0x0) at ../389-ds-base/ldap/servers/slapd/result.c:1507 > #4 0x00007fc8b0639194 in send_ldap_search_entry (pb=0x7fc880bff120, e=0x7fc882667180, ectrls=0x0, attrs=0x0, attrsonly=0) > at ../389-ds-base/ldap/servers/slapd/result.c:1050 > #5 0x00007fc8b060ded1 in send_entry (pb=0x7fc880bff120, e=0x7fc882667180, operation=0x7fc880c3c400, attrs=0x0, attrsonly=0, pnentries=0x7fc8825f62ec) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1132 > #6 0x00007fc8b060e497 in iterate (pb=0x7fc880bff120, be=0x7fc8aee457e0, send_result=1, pnentries=0x7fc8825f62ec, pagesize=-1, pr_statp=0x7fc8825f62dc) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1463 > #7 0x00007fc8b060e8a5 in send_results_ext (pb=0x7fc880bff120, send_result=1, nentries=0x7fc8825f62ec, pagesize=-1, pr_stat=0x7fc8825f62dc) > at ../389-ds-base/ldap/servers/slapd/opshared.c:1680 > #8 0x00007fc8b060d444 in op_shared_search (pb=0x7fc880bff120, send_result=1) at ../389-ds-base/ldap/servers/slapd/opshared.c:875 > #9 0x00007fc8b06258c8 in search_internal_callback_pb (pb=0x7fc880bff120, callback_data=0x7fc8825fa9b0, prc=0x7fc8b062534c <internal_plugin_result_callback>, > psec=0x7fc8b0625257 <internal_plugin_search_entry_callback>, prec=0x7fc8b06252e2 <internal_plugin_search_referral_callback>) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:730 > #10 0x00007fc8b062542e in search_internal_pb (pb=0x7fc880bff120) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:594 > #11 0x00007fc8b06251ca in slapi_search_internal_pb (pb=0x7fc880bff120) at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:504 > #12 0x00007fc8b0625cb8 in slapi_search_internal_get_entry (dn=0x7fc8abe90240, attrs=0x0, ret_entry=0x7fc8825faa68, component_identity=0x7fc8ac6985e0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:864 > #13 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at ../389-ds-base/ldap/servers/plugins/mep/mep.c:2169 > #14 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/plugins/mep/mep.c:2303 > #15 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00, operation=461, pb=0x7fc880bff000, call_one=0) at ../389-ds-base/ldap/servers/slapd/plugin.c:2030 > #16 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00, operation=461, pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/plugin.c:1973 > #17 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000, whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442 > #18 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687 > #19 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #20 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #21 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #22 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #23 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #24 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #25 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68300) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8b0625d24 in slapi_search_internal_get_entry (dn=0x7fc8abe90240, attrs=0x0, ret_entry=0x7fc8825faa68, component_identity=0x7fc8ac6985e0) > at ../389-ds-base/ldap/servers/slapd/plugin_internal_op.c:873 > #2 0x00007fc8ac4ca221 in mep_pre_op (pb=0x7fc880bff000, modop=4) at ../389-ds-base/ldap/servers/plugins/mep/mep.c:2169 > #3 0x00007fc8ac4ca64f in mep_mod_pre_op (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/plugins/mep/mep.c:2303 > #4 0x00007fc8b061ed1c in plugin_call_func (list=0x7fc8ac6c3e00, operation=461, pb=0x7fc880bff000, call_one=0) at ../389-ds-base/ldap/servers/slapd/plugin.c:2030 > #5 0x00007fc8b061eb85 in plugin_call_list (list=0x7fc8aef6fe00, operation=461, pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/plugin.c:1973 > #6 0x00007fc8b061b912 in plugin_call_plugins (pb=0x7fc880bff000, whichfunction=461) at ../389-ds-base/ldap/servers/slapd/plugin.c:442 > #7 0x00007fc8ac53b87c in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:687 > #8 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #9 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #10 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #11 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #12 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #13 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #14 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > #0 slapi_entry_dup (e=0x7fc880c68180) at ../389-ds-base/ldap/servers/slapd/entry.c:1997 > #1 0x00007fc8ac53bfb6 in ldbm_back_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/back-ldbm/ldbm_modify.c:844 > #2 0x00007fc8b0604eba in op_shared_modify (pb=0x7fc880bff000, pw_change=0, old_pw=0x0) at ../389-ds-base/ldap/servers/slapd/modify.c:1021 > #3 0x00007fc8b06033bb in do_modify (pb=0x7fc880bff000) at ../389-ds-base/ldap/servers/slapd/modify.c:380 > #4 0x0000000000418c63 in connection_dispatch_operation (conn=0x7fc8a7acfa50, op=0x7fc8ac85a000, pb=0x7fc880bff000) > at ../389-ds-base/ldap/servers/slapd/connection.c:624 > #5 0x000000000041ad49 in connection_threadmain () at ../389-ds-base/ldap/servers/slapd/connection.c:1753 > #6 0x00007fc8b00eeb34 in _pt_root () from target:/lib64/libnspr4.so > #7 0x00007fc8b00834e2 in start_thread () from target:/lib64/libpthread.so.0 > #8 0x00007fc8afea26a3 in clone () from target:/lib64/libc.so.6 > > > -- > > 389 Directory Server Development Team > _______________________________________________ > 389-devel mailing list -- 389-devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to 389-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/389-devel@xxxxxxxxxxxxxxxxxxxxxxx — Sincerely, William Brown Senior Software Engineer, 389 Directory Server SUSE Labs _______________________________________________ 389-devel mailing list -- 389-devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to 389-devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@xxxxxxxxxxxxxxxxxxxxxxx
