Re: Unindexed search even on indexed database

[Pierre Rogier <progier@xxxxxxxxxx>]

Hi Jan, 

IMHO the allids threshold has been reached for the "issued" value for entryStatus  index
 (in other words: too many entries have  entryStatus: issued: )
You may want to grep for "conn=115 op=1" in access log and check how many entries where returned by the search
If the value is higher than allids threshold then it is normal  

Note: you may increase the allids threshold and reindex
   but that is not always the best thing to do. 

(It depends of the ratio of entries having entryStatus: issued versus the total number of entries in the backend:

  if most of the entries are in issued state it is better to keep the search unindexed:

  there is no real interest to manage huge index when we have to return
   most of the entries)

Regards
   Pierre 

On Thu, Mar 4, 2021 at 10:18 AM Jan Tomasek <jan@xxxxxxxxxx> wrote:

Hello,

I'm worrying about log lines:

[04/Mar/2021:10:08:47.982170561 +0100] - NOTICE - ldbm_back_search -
Unindexed search: search base="o=tcs2,o=apps,dc=cesnet,dc=cz" scope=2
filter="(entryStatus=issued)" conn=115 op=1

Index is defined:

# dsconf -D "cn=Directory Manager" -w "$pswd" ldap://localhost backend
index get TCS2_apps_cesnet_cz --attr entryStatus
dn: cn=entryStatus,cn=index,cn=TCS2_apps_cesnet_cz,cn=ldbm
database,cn=plugins,cn=config
cn: entryStatus
nsIndexType: eq
nsIndexType: pres
nsSystemIndex: False
objectClass: top
objectClass: nsIndex

Database is freshly reindexed:

# dsconf -D "cn=Directory Manager" -w "$pswd" ldap://localhost backend
index reindex TCS2_apps_cesnet_cz --attr entryStatus
Index task index_attrs_03042021_100813 completed successfully
Successfully reindexed database
# tail /var/log/dirsrv/slapd-cml3/errors
[04/Mar/2021:10:08:19.181006893 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 43000 entries (83%).
[04/Mar/2021:10:08:19.304566154 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 44000 entries (85%).
[04/Mar/2021:10:08:19.430861272 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 45000 entries (86%).
[04/Mar/2021:10:08:19.554529568 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 46000 entries (88%).
[04/Mar/2021:10:08:19.671814136 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 47000 entries (90%).
[04/Mar/2021:10:08:19.791473662 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 48000 entries (92%).
[04/Mar/2021:10:08:19.911157930 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 49000 entries (94%).
[04/Mar/2021:10:08:20.032595700 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 50000 entries (96%).
[04/Mar/2021:10:08:20.153813121 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 51000 entries (98%).
[04/Mar/2021:10:08:20.244942556 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Finished indexing.

But server is still complaining:

# time ldapsearch -H ldap://localhost -x -b
o=TCS2,o=apps,dc=cesnet,dc=cz '(entryStatus=issued)'
# extended LDIF
#
# LDAPv3
# base <o=TCS2,o=apps,dc=cesnet,dc=cz> with scope subtree
# filter: (entryStatus=issued)
# requesting: ALL
#

# search result
search: 2
result: 0 Success

# numResponses: 1

real    0m0.920s
user    0m0.014s
sys     0m0.001s
# tail /var/log/dirsrv/slapd-cml3/errors
...
[04/Mar/2021:10:08:20.153813121 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Indexed 51000 entries (98%).
[04/Mar/2021:10:08:20.244942556 +0100] - INFO - bdb_db2index -
TCS2_apps_cesnet_cz: Finished indexing.
[04/Mar/2021:10:08:47.982170561 +0100] - NOTICE - ldbm_back_search -
Unindexed search: search base="o=tcs2,o=apps,dc=cesnet,dc=cz" scope=2
filter="(entryStatus=issued)" conn=115 op=1

Some DB were created during reindex:

# ls -l /var/lib/dirsrv/slapd-cml3/db/TCS2_apps_cesnet_cz/
total 358212
-rw------- 1 dirsrv dirsrv     16384 Feb 22 10:08 aci.db
-rw------- 1 dirsrv dirsrv   2760704 Feb 22 10:09 ancestorid.db
-rw------- 1 dirsrv dirsrv  19668992 Mar  3 16:53 cn.db
-rw------- 1 dirsrv dirsrv        51 Mar  4 09:54 DBVERSION
-rw------- 1 dirsrv dirsrv     24576 Mar  3 16:53 dc.db
-rw------- 1 dirsrv dirsrv  13254656 Feb 22 10:09 entryrdn.db
-rw------- 1 dirsrv dirsrv   1114112 Mar  4 10:08 entryStatus.db
-rw------- 1 dirsrv dirsrv     16384 Feb 22 10:07 entryusn.db
-rw------- 1 dirsrv dirsrv   3063808 Mar  3 16:54 givenName.db
-rw------- 1 dirsrv dirsrv 285548544 Mar  4 10:09 id2entry.db
-rw------- 1 dirsrv dirsrv  16056320 Mar  3 16:54 mail.db
-rw------- 1 dirsrv dirsrv     16384 Feb 22 10:08 nscpEntryDN.db
-rw------- 1 dirsrv dirsrv   3891200 Feb 22 10:09 nsuniqueid.db
-rw------- 1 dirsrv dirsrv     24576 Feb 22 10:09 numsubordinates.db
-rw------- 1 dirsrv dirsrv   1466368 Feb 22 10:09 objectclass.db
-rw------- 1 dirsrv dirsrv    811008 Feb 22 10:09 parentid.db
-rw------- 1 dirsrv dirsrv    258048 Mar  4 10:00 replication_changelog.db
-rw------- 1 dirsrv dirsrv   3735552 Mar  3 16:55 sn.db
-rw------- 1 dirsrv dirsrv    335872 Mar  4 09:26 tcs2certificate.db
-rw------- 1 dirsrv dirsrv     32768 Mar  3 16:55 tcs2cesnetorgdn.db
-rw------- 1 dirsrv dirsrv   2826240 Mar  3 16:55 tcs2crtserialnumber.db
-rw------- 1 dirsrv dirsrv   3809280 Mar  3 16:55 tcs2crtsubject.db
-rw------- 1 dirsrv dirsrv     16384 Mar  3 16:55 tcs2idpentityid.db
-rw------- 1 dirsrv dirsrv   3276800 Mar  3 16:56 tcs2requesterdn.db
-rw------- 1 dirsrv dirsrv    393216 Mar  3 16:56 tcs2role.db
-rw------- 1 dirsrv dirsrv   3219456 Mar  3 16:56 telephoneNumber.db
-rw------- 1 dirsrv dirsrv    516096 Mar  3 16:56 uid.db
-rw------- 1 dirsrv dirsrv    647168 Mar  3 16:57 unstructuredname.db

Any ideas how to fix the problem?
--
-----------------------
Jan Tomasek aka Semik
http://www.tomasek.cz/
_______________________________________________
389-users mailing list -- 389-users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to 389-users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

--

--

389 Directory Server Development Team

_______________________________________________
389-users mailing list -- 389-users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to 389-users-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-users@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure