Finally got there, without TLS for now, that is tomorrows job!
I am posting my findings in case they can help someone in the
future.
As expected I was really not understanding how the parameters were used and therefore had them all wrong.
ptload does do recursive searches and the behaviour for the first (with attributeDescription: 1.1) seems to be completely OK, the problem is if the settings for the other searches are wrong, it doesn't progress to them.
It looks like saslauthd caches successfully authentications because I only see it in the traffic the first time I make a successful connection for a user, I didn't round to working out how to clear the cache, but as far as I can tell this step is runs after ptload gets the authorization details, so if ptloader doesn't work, no authentication.
Users are in ou=SBSUsers,ou=Users,ou=MyBusiness,dc=MyCompany,dc=com
Groups are in ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=com
Groups and users have 'sAMAccountName' which is single word, the part before @, for a group the 'name' attribute is the same single word, but for users it is firstname lastname (this is just how I put users and groups in the directory). To avoid confusion for anyone reading my conf later I am using 'sAMAccountName' for users and 'name' for groups although 'sAMAccountName' will also work for groups and may be more reliable for some setups.
Active directory stores group members in the 'member' attribute
of each group as dn's. There is no attribute that matches
sAMAccountNames to groups. Also there is no attribute of a user
which lists the groups it is in, that is to say, I can't see one
in the LDAP attributes and whilst memberOf can be used from
powershell I suspect it is doing something other than matching an
ldap attribute, because it returned nothing for me.
The token %u passes the full username including any domain from the client, %U strips the domain off if present - in my setup we only login with the part before the @ so these work the same, but hopefully I have used %U everywhere to ensure compatability with 'sAMAccountName'
The token I had failed to note earlier is %D which passes the dn that is found in the previous search, we need this to filter for groups by their 'member' attribute.
So I need to search for a user in the user base using a filter, but use the default user attribute instead of setting one (seems to default to (objectClass=*)).
I need to filter groups using the group base, there is no attribute option for the group. I think this just makes sure the member search only looks in groups, in my case I have used the filter to restrict to distribution groups by specifying not some security group types I have identified in my directory, they may not be correct or exhaustive for other directories. This is all used in the member search and is not a search in its own right, at least as far as I can tell.
I also need to search for members in the groups and I use the
filter method. With filter the attribute needs to return a single
value - the group name. It seems that with the attribute method
the attribute can hold multiple values so presumably you would use
this if your user has an attribute with a list of groups it is a
member of? So I need to match my user in the 'member' attribute
of the groups (limited by the group base and filter), which means
I need to supply the users dn instead of the login name, i.e. %D
instead of %U. And what I want back is the groups name or
sAMAccountName.
So when ptload finds the user, gets the users dn, and searches
for members of the groups by dn it returns all the (distribution)
groups that the user is a member of, and then I guess exits
successfully passing control to saslauthd to authenticate.
So, the section of my imapd.conf that does this, with our actual company name and bind credentials altered, but the ou's left the same otherwise, looks like this:
##
## Other LDAP items
## This is for AUTHORIZATION, we use saslauthd for AUTHENTICATION
##
# These may or may not be required, but since the idea is to keep
all
# the user and group information in active directory we must
surely need
# to tell cyrus how to find it?
#
# First we need to tell it to use ptloader for authorization
auth_mech: pts
#
# And tell ptloader to use LDAP
pts_module: ldap
ptloader_sock: /var/lib/cyrus/ptclient/ptsock
#
# The defaults for the cache settings should be fine
# db type defaults to twoskip, db_path to
configdirectory/ptscache.db
# other settings are only for kerberos module
ptscache_db: twoskip
ptscache_db_path: /var/lib/cyrus/ptclient/ptscache.db
#
# General settings
# Probably also useful to tell cyrus where the LDAP is
#ldap_uri: ldaps:/temp-std.MyCompany.local:636
ldap_uri: ldap://temp-std.MyCompany.local:389
ldap_bind_dn:
cn=auser,ou=SBSUsers,ou=Users,ou=MyBusiness,dc=MyCompany,dc=local
ldap_password: somethingsecret
# Don't attempt SASL for authorization, it is used for
authentication already
ldap_sasl: 0
# For ldaps we will need version 3
ldap_version: 3
# Make sure cyrus can find the CA file to accept LDAP servers
certificate
#ldap_ca_dir: /etc/ssl/certs/
# And ensure that we check the certificate
#ldap_verify_peer: 1
# Might also be worth specifying the ciphers we want
#ldap_ciphers: TLSv1.3:TLSv1.2:+TLSv1:+HIGH:!aNULL:@STRENGTH
# Set a limit on number of record for single query
ldap_size_limit: 100
#
# User lookups
# Set a default search base although it looks like we can set
separately for users and groups
# This filter works to make sure the account is a user and not
disabled
ldap_base: ou=MyBusiness,dc=MyCompany,dc=local
ldap_scope: sub
ldap_filter:
(&(objectClass=person)(sAMAccountName=%u)(!(userAccountControl=514)))
# But lets have a simpler testing filter
#ldap_filter: (sAMAccountName=%U)
#
#
# Groups - we will need these for shared folder ACIs
# Set a filter to identify a group, this one ensures it is a
distribution group
# and not a security group
ldap_group_base:
ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_group_scope: sub
ldap_group_filter:
(&(objectClass=group)(!(groupType=2147483656))(!(groupType=2147483652)))
#
# Method to extract members from the group, this is poorly
documented but after much trial an
# error, the 'member' attribute in AD groups contains
distinguished names (DNs) so need to
# use a filter to return all of the group names that contain the
DN for our supplied username
# in their 'member' attribute'. %D is the token for the user dn
# The attribute for the group name we set to 'name' although
'sAMAccountName' returns the
# same value it is a bit confusing when debugging.
ldap_member_base:
ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_member_scope: sub
ldap_member_method: filter
ldap_member_filter: (member=%D)
ldap_member_attribute: name
##
#
And with this configuration I can now authenticate users and
ptdump now has information to display, which includes the groups
for all users.
I still need to test with TLS re-enabled at each level (between cyrus and ldap server, and between client and cyrus) but I do not forsee this as an issue.
The only issue I see is that ptload has converted the group names to all lowercase and most of them had capitalisation, I assume this will confuse cyrus since the shared folders I will import have capitals but I'm sure I can rename them in cyradm if this is an issue and rejig the ACLS to suit.
I hope this info helps someone else.
I feel like I am getting somewhere.
If I change the search base back to dc=MyCompany,dc=local in imapd.conf (matching saslauthd), when I run imtest with pts enabled I do get searchResRef stanzas back from the server. These represent other branches of the directory.
Lightweight Directory Access Protocol
LDAPMessage searchResEntry(2) "CN=Test User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local" [1 result]
messageID: 2
protocolOp: searchResEntry (4)
searchResEntry
objectName: CN=Test User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local
attributes: 0 items
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL: ldap://ForestDnsZones.MyCompany.local/DC=ForestDnsZones,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL: ldap://DomainDnsZones.MyCompany.local/DC=DomainDnsZones,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResRef(2)
messageID: 2
protocolOp: searchResRef (19)
searchResRef: 1 item
LDAPURL: ldap://MyCompany.local/CN=Configuration,DC=MyCompany,DC=local
[Response To: 679124]
[Time: 0.000733000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResDone(2) success [1 result]
messageID: 2
protocolOp: searchResDone (5)
searchResDone
resultCode: success (0)
matchedDN:
errorMessage:
[Response To: 679124]
[Time: 0.000733000 seconds]When I look at the searches (and wireshark hung so I had to start a new session to get these) I see:
Pts enabled:
Lightweight Directory Access Protocol
LDAPMessage searchRequest(2) "dc=MyCompany,dc=local" wholeSubtree
messageID: 2
protocolOp: searchRequest (3)
searchRequest
baseObject: dc=MyCompany,dc=local
scope: wholeSubtree (2)
derefAliases: neverDerefAliases (0)
sizeLimit: 1
timeLimit: 5
typesOnly: False
Filter: (sAMAccountName=testuser)
filter: equalityMatch (3)
attributes: 1 item
AttributeDescription: 1.1
[Response In: 588]
PTS disabled:
Actually, this is coming through, saslauthd must have cached the last search and is authenticating without contacting the server, but I'm pretty sure the only difference in the search is that saslauthd is looking for attribute "dn" where ptload is looking for "1.1"
Whatever ldap_member_attribute I set, the search always has AttributeDescription: 1.1 so is it an issue with the attribute that ptload is requesting? I need to go through the man page for imapd.conf again and see if i have missed something that defines an attribute which is overriding ldap_user_attribute, ldap_group_attribute and ldap_member_attribute.
Thanks
Jim
On 17/06/2021 11:29, Jim Wallis wrote:Hi Andrew,
If I am understanding the packet logs correctly, the LDAP server is responding (only inserting the ldap parts of the log entry):
The search:
Lightweight Directory Access Protocol
LDAPMessage searchRequest(2) "ou=MyBusiness,dc=MyCompany,dc=local" wholeSubtree
messageID: 2
protocolOp: searchRequest (3)
searchRequest
baseObject: ou=MyBusiness,dc=MyComapny,dc=local
scope: wholeSubtree (2)
derefAliases: neverDerefAliases (0)
sizeLimit: 1
timeLimit: 5
typesOnly: False
Filter: (sAMAccountName=testuser)
filter: equalityMatch (3)
equalityMatch
attributeDesc: sAMAccountName
assertionValue: testuser
attributes: 1 item
AttributeDescription: 1.1
[Response In: 557126]The results:
Lightweight Directory Access Protocol
LDAPMessage searchResEntry(2) "CN=Test User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local" [1 result]
messageID: 2
protocolOp: searchResEntry (4)
searchResEntry
objectName: CN=Test User,OU=SBSUsers,OU=Users,OU=MyBusiness,DC=MyCompany,DC=local
attributes: 0 items
[Response To: 557125]
[Time: 0.047382000 seconds]
Lightweight Directory Access Protocol
LDAPMessage searchResDone(2) success [1 result]
messageID: 2
protocolOp: searchResDone (5)
searchResDone
resultCode: success (0)
matchedDN:
errorMessage:
[Response To: 557125]
[Time: 0.047382000 seconds]I did change my ldap base from dc=MyCompany,dc=local when I noticed it seemed to be querying additional parts of the directory, although I am quite sure all users are unique. It is now:
ldap_base: ou=MyBusiness,dc=MyCompany,dc=local
With the group and member bases set to DistributionGroups:
ldap_group_base: ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
ldap_member_base: ou=DistributionGroups,ou=MyBusiness,dc=MyCompany,dc=local
The LDAP is still mostly laid out as per the MS Small Business Server 2003 way of setting up the schema (but on 2012 R2), I have no idea if it is normal for an AD installation. The distribution groups ou is one I added to try to keep my distribution groups away from the security groups.
Anyway, going back to the results, I don't understand this at low level but you are asking about searchresref and the server seems to be giving searchResEntry - is this as simple as MS implementing non-standard methods?
But, in searchResEntry, whilst it finds the objectName, which is a DN, it is returning 0 attributes, so it can't be as simple as ptloader doesn't understand the response, the response also appears incomplete? (I have ldap_member_attribute: set and have tried all sorts of attributes that I know are in the directory, including sAMAccoutName which we can see it recognises in the search itself).
I already set the debug option on ptloader (forgot to say in first post) in cyrus.conf:
ptloader cmd="ptloader -d1" listen="var/lib/cyrus/ptclient/ptsock"
And have tried increasing from 1 but as others have reported it does not seem to affect verbosity. The debug output goes in syslog similar to the lines I had before although now that I at least have the bind working I now get:
cyrus/ptloader[63125]: searching ou=MyBusiness,dc=MyCompany,dc=local with (sAMAccountName=testuser) failed
Now, what is interesting and needs more investigation later, is that if I disable pts again and use the same imtest command, it first preforms a bind using the saslauthd user (so I know sasluthd is making the bind), the results do include searchResRef entries (several, I think because I am searching a at dc level?) and cyrus goes on to make a subsequent bind as testuser, using testuser's password (in the pts examples the initial bind is by the cyrus user, and there is no subsequent bind for the test user).
I need to make a site visit now so will dig into this some more later.
thanks for your help so far!
Jim
On 16/06/2021 18:45, AndrewHardy via Info wrote:Hi Jim,It sounds like you’ve spotted a few issues and have fixed them. I’m quite interested to know what your imapd configuration is for pts and also how the configuration knows what OU the user you’re testing resides in? I haven't personally set up LDAP auth with Cyrus although it’s been something I’ve been wanting to do for some time (so showing a bit of interest in this thread).When you mentioned nothing seems to get returned to ptloader, is it possible to confirm if there was a ldap response from the user search in the ldap packet trace?Was the result count returned and what was the format of searchresref? If that is returned on the wire (confirming user was found after the initial search) then I guess next step is to debug ptloader to see what that’s expecting to receive and whether the ldap response data gets there ? I’m assuming the search does find the user of interest and the result count is 1 and the distinguished name of the object is returned over the wire to the mail server? If the user is searched but not found, that’s likely a place to investigate e.g where in the tree is it trying to find the user - does the attribute you’re checking for and the user searched match up?For ptloader, there is a debug option. May be worth enabling that although not sure what that technically offers in terms of troubleshooting info. Im a bit unsure on how to attach strace in this context, which process ptloader operates under or whether it’s it’s own seperate process with its own pid. If it’s the later it’s probably going to be a bit easier. Bit of trial and error most likely.RegardsAndrewSent from my iPhoneOn 17/06/2021, at 03:48, Jim Wallis <jwallis@xxxxxxxxxxxxxxxxx> wrote:Hi Andrew,
This has been very helpful. I didn't think of using wireshark before, it has enabled me to at least sort out a typo in my bind DN , but I am still not much further on.
With auth_mech: pts commented out, and using an imtest line like:
imtest -u test -a test -w passfortest localhost
I can see from the wireshark capture that Saslauthd is binding as the test user and imtest authenticates OK
with auth_mech: pts uncommented I need to provide bind credentials in imapd.conf (I use the ones for my cyrus user so I can tell what is being passed). This cyrus user binds successefully and looks up the test user, but nothing seems to get returned to ptloader, so I probably need to spend more time thinking about what attribute I need to return (I have tried several likely candidates). It doesn't look as though saslauthd is used at all when I have pts on. Imtest responds not authenticated.
More work to do... I guess I need to learn how to use strace.
Thanks
Jim
On 15/06/2021 20:30, AndrewHardy via Info wrote:Hi Jim,I’m not sure specifically what your issue is here but I’ll help provide some options for debugging ldap (specific to the windows logging question). I’d suggest dropping the TLS 636 LDAPS and just start with standard LDAP 389, removing TLS complexity whilst you validate a working configuration.I’d suggest also just running wireshark on the domain controller where you expect ldap connections to reach and capture the 389 traffic with a filter of your source mail server (or component doing the authentication). You’d be able to assess the ldap flow; whether the server sees it or not (and what the response was from the server, if any).You could also open another terminal, open saslauthd with strace and dump the authentication process to file to analyse what may be happening. If you can’t wireshark the target authentication service, you could perhaps tcpdump on the ldap client side, drop that into a .pcap file then open this in wireshark on your desktop for further analysis.I’m hoping those potential options can help verify what is/isn’t happening. An absence of traffic leaving the server would imply something on the client side, ruling out the ldap server at fault - at least initially.Hope this helps simplify troubleshooting and helps confirm what is/isn’t happening at least from an ldap perspective. Unsure of enabling ldap debugging in windows, I’ve searched for that for years and never really come up with a good solution. Wireshark and/or tcpdump has helped me solve all problems I’ve had with ldap/authentication although ldaps can be a bit more problematic when just using packet tracing software. Can still come in useful though.RegardsAndrewSent from my iPhoneOn 16/06/2021, at 06:30, jwallis@xxxxxxxxxxxxxxxxx wrote:It seemed like it would be a good idea to use the existing Active Directory DC as the LDAP source for all mail users.I got cyrus (3.2.6 from Buster backports) running using saslauthd for authentication against the directory, and test users could authenticate OK and see their mailbox in Thunderbird. relevant entires in imapd.conf:allowplaintext: yessasl_mech_list: PLAINsasl_pwcheck_method: saslauthdBecause I want to use some groups for shared folders and distribution lists, I also want to authorize users and groups against the directory so enabled:auth_mech: ptspts_module: ldapptloader_sock: var/lib/cyrus/ptclient/ptsockldap_uri: ldaps:/companydomaincontroller:636ldap_bind_dn: a cn that works in other bindsldap_password: the password for aboveldap_sasl: 0ldap_version: 3ldap_ca_dir: /etc/ssl/certs (which is where the ca cert that works with saslauthd resides)ldap_verify_peer: 1ldap_base: base that other binds can successfully find users fromldap_scope: subldap_filter: (sAMAccountName=%u) (I have tried more complex filters and decided on this as one that should work)ldap_user_attribute: mailldap_size_limit: 1024I have commented out the group member stuff for now, need to walk before I can run!With these options enabled, no one can authenticate, even though my understanding is that authentication is distinct from authorization. Thunderbird users can no longer login and imtest for user cyrus (which is in the directory) gives the same output up to a line :C: A01 AUTHENTICATE PLAIN {a hash}S: A01 NO authentication failureAuthentication failed. generic failureSecurity strength factor: 256(the SSF is reported the same, but I have for now only asked for level 0)In /var/log/syslog I find the following lines:cyrus/ptloader[8230]: ldap_initialize failed (ldaps:/DC:636)cyrus/imap[8229]: timeout_select exiting. r = 1; errno = 0cyrus/imap[8229]: timeout_select: sock = 11, rp = 0x7ffecbb6ad30, wp = 0x0, sec = 30cyrus/imap[8229]: timeout_select exiting. r = 1; errno = 0cyrus/imap[8229]: ptload read data backcyrus/imap[8229]: ptload(): bad response from ptloader server: ptsmodule_connect() failedcyrus/imap[8229]: No data available at all from ptload()cyrus/imap[8229]: ptload completely failed: unable to canonify identifer: cyruscyrus/imap[8229]: SASL bad userid authenticatedcyrus/imap[8229]: badlogin: localhost [::1] PLAIN (-notset-) [SASL(-6): can't request information until later in exchange: Information that was requested is not yet available.]I have been searching for answers for days and at one point found a reference that claimed ptsloader is not enabled by default in Debian, so I have downloaded the source package and compiled cyrus-imaps using a configure script based on the Debain default config options with some extra options: --with-auth=pts --with-pts=ldap and --with-ldapThis has made no difference.I have also downlaoded the 3.4.1 source package from experimental and compiled with the same options ands still no difference to behaviour so suspect this is a red herring?So why is ptloader not retrieving any data?The ldap_bind credentials I have given it work fine with saslauthd or postfix or ldapsearch.Presumably then my filter and user attributes are bad? But I can't see why.What is the order of operations within cyrus?I assume that it authenticates first using saslauthd, and then uses the same username to check authorization in pts, but enabling PTS seems to prevent authentication. Although in syslog it is suggesting that SASL has authenticated, but with a bad userid?The windows DC uses a directory migrated from an older one on a small business server where microsoft recommended using an internal .local domain which has always been a bit of a headache for me. This means that our search base is a DC=local, as are the bind DN and userPrincipleName, but the mail and proxyAddresses email addresses are all .comIs the problem related to this? do I need to enable virtual domains and/or cross realm authentication for ptloader to get a response from the server?Is there any way to call ptloader outside of master to try to work out exactly what is being passed and what result it achieves?Also, how does ptdump work? I get no indication that it has done anything, is this simply because ptloader has never yet obtained any data for it to dump?As for checking the LDAP server logs, if anyone knows how I can do this on windows 2012 please advise! Looking at directory services in event viewer I see very few entries and nothing relating to communication from my mail server. I assume I need to enable a different log level but I can't find out how or what.All the examples I can find are based on openldap installations, is what I am trying to do possible, or are the Active Directory schema completely incompatible with ptloader?Jim Wallis
