SSSD provides two major features - obtaining information about users and authenticating users. Each of these hook into different system APIs and should be viewed separately. However, a successful authentication can only be performed when the information about a user can be retrieved, so if authentication doesn’t work in your case, please make sure you can at least obtain info from about the user with
getent passwd $user and
Each SSSD process is represented by a section in the
sssd.conf config file. To enable debugging persistently across SSSD service restarts, put the directive
debug_level=N, where N typically stands for a number between 1 and 10 into the particular section. Debug levels up to 3 should log mostly failures and anything above level 8 provides a large number of log messages. Level 6 might be a good starting point for debugging problems. You can also use the
sss_debuglevel(8) tool to enable debugging on the fly without having to restart the daemon.
On Fedora/RHEL, debug logs are stored under
/var/log/sssd. There is one log file per SSSD process. The services (also called responders) log into a log file called
sssd_$service, for example NSS responder logs to
/var/log/sssd/sssd_nss.log. Domain sections log to files called
sssd_$domainname.log. The short-lived helper processes also log to their own log files, such as
Enabling debug_level in the [sssd] section only enables debugging of the sssd process itself, not all the worker processes!
Before diving into the SSSD logs and config files it is very beneficial to know what the SSSD Architecture looks like. Please note the examples of the
DEBUG messages are subject to change in future SSSD versions.
To avoid SSSD caching, it is often useful to reproduce the bugs with an empty cache or at least invalid cache. However, keep in mind that also the cached credentials are stored in the cache! Do not remove the cache files if your system is offline and it relies on SSSD authentication!
Before sending the logs and/or config files to a publicly-accessible space, such as mailing lists or bug trackers, check the files for any sensitive information.
Please only send log files relevant to the occurrence of the issue. Issues in log files that are mega- or gigabytes large are more likely to be skipped
Unless the problem you’re trying to diagnose is related to enumeration, always reproduce the issue with
enumerate=false. Triaging logs with enumeration enabled obfuscates the normal data flow..
Follow the usual name-service request flow:
Is sssd running at all? On most recent systems, running the below command would display the service status. Alternatively, check for the sssd processes with
ps -ef | grep sssdor similar.
# systemctl status sssd
sssmodule present in
/etc/nsswitch.conffor all databases?
If there is a separate initgroups database configured, make sure it either contains the
sssmodule as well or comment the
Does the request reach the SSSD responder processes? Enable debugging by putting
debug_level=6(or higher) into the
[nss]section. Restart SSSD and check the nss log for incoming requests with the matching timestamp to your
idcommand. Here is how an incoming request looks like with SSSD-1.15:
[sssd[nss]] [get_client_cred] (0x4000): Client creds: euid egid pid. [sssd[nss]] [setup_client_idle_timer] (0x4000): Idle timer re-set for client [0x13c9960] [sssd[nss]] [accept_fd_handler] (0x0400): Client connected! [sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version . [sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version . [sssd[nss]] [nss_getby_name] (0x0400): Input name: admin
If the command is reaching the NSS responder, does it get forwarded to the Data Provider? Does the Data Provider request end successfully? An unsuccessful request would look like this:
[sssd[nss]] [cache_req_search_dp] (0x0400): CR #3: Looking up [firstname.lastname@example.org] in data provider [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41e51c:1:email@example.com@ipa.test] [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.test][0x1][BE_REQ_USER][firstname.lastname@example.org:-] [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x41e51c:1:email@example.com@ipa.test] [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x41e51c:1:firstname.lastname@example.org@win.trust.test] [sssd[nss]] [sss_dp_get_reply] (0x0010): The Data Provider returned an error [org.freedesktop.sssd.Error.DataProvider.Offline] [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #3: Data Provider Error: 3, 5, Failed to get reply from Data Provider [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #3: Due to an error we will return cached data
In contrast, a request that ran into completion would look like this:
[sssd[nss]] [cache_req_search_dp] (0x0400): CR #3: Looking up [email@example.com] in data provider [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x41e51c:1:firstname.lastname@example.org@ipa.test] [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.test][0x1][BE_REQ_USER][email@example.com:-] [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success [sssd[nss]] [cache_req_search_cache] (0x0400): CR #3: Looking up [firstname.lastname@example.org] in cache
If the Data Provider request had finished completely, but you’re still not seeing any data, then chances are the search didn’t match any object. Either, way, the next step is to look into the logs from the
[domain] section. Put
debug_level=6 or higher into the appropriate [domain] section, restart SSSD, re-run the lookup and continue debugging in the next section.
The back end performs several different operations, so it might be difficult to see where the problem is at first. At the highest level, the back end performs these steps, in this order
The request is received from the responder
The back end resolves the server to connect to. This step might involve locating the client site or resolving a SRV query
The back end establishes connection to the server. In case the connection is authenticated, then a proper keytab or a certificate might be required
Once connection is established, the back end runs the search. You should see the LDAP filter, search base and requested attributes.
After the search finishes, the entries that matched are stored to the cache
When the request ends (correctly or not), the status code is returned to the responder
Make sure the back end is in “neutral” or “online” state when you run the search.
With some responder/provider combinations, SSSD might run a search immediately after startup, which, in case of misconfiguration, might mark the back end offline even before the first request by the user arrives.
You can forcibly set SSSD into offline or online state using the
SIGUSR2signals, see the
sssd(8)man page for details.
Can the remote server be resolved? Check if the DNS servers in
/etc/resolv.confare correct. With AD or IPA back ends, you generally want them to point to the AD or IPA server directly. * Use the
digutility to test SRV queries, for instance:
dig -t SRV _ldap._tcp.ad.example.com @127.0.0.1
Can the connection be established with the same security properties SSSD uses?
Many back ends require the connection to be authenticated. In case of AD and IPA, the connection is authenticated using the system keytab, the LDAP back end often uses certificates.
ldapsearch -ZZis useful to test problems with certificates, since SSSD uses openldap libraries under the hood.
For debugging GSSAPI authentication,
kinitis useful, often together with
KRB5_TRACE. Take care to select the correct principal, especially with the AD back end. If you select the highest
debug_level = 10, then
ldap_child.logwould contain the Kerberos tracing information as well.
Are the LDAP search properties correct?
- Check if all the attributes required by the search are present on the server. This is especially important with the AD provider where the entries might not contain the POSIX attributes at all or might not have the POSIX attributes replicated to Global Catalog, in case SSSD is connecting to the GC.
As of SSSD-1.15, try looking for
Is the search base correct, especially with trusted subdomains? Incorrect search base with an AD subdomain would yield a referral.
Try running the same search with the ldapsearch utility. Don’t forget to use the same authentication method as SSSD uses! For
ipathis means adding
-Y GSSAPIto the
In order for authentication to be successful, the user information must be accurately provided first. Before debugging authentication, please make sure the user information is resolvable with
getent passwd $user or
id $user. Failing to retrieve the user info would also manifest in the secure logs or the journal with message such as:
pam_sss(sshd:account): Access denied for user admin: 10 (User not known to the underlying authentication module)
Authentication happens from PAM’s
auth stack and corresponds to SSSD’s
auth_provider. Access control takes place in PAM
account phase and is linked with SSSD’s
access_provider. And lastly, password changes go through the
password stack on the PAM side to SSSD’s
If the user info can be retrieved, but authentication fails, the first place to look into is
/var/log/secure or the system journal. Look for messages from
pam_sss. Please note that not all authentication requests come through SSSD. Notably, SSH key authentication and GSSAPI SSH authentication happen directly in SSHD and SSSD is only contacted for the
The PAM authentication flow follows this pattern:
The PAM-aware application starts the PAM conversation. Depending on the PAM stack configuration, the
pam_sssmodule would be contacted. To debug the authentication process, first check in the secure log or journal if
pam_sssis called at all. If you don’t see
pam_sssmentioned, chances are your PAM stack is misconfigured. If you see
pam_sssbeing contacted, enable debugging in pam responder logs
SSSD’s PAM responder receives the authentication request and in most cases forwards it to the back end. Please note that unlike identity requests, the authentication/access control is typically not cached and always contacts the server. This might manifest as a slowdown in some cases, but it’s quite important, because the supplementary groups in GNU/Linux are only set during login time.
The PAM responder logs should show the request being received from the pam stack and then forwarded to the back end.
If you see the authentication request getting to the PAM responder, but receiving an error from the back end, check the back end logs. An example error output might look like:
[sssd[pam]] [sss_dp_issue_request] (0x0400): Issuing request for [0x411d44:3:email@example.com] [sssd[pam]] [sss_dp_get_account_msg] (0x0400): Creating request for [ipa.example.com][name=admin] [sssd[pam]] [sss_dp_internal_get_send] (0x0400): Entering request [0x411d44:3:firstname.lastname@example.org] [sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Offline [sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to get information from Data Provider Error: 1, 11, Offline
The back end processes the request. This might include the equivalent of
kinitdone in the
krb5_childprocess, an LDAP bind or consulting an access control list. After the back end request finishes, the result is sent back to the PAM responder.
For Kerberos-based (that includes the IPA and AD providers)
auth_provider, look into the
krb5_child.logfile as well. Setting
debug_levelto 10 would also enable low-level Kerberos tracing information in that logfile. You can also simulate the authentication with
If the back end’s
auth_provideris LDAP-based, you can simulate the authentication by performing a base-scoped bind as the user who is logging in:
ldapsearch -x -H ldap://master.ipa.example.com -b uid=admin,cn=users,cn=accounts,dc=ipa,dc=example,dc=com -s base -W
The PAM responder receives the result and forwards it back to the
pam_sssmodule. The error or status message is displayed in
Logins take too long or the time to execute
id $usernametakes too long.
First, make sure to understand what does id username do. Do you really care about its performance? Chances are you’re more interested in
Check out the
ignore_group_membersoptions in the
Some users improved their SSSD performance a lot by mounting the cache into
getent groupdoesn’t display any users or groups.
Enumeration is disabled by design. See the FAQ page for explanation
Changes on the server are not reflected on the client for quite some time
The SSSD caches identity information for some time. You can force cache refresh on next lookup using the
Please note that during login, updated information is always re-read from the server
After enrolling the same machine to a domain with different users (perhaps a test VM was enrolled to a newly provisioned server), no users can be resolved or log in
Probably the new server has different ID values even if the users are named the same (like admin in an IPA domain). Currently UID changes are not supported, caches must be removed.
How do I enable LDAP authentication over an unsecure connection?
Not possible, sorry. SSSD requires the use of either TLS or LDAPS for LDAP authentication. Perimeter security is just not enough.
There are no messages from
Your PAM stack is likely misconfigured. Use the
authselecttool if available together with
Alternatively, check that the authentication you are using is PAM-aware, because some authentication methods, like SSH public keys are handled directly in the SSHD and do not use PAM at all.
suto an SSSD user from root, but not from a regular user, SSH doesn’t work either
If you su to another user from root, you typically bypass SSSD authentication completely by using the
pam_rootok.somodule. Your SSSD setup is likely broken, please log in as an ordinary user and continue debugging in this section
System Error (4)in the authentication logs
System Error is an “Unhandled Exception” during authentication. It can either be an SSSD bug or a fatal error during authentication. Either way, please bring up your issue on the sssd-users mailing list
Access denied for user $user: 6 (Permission denied)
Authentication went fine, but the user was denied access to the client machine. You can temporarily disable access control with setting
access_provider=permittemporarily. Don’t forget to reset the access provider to a stricter setting after finding out the root cause!
If disabling access control doesn’t help, the account might be locked on the server side. Check the SSSD domain logs to find out more.
I can’t get my LDAP-based access control filter right for group access control using the memberOf attribute
The LDAP-based access control is really tricky to get right and doesn’t typically handle nested groups well. Use the simple access provider