Debugging SSSD

SSSD consists of multiple processes, namely:

The monitor

This is the main sssd process.

Responders

There is one process for each responder with distinguishable name, for example sssd_nss, sssd_pam, etc.

Backends

There is one process for each configured SSSD domain. Each process is named sssd_be and the domain is determined by the --domain parameter.

Child processes

These are special processes that are usually invoked from backends to perform individual tasks such as Kerberos authentication or reading GPO data. They are located in /usr/libexec/sssd and suffixed with _child, for example krb5_child, ldap_child or gpo_child.

The following snippet shows a list of running SSSD process with three responders (nss, pam and pac) and three different domains (ldap.test, samba.test and ipa.test).

# ps aux | grep sssd
root  44  0.4  0.0  29248 11308 ?  Ss  10:51  0:00 /usr/sbin/sssd -i --logger=files
root  57  0.2  0.0  24728 11412 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_be --domain ldap.test --uid 0 --gid 0 --logger=files
root  58  0.3  0.1  88920 20748 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_be --domain samba.test --uid 0 --gid 0 --logger=files
root  59  0.3  0.1  73300 19872 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_be --domain ipa.test --uid 0 --gid 0 --logger=files
root  60  0.3  0.2  54408 37984 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
root  61  0.0  0.0  28416  9864 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files
root  62  0.1  0.0  74416 16220 ?  S   10:51  0:00 /usr/libexec/sssd/sssd_pac --uid 0 --gid 0 --logger=files

# ll /usr/libexec/sssd
-rwxr-xr-x. 1 root root  32584 Jan 25 12:03 gpo_child
-rwxr-xr-x. 1 root root     71 Mar 10 12:03 krb5_child
-rwxr-x---. 1 root root 119344 Jan 25 12:03 krb5_child.exe
-rwxr-x---. 1 root root  53136 Jan 25 12:03 ldap_child
-rwxr-xr-x. 1 root root  69416 Jan 25 12:03 p11_child
-rwxr-x---. 1 root root  28424 Jan 25 12:03 proxy_child
-rwxr-x---. 1 root root  32544 Jan 25 12:03 selinux_child
-rwxr-xr-x. 1 root root     72 Jan 25 10:44 sss_analyze
-rwxr-xr-x. 1 root root  15960 Jan 25 12:03 sss_signal
-rwxr-xr-x. 1 root root 208088 Jan 25 12:03 sssd_autofs
-rwxr-xr-x. 1 root root 245728 Jan 25 12:03 sssd_be
-rwxr-xr-x. 1 root root  15992 Jan 25 12:03 sssd_check_socket_activated_responders
-rwxr-xr-x. 1 root root 303048 Jan 25 12:03 sssd_ifp
-rwxr-xr-x. 1 root root 212160 Jan 25 12:03 sssd_kcm
-rwxr-xr-x. 1 root root 287696 Jan 25 12:03 sssd_nss
-rwxr-xr-x. 1 root root 203848 Jan 25 12:03 sssd_pac
-rwxr-xr-x. 1 root root 278368 Jan 25 12:03 sssd_pam
-rwxr-xr-x. 1 root root 216096 Jan 25 12:03 sssd_ssh
-rwxr-xr-x. 1 root root 216336 Jan 25 12:03 sssd_sudo

Disabling the watchdog

There is a special deadlock guard called the watchdog that runs inside each SSSD process. Its purpose is to make sure that the process is still responsive and kill it if it is not so it can be restarted again by the monitor.

The watchdog uses timers and signals mechanisms therefore it interrupts the process every now and then which makes it difficult to debug the process in the debugger, not to mention that the process will be killed if you keep it hanging on a breakpoint for a longer time.

You could tell the debugger to don’t react on the SIG34 signal so it is passed directly to the process.

$ sudo gdb program $(pidof sssd_nss) -ex 'handle SIG34 nostop noprint'

This however will not solve the problem when the process is waiting on a breakpoint and therefore it eventually gets terminated by the watchdog. To solve this, it is recommended to set a long watchdog interval using the timeout option in sssd.conf, for example to 30000 seconds.

[sssd]
config_file_version = 2
services = nss, pam
domains = ldap.test, samba.test, ipa.test
user = root

[nss]
timeout=30000
...

[pam]
timeout=30000
...

[domain/ldap.test]
timeout=30000
...

[domain/samba.test]
timeout=30000
...

[domain/ipa.test]
timeout=30000
...

Attaching debugger to a running process

There is only one process for each responder that can be distinguished by name, therefore it is simple to attach a debugger to the running process. For example:

$ sudo gdb program `pgrep sssd_nss`

There can be multiple backend sssd_be processes and we need to use the --domain parameter to distinguish between them. Therefore we want to use the -f/--full option of the pgrep command to make it match the whole command line and not only the process name. The following snippet shows how to attach the debugger to ldap.test domain:

$ sudo gdb program `pgrep -f "sssd_be.+ldap.test"`

Debugging forked process from its start

Usually, you will suffice with attaching debugger to a running process. But sometimes, you want to debug the process from the very beginning. This applies especially to the different child processes that are forked from sssd_be to perform various stuff like Kerberos authentication.

We can use gdbserver for that which provides a remote access for gdb. It can either listen on given device or a TCP connection which we will use in our examples. To install it, run:

$ sudo dnf install -y gdb-gdbserver

$ sudo yum install -y gdb-gdbserver

$ sudo apt-get install -y gdbserver

Now, we need to create a wrapper that would execute the process in the gdbserver that will listen on a specific port.

$ process=/usr/libexec/sssd/krb5_child
$ sudo mv "$process" "$process.exe"
$ sudo cat << 'EOF' > $process
#!/bin/bash
exec gdbserver :1234 /usr/libexec/sssd/krb5_child.exe "$@"
EOF
$ sudo chmod +x "$process"

Now we need to increase timeout of the child process in order to avoid its termination during the debugging session. We’ll use the krb5_auth_timeout for that.

[domain/ipa.test]
timeout=30000
krb5_auth_timeout=30000

Now you can start SSSD and let it get to the process that you want to debug, it is the krb5_child in our example so we can try authenticate as some user, i.e. su admin@ipa.test. Then start gdb and attach it to the server:

$ sudo gdb /usr/libexec/sssd/krb5_child.exe -ex "target remote :1234" -ex "b main" -ex "c"