hurd: Global signal disposition

This adds _hurd_sigstate_set_global_rcv used by libpthread to enable
POSIX-confirming behavior of signals on a per-thread basis.

This also provides a sigstate destructor _hurd_sigstate_delete, and a
global process signal state, which needs to be locked and check when
global disposition is enabled, thus the addition of _hurd_sigstate_lock
_hurd_sigstate_actions _hurd_sigstate_pending _hurd_sigstate_unlock helpers.

This also updates all the glibc code accordingly.

This also drops support for get_int(INIT_SIGMASK), which did not make sense
any more since we do not have a single signal thread any more.

During fork/spawn, this also reinitializes the child global sigstate's
lock. That cures an issue that would very rarely cause a deadlock in the
child in fork, tries to unlock ss' critical section lock at the end of
fork.  This will typically (always?) be observed in /bin/sh, which is not
surprising as that is the foremost caller of fork.

To reproduce an intermediate state, add an endless loop if
_hurd_global_sigstate is locked after __proc_dostop (cast through
volatile); that is, while still being in the fork's parent process.

When that triggers (use the libtool testsuite), the signal thread has
already locked ss (which is _hurd_global_sigstate), and is stuck at
hurdsig.c:685 in post_signal, trying to lock _hurd_siglock (which the
main thread already has locked and keeps locked until after
__task_create).  This is the case that ss->thread == MACH_PORT_NULL, that
is, a global signal.  In the main thread, between __proc_dostop and
__task_create is the __thread_abort call on the signal thread which would
abort any current kernel operation (but leave ss locked).  Later in fork,
in the parent, when _hurd_siglock is unlocked in fork, the parent's
signal thread can proceed and will unlock eventually the global sigstate.
In the client, _hurd_siglock will likewise be unlocked, but the global
sigstate never will be, as the client's signal thread has been configured
to restart execution from _hurd_msgport_receive.  Thus, when the child
tries to unlock ss' critical section lock at the end of fork, it will
first lock the global sigstate, will spin trying to lock it, which can
never be successful, and we get our deadlock.

Options seem to be:

  * Move the locking of _hurd_siglock earlier in post_signal -- but that
    may generally impact performance, if this locking isn't generally
    needed anyway?

    On the other hand, would it actually make sense to wait here until we
    are not any longer in a critical section (which is meant to disable
    signal delivery anyway (but not for preempted signals?))?

  * Clear the global sigstate in the fork's child with the rationale that
    we're anyway restarting the signal thread from a clean state.  This
    has now been implemented.

Why has this problem not been observed before Jérémie's patches?  (Or has
it?  Perhaps even more rarely?)  In _S_msg_sig_post, the signal is now
posted to a *global receiver thread*, whereas previously it was posted to
the *designated signal-receiving thread*.  The latter one was in a
critical section in fork, so didn't try to handle the signal until after
leaving the critical section?  (Not completely analyzed and verified.)

Another question is what the signal is that is being received
during/around the time __proc_dostop executes.

13 files changed, 99 insertions, 55 deletions

diff --git a/sysdeps/mach/hurd/Makefile b/sysdeps/mach/hurd/Makefile
index 6b48d8f843..2a6cd33b77 100644
--- a/sysdeps/mach/hurd/Makefile
+++ b/sysdeps/mach/hurd/Makefile
@@ -211,4 +211,6 @@ ifeq ($(subdir),nis)
 CFLAGS-ypclnt.c += -DUSE_BINDINGDIR=1
 endif
 
+LDLIBS-pthread.so += $(objdir)/hurd/libhurduser.so
+
 endif	# in-Makerules
diff --git a/sysdeps/mach/hurd/fork.c b/sysdeps/mach/hurd/fork.c
index d0bf7642b8..395cbe1e07 100644
--- a/sysdeps/mach/hurd/fork.c
+++ b/sysdeps/mach/hurd/fork.c
@@ -445,6 +445,7 @@ __fork (void)
 					      thread,
 					      MACH_MSG_TYPE_COPY_SEND)))
 	LOSE;
+      /* XXX consumed? (_hurd_sigthread is no more) */
       if (thread_refs > 1
 	  && (err = __mach_port_mod_refs (newtask, ss->thread,
 					  MACH_PORT_RIGHT_SEND,
@@ -610,10 +611,6 @@ __fork (void)
       for (i = 0; i < _hurd_nports; ++i)
 	__spin_unlock (&_hurd_ports[i].lock);
 
-      /* We are one of the (exactly) two threads in this new task, we
-	 will take the task-global signals.  */
-      _hurd_sigthread = ss->thread;
-
       /* Claim our sigstate structure and unchain the rest: the
 	 threads existed in the parent task but don't exist in this
 	 task (the child process).  Delay freeing them until later
@@ -633,6 +630,25 @@ __fork (void)
       ss->next = NULL;
       _hurd_sigstates = ss;
       __mutex_unlock (&_hurd_siglock);
+      /* Earlier on, the global sigstate may have been tainted and now needs to
+         be reinitialized.  Nobody is interested in its present state anymore:
+         we're not, the signal thread will be restarted, and there are no other
+         threads.
+
+         We can't simply allocate a fresh global sigstate here, as
+         _hurd_thread_sigstate will call malloc and that will deadlock trying
+         to determine the current thread's sigstate.  */
+#if 0
+      _hurd_thread_sigstate_init (_hurd_global_sigstate, MACH_PORT_NULL);
+#else
+      /* Only reinitialize the lock -- otherwise we might have to do additional
+         setup as done in hurdsig.c:_hurdsig_init.  */
+      __spin_lock_init (&_hurd_global_sigstate->lock);
+#endif
+
+      /* We are one of the (exactly) two threads in this new task, we
+	 will take the task-global signals.  */
+      _hurd_sigstate_set_global_rcv (ss);
 
       /* Fetch our new process IDs from the proc server.  No need to
 	 refetch our pgrp; it is always inherited from the parent (so
@@ -641,8 +657,10 @@ __fork (void)
       err = __USEPORT (PROC, __proc_getpids (port, &_hurd_pid, &_hurd_ppid,
 					     &_hurd_orphaned));
 
-      /* Forking clears the trace flag.  */
+      /* Forking clears the trace flag and pending masks.  */
       __sigemptyset (&_hurdsig_traced);
+      __sigemptyset (&_hurd_global_sigstate->pending);
+      __sigemptyset (&ss->pending);
 
       /* Release malloc locks.  */
       _hurd_malloc_fork_child ();
diff --git a/sysdeps/mach/hurd/htl/pt-sigstate-destroy.c b/sysdeps/mach/hurd/htl/pt-sigstate-destroy.c
index 02868a03c5..c829f86234 100644
--- a/sysdeps/mach/hurd/htl/pt-sigstate-destroy.c
+++ b/sysdeps/mach/hurd/htl/pt-sigstate-destroy.c
@@ -17,10 +17,12 @@
    <https://www.gnu.org/licenses/>.  */
 
 #include <pthread.h>
+#include <hurd/signal.h>
 
 #include <pt-internal.h>
 
 void
 __pthread_sigstate_destroy (struct __pthread *thread)
 {
+  _hurd_sigstate_delete (thread->kernel_thread);
 }
diff --git a/sysdeps/mach/hurd/htl/pt-sigstate-init.c b/sysdeps/mach/hurd/htl/pt-sigstate-init.c
index 97bf0c3a78..951f909855 100644
--- a/sysdeps/mach/hurd/htl/pt-sigstate-init.c
+++ b/sysdeps/mach/hurd/htl/pt-sigstate-init.c
@@ -35,7 +35,7 @@ __pthread_sigstate_init (struct __pthread *thread)
   if (do_init_global)
     {
       struct hurd_sigstate *ss = _hurd_thread_sigstate (thread->kernel_thread);
-      (void) ss;
+      _hurd_sigstate_set_global_rcv (ss);
     }
   else if (__pthread_num_threads >= 2)
     do_init_global = 1;
diff --git a/sysdeps/mach/hurd/htl/pt-sigstate.c b/sysdeps/mach/hurd/htl/pt-sigstate.c
index 845152ee48..bc16629672 100644
--- a/sysdeps/mach/hurd/htl/pt-sigstate.c
+++ b/sysdeps/mach/hurd/htl/pt-sigstate.c
@@ -20,6 +20,7 @@
 #include <assert.h>
 #include <signal.h>
 #include <hurd/signal.h>
+#include <hurd/msg.h>
 
 #include <pt-internal.h>
 
@@ -29,11 +30,12 @@ __pthread_sigstate (struct __pthread *thread, int how,
 {
   error_t err = 0;
   struct hurd_sigstate *ss;
+  sigset_t pending;
 
   ss = _hurd_thread_sigstate (thread->kernel_thread);
   assert (ss);
 
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
 
   if (oset != NULL)
     *oset = ss->blocked;
@@ -64,7 +66,13 @@ __pthread_sigstate (struct __pthread *thread, int how,
   if (!err && clear_pending)
     __sigemptyset (&ss->pending);
 
-  __spin_unlock (&ss->lock);
+  pending = _hurd_sigstate_pending (ss) & ~ss->blocked;
+  _hurd_sigstate_unlock (ss);
+
+  if (!err && pending)
+    /* Send a message to the signal thread so it
+       will wake up and check for pending signals.  */
+    __msg_sig_post (_hurd_msgport, 0, 0, __mach_task_self ());
 
   return err;
 }
diff --git a/sysdeps/mach/hurd/i386/sigreturn.c b/sysdeps/mach/hurd/i386/sigreturn.c
index a2bb6d0dfb..cbf32133bc 100644
--- a/sysdeps/mach/hurd/i386/sigreturn.c
+++ b/sysdeps/mach/hurd/i386/sigreturn.c
@@ -30,7 +30,7 @@ static void
 __sigreturn2 (int *usp)
 {
   struct hurd_sigstate *ss = _hurd_self_sigstate ();
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
 
   sp = usp;
 #define A(line) asm volatile (#line)
@@ -68,7 +68,7 @@ __sigreturn (struct sigcontext *scp)
     }
 
   ss = _hurd_self_sigstate ();
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
 
   /* Remove the link on the `active resources' chain added by
      _hurd_setup_sighandler.  Its purpose was to make sure
@@ -80,19 +80,19 @@ __sigreturn (struct sigcontext *scp)
   ss->intr_port = scp->sc_intr_port;
 
   /* Check for pending signals that were blocked by the old set.  */
-  if (ss->pending & ~ss->blocked)
+  if (_hurd_sigstate_pending (ss) & ~ss->blocked)
     {
       /* There are pending signals that just became unblocked.  Wake up the
 	 signal thread to deliver them.  But first, squirrel away SCP where
 	 the signal thread will notice it if it runs another handler, and
 	 arrange to have us called over again in the new reality.  */
       ss->context = scp;
-      __spin_unlock (&ss->lock);
+      _hurd_sigstate_unlock (ss);
       __msg_sig_post (_hurd_msgport, 0, 0, __mach_task_self ());
       /* If a pending signal was handled, sig_post never returned.
 	 If it did return, the pending signal didn't run a handler;
 	 proceed as usual.  */
-      __spin_lock (&ss->lock);
+      _hurd_sigstate_lock (ss);
       ss->context = NULL;
     }
 
diff --git a/sysdeps/mach/hurd/i386/trampoline.c b/sysdeps/mach/hurd/i386/trampoline.c
index 0c5d5a194c..68db596cca 100644
--- a/sysdeps/mach/hurd/i386/trampoline.c
+++ b/sysdeps/mach/hurd/i386/trampoline.c
@@ -37,6 +37,7 @@ _hurd_setup_sighandler (struct hurd_sigstate *ss, __sighandler_t handler,
   void firewall (void);
   extern const void _hurd_intr_rpc_msg_cx_sp;
   extern const void _hurd_intr_rpc_msg_sp_restored;
+  const struct sigaction *action;
   void *volatile sigsp;
   struct sigcontext *scp;
   struct
@@ -93,7 +94,11 @@ _hurd_setup_sighandler (struct hurd_sigstate *ss, __sighandler_t handler,
      the SP on sigreturn.  */
     state->basic.uesp = state->basic.ecx;
 
-  if ((ss->actions[signo].sa_flags & SA_ONSTACK)
+  /* XXX what if handler != action->handler (for instance, if a signal
+   * preemptor took over) ? */
+  action = & _hurd_sigstate_actions (ss) [signo];
+
+  if ((action->sa_flags & SA_ONSTACK)
       && !(ss->sigaltstack.ss_flags & (SS_DISABLE|SS_ONSTACK)))
     {
       sigsp = ss->sigaltstack.ss_sp + ss->sigaltstack.ss_size;
diff --git a/sysdeps/mach/hurd/sigaction.c b/sysdeps/mach/hurd/sigaction.c
index 39ce9a1388..9498995277 100644
--- a/sysdeps/mach/hurd/sigaction.c
+++ b/sysdeps/mach/hurd/sigaction.c
@@ -46,15 +46,15 @@ __sigaction (int sig, const struct sigaction *act, struct sigaction *oact)
   ss = _hurd_self_sigstate ();
 
   __spin_lock (&ss->critical_section_lock);
-  __spin_lock (&ss->lock);
-  old = ss->actions[sig];
+  _hurd_sigstate_lock (ss);
+  old = _hurd_sigstate_actions (ss) [sig];
   if (act != NULL)
-    ss->actions[sig] = a;
+    _hurd_sigstate_actions (ss) [sig] = a;
 
   if (act != NULL && sig == SIGCHLD
       && (a.sa_flags & SA_NOCLDSTOP) != (old.sa_flags & SA_NOCLDSTOP))
     {
-      __spin_unlock (&ss->lock);
+      _hurd_sigstate_unlock (ss);
 
       /* Inform the proc server whether or not it should send us SIGCHLD for
 	 stopped children.  We do this in a critical section so that no
@@ -62,8 +62,8 @@ __sigaction (int sig, const struct sigaction *act, struct sigaction *oact)
       __USEPORT (PROC,
 		 __proc_mod_stopchild (port, !(a.sa_flags & SA_NOCLDSTOP)));
 
-      __spin_lock (&ss->lock);
-      pending = ss->pending & ~ss->blocked;
+      _hurd_sigstate_lock (ss);
+      pending = _hurd_sigstate_pending (ss) & ~ss->blocked;
     }
   else if (act != NULL && (a.sa_handler == SIG_IGN || a.sa_handler == SIG_DFL))
     /* We are changing to an action that might be to ignore SIG signals.
@@ -72,11 +72,11 @@ __sigaction (int sig, const struct sigaction *act, struct sigaction *oact)
        back and then SIG is unblocked, the signal pending now should not
        arrive.  So wake up the signal thread to check the new state and do
        the right thing.  */
-    pending = ss->pending & __sigmask (sig);
+    pending = _hurd_sigstate_pending (ss) & __sigmask (sig);
   else
     pending = 0;
 
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
   __spin_unlock (&ss->critical_section_lock);
 
   if (pending)
diff --git a/sysdeps/mach/hurd/sigpending.c b/sysdeps/mach/hurd/sigpending.c
index 397d5ecd22..538a0237f4 100644
--- a/sysdeps/mach/hurd/sigpending.c
+++ b/sysdeps/mach/hurd/sigpending.c
@@ -36,9 +36,9 @@ sigpending (sigset_t *set)
     }
 
   ss = _hurd_self_sigstate ();
-  __spin_lock (&ss->lock);
-  pending = ss->pending;
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_lock (ss);
+  pending = _hurd_sigstate_pending (ss);
+  _hurd_sigstate_unlock (ss);
 
   *set = pending;
   return 0;
diff --git a/sysdeps/mach/hurd/sigprocmask.c b/sysdeps/mach/hurd/sigprocmask.c
index 240c4290df..73cd2dbc38 100644
--- a/sysdeps/mach/hurd/sigprocmask.c
+++ b/sysdeps/mach/hurd/sigprocmask.c
@@ -36,7 +36,7 @@ __sigprocmask (int how, const sigset_t *set, sigset_t *oset)
 
   ss = _hurd_self_sigstate ();
 
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
 
   old = ss->blocked;
 
@@ -57,7 +57,7 @@ __sigprocmask (int how, const sigset_t *set, sigset_t *oset)
 	  break;
 
 	default:
-	  __spin_unlock (&ss->lock);
+	  _hurd_sigstate_unlock (ss);
 	  errno = EINVAL;
 	  return -1;
 	}
@@ -65,9 +65,9 @@ __sigprocmask (int how, const sigset_t *set, sigset_t *oset)
       ss->blocked &= ~_SIG_CANT_MASK;
     }
 
-  pending = ss->pending & ~ss->blocked;
+  pending = _hurd_sigstate_pending (ss) & ~ss->blocked;
 
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
 
   if (oset != NULL)
     *oset = old;
diff --git a/sysdeps/mach/hurd/sigsuspend.c b/sysdeps/mach/hurd/sigsuspend.c
index 9551285a70..f4642b7702 100644
--- a/sysdeps/mach/hurd/sigsuspend.c
+++ b/sysdeps/mach/hurd/sigsuspend.c
@@ -40,7 +40,7 @@ __sigsuspend (const sigset_t *set)
 
   ss = _hurd_self_sigstate ();
 
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
 
   oldmask = ss->blocked;
   if (set != NULL)
@@ -48,11 +48,11 @@ __sigsuspend (const sigset_t *set)
     ss->blocked = newmask & ~_SIG_CANT_MASK;
 
   /* Notice if any pending signals just became unblocked.  */
-  pending = ss->pending & ~ss->blocked;
+  pending = _hurd_sigstate_pending (ss) & ~ss->blocked;
 
   /* Tell the signal thread to message us when a signal arrives.  */
   ss->suspended = wait;
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
 
   if (pending)
     /* Tell the signal thread to check for pending signals.  */
@@ -63,10 +63,11 @@ __sigsuspend (const sigset_t *set)
 	      MACH_MSG_TIMEOUT_NONE, MACH_PORT_NULL);
   __mach_port_destroy (__mach_task_self (), wait);
 
-  __spin_lock (&ss->lock);
-  ss->blocked = oldmask;	/* Restore the old mask.  */
-  pending = ss->pending & ~ss->blocked;	/* Again check for pending signals.  */
-  __spin_unlock (&ss->lock);
+  /* Restore the old mask and check for pending signals again.  */
+  _hurd_sigstate_lock (ss);
+  ss->blocked = oldmask;
+  pending = _hurd_sigstate_pending(ss) & ~ss->blocked;
+  _hurd_sigstate_unlock (ss);
 
   if (pending)
     /* Tell the signal thread to check for pending signals.  */
diff --git a/sysdeps/mach/hurd/sigwait.c b/sysdeps/mach/hurd/sigwait.c
index b70f7685b3..2594761245 100644
--- a/sysdeps/mach/hurd/sigwait.c
+++ b/sysdeps/mach/hurd/sigwait.c
@@ -27,7 +27,7 @@ int
 __sigwait (const sigset_t *set, int *sig)
 {
   struct hurd_sigstate *ss;
-  sigset_t mask, ready;
+  sigset_t mask, ready, blocked;
   int signo = 0;
   struct hurd_signal_preemptor preemptor;
   jmp_buf buf;
@@ -49,8 +49,8 @@ __sigwait (const sigset_t *set, int *sig)
       /* Make sure this is all kosher */
       assert (__sigismember (&mask, signo));
 
-      /* Make sure this signal is unblocked */
-      __sigdelset (&ss->blocked, signo);
+      /* Restore the blocking mask. */
+      ss->blocked = blocked;
 
       return pe->handler;
     }
@@ -71,10 +71,11 @@ __sigwait (const sigset_t *set, int *sig)
     __sigemptyset (&mask);
 
   ss = _hurd_self_sigstate ();
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
 
   /* See if one of these signals is currently pending.  */
-  __sigandset (&ready, &ss->pending, &mask);
+  sigset_t pending = _hurd_sigstate_pending (ss);
+  __sigandset (&ready, &pending, &mask);
   if (! __sigisemptyset (&ready))
     {
       for (signo = 1; signo < NSIG; signo++)
@@ -102,7 +103,11 @@ __sigwait (const sigset_t *set, int *sig)
       preemptor.next = ss->preemptors;
       ss->preemptors = &preemptor;
 
-      __spin_unlock (&ss->lock);
+      /* Unblock the expected signals */
+      blocked = ss->blocked;
+      ss->blocked &= ~mask;
+
+      _hurd_sigstate_unlock (ss);
 
       /* Wait. */
       __mach_msg (&msg, MACH_RCV_MSG, 0, sizeof (msg), wait,
@@ -113,7 +118,7 @@ __sigwait (const sigset_t *set, int *sig)
     {
       assert (signo);
 
-      __spin_lock (&ss->lock);
+      _hurd_sigstate_lock (ss);
 
       /* Delete our preemptor. */
       assert (ss->preemptors == &preemptor);
@@ -122,7 +127,7 @@ __sigwait (const sigset_t *set, int *sig)
 
 
 all_done:
-  spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
 
   __mach_port_destroy (__mach_task_self (), wait);
   *sig = signo;
diff --git a/sysdeps/mach/hurd/spawni.c b/sysdeps/mach/hurd/spawni.c
index e6b032b8a4..8a704512bd 100644
--- a/sysdeps/mach/hurd/spawni.c
+++ b/sysdeps/mach/hurd/spawni.c
@@ -336,26 +336,29 @@ __spawni (pid_t *pid, const char *file,
   assert (! __spin_lock_locked (&ss->critical_section_lock));
   __spin_lock (&ss->critical_section_lock);
 
-  __spin_lock (&ss->lock);
+  _hurd_sigstate_lock (ss);
   ints[INIT_SIGMASK] = ss->blocked;
-  ints[INIT_SIGPENDING] = ss->pending;
+  ints[INIT_SIGPENDING] = 0;
   ints[INIT_SIGIGN] = 0;
   /* Unless we were asked to reset all handlers to SIG_DFL,
      pass down the set of signals that were set to SIG_IGN.  */
-  if ((flags & POSIX_SPAWN_SETSIGDEF) == 0)
-    for (i = 1; i < NSIG; ++i)
-      if (ss->actions[i].sa_handler == SIG_IGN)
-	ints[INIT_SIGIGN] |= __sigmask (i);
-
-  /* We hold the sigstate lock until the exec has failed so that no signal
-     can arrive between when we pack the blocked and ignored signals, and
-     when the exec actually happens.  A signal handler could change what
+  {
+    struct sigaction *actions = _hurd_sigstate_actions (ss);
+    if ((flags & POSIX_SPAWN_SETSIGDEF) == 0)
+      for (i = 1; i < NSIG; ++i)
+	if (actions[i].sa_handler == SIG_IGN)
+	  ints[INIT_SIGIGN] |= __sigmask (i);
+  }
+
+  /* We hold the critical section lock until the exec has failed so that no
+     signal can arrive between when we pack the blocked and ignored signals,
+     and when the exec actually happens.  A signal handler could change what
      signals are blocked and ignored.  Either the change will be reflected
      in the exec, or the signal will never be delivered.  Setting the
      critical section flag avoids anything we call trying to acquire the
      sigstate lock.  */
 
-  __spin_unlock (&ss->lock);
+  _hurd_sigstate_unlock (ss);
 
   /* Set signal mask.  */
   if ((flags & POSIX_SPAWN_SETSIGMASK) != 0)