fix false ownership of mutexes due to tid reuse, using robust list

per the resolution of Austin Group issue 755, the POSIX requirement
that ownership be enforced for recursive and error-checking mutexes
does not allow a random new thread to acquire ownership of an orphaned
mutex just because it happened to be assigned the same tid as the
original owner that exited with the mutex locked.

one possible fix for this issue would be to disallow the kernel thread
to terminate when it exited with mutexes held, permanently reserving
the tid against reuse. however, this does not solve the problem for
process-shared mutexes where lifetime cannot be controlled, so it was
not used.

the alternate approach I've taken is to reuse the robust mutex system
for non-robust recursive and error-checking mutexes. when a thread
exits, the kernel (or the new userspace robust-list code added in
commit b092f1c5fa9c048e12d002c7b972df5ecbe96d1d) will set the
owner-died bit for these orphaned mutexes, but since the mutex-type is
not robust, pthread_mutex_trylock will not allow a new owner to
acquire them. instead, they remain in a state of being permanently
locked, as desired.

1 files changed, 16 insertions, 12 deletions

diff --git a/src/thread/pthread_mutex_trylock.c b/src/thread/pthread_mutex_trylock.c
index 850fcb90..f871e9e0 100644
--- a/src/thread/pthread_mutex_trylock.c
+++ b/src/thread/pthread_mutex_trylock.c
@@ -7,12 +7,9 @@ int __pthread_mutex_trylock_owner(pthread_mutex_t *m)
 	pthread_t self = __pthread_self();
 	int tid = self->tid;
 
-	if (type >= 4) {
-		if (!self->robust_list.off)
-			__syscall(SYS_set_robust_list,
-				&self->robust_list, 3*sizeof(long));
+	if (!self->robust_list.off) {
+		__syscall(SYS_set_robust_list, &self->robust_list, 3*sizeof(long));
 		self->robust_list.off = (char*)&m->_m_lock-(char *)&m->_m_next;
-		self->robust_list.pending = &m->_m_next;
 	}
 
 	old = m->_m_lock;
@@ -23,21 +20,28 @@ int __pthread_mutex_trylock_owner(pthread_mutex_t *m)
 		return 0;
 	}
 
-	if ((own && !(own & 0x40000000)) || a_cas(&m->_m_lock, old, tid)!=old)
-		return EBUSY;
-
-	if (type < 4) return 0;
+	self->robust_list.pending = &m->_m_next;
 
-	if (type >= 8) {
-		m->_m_lock = 0;
-		return ENOTRECOVERABLE;
+	if ((own && (!(own & 0x40000000) || !(type & 4)))
+	    || a_cas(&m->_m_lock, old, tid) != old) {
+		self->robust_list.pending = 0;
+		return EBUSY;
 	}
+
 	m->_m_next = self->robust_list.head;
 	m->_m_prev = &self->robust_list.head;
 	if (self->robust_list.head)
 		self->robust_list.head[-1] = &m->_m_next;
 	self->robust_list.head = &m->_m_next;
 	self->robust_list.pending = 0;
+
+	if (type < 4) return 0;
+
+	if (type >= 8) {
+		m->_m_lock = 0;
+		return ENOTRECOVERABLE;
+	}
+
 	if (own) {
 		m->_m_count = 0;
 		m->_m_type += 8;