about summary refs log tree commit diff
path: root/src/thread
Commit message (Collapse)AuthorAgeFilesLines
* fix possible fd leak via missing O_CLOEXEC in pthread_setname_npÉrico Rolim2021-01-301-1/+1
| | | | | the omission of the flag here seems to have been an oversight when the function was added in 8fb28b0b3e7a5e958fb844722a4b2ef9bc244af1
* fix omission of non-stub pthread_mutexattr_getprotocolRich Felker2020-12-071-1/+1
| | | | | | | this change should have been made when priority inheritance mutex support was added. if priority protection is also added at some point the implementation will need to change and will probably no longer be a simple bit shuffling.
* fix failure to preserve r6 in s390x asm; per ABI it is call-savedRich Felker2020-12-042-0/+8
| | | | | | | | | | | | | | | | | | both __clone and __syscall_cp_asm failed to restore the original value of r6 after using it as a syscall argument register. the extent of breakage is not known, and in some cases may be mitigated by the only callers being internal to libc; if they used r6 but no longer needed its value after the call, they may not have noticed the problem. however at least posix_spawn (which uses __clone) was observed returning to the application with the wrong value in r6, leading to crash. since the call frame ABI already provides a place to spill registers, fixing this is just a matter of using it. in __clone, we also spuriously restore r6 in the child, since the parent branch directly returns to the caller. this takes the value from an uninitialized slot of the child's stack, but is harmless since there is no caller to return to in the child.
* fix regression in pthread_exitRich Felker2020-11-201-0/+1
| | | | | | | | | | | | | | | commit d26e0774a59bb7245b205bc8e7d8b35cc2037095 moved the detach state transition at exit before the thread list lock was taken. this inadvertently allowed pthread_join to race to take the thread list lock first, and proceed with unmapping of the exiting thread's memory. we could fix this by just revering the offending commit and instead performing __vm_wait unconditionally before taking the thread list lock, but that may be costly. instead, bring back the old DT_EXITING vs DT_EXITED state distinction that was removed in commit 8f11e6127fe93093f81a52b15bb1537edc3fc8af, and don't transition to DT_EXITED (a value of 0, which is what pthread_join waits for) until after the lock has been taken.
* protect destruction of process-shared mutexes against robust list racesRich Felker2020-11-191-1/+5
| | | | | | | | | | | | | | | after a non-normal-type process-shared mutex is unlocked, it's immediately available to another thread to lock, unlock, and destroy, but the first unlocking thread may still have a pointer to it in its robust_list pending slot. this means, on async process termination, the kernel may attempt to access and modify the memory that used to contain the mutex -- memory that may have been reused for some other purpose after the mutex was destroyed. setting up for this kind of race to occur is difficult to begin with, requiring dynamic use of shared memory maps, and actually hitting the race is very difficult even with a suitable setup. so this is mostly a theoretical fix, but in any case the cost is very low.
* pthread_exit: don't __vm_wait under thread list lockRich Felker2020-11-191-9/+15
| | | | | | | | | | | | | | | the __vm_wait operation can delay forward progress arbitrarily long if a thread holding the lock is interrupted by a signal. in a worst case this can deadlock. any critical section holding the thread list lock must respect lock ordering contracts and must not take any lock which is not AS-safe. to fix, move the determination of thread joinable/detached state to take place before the killlock and thread list lock are taken. this requires reverting the atomic state transition if we determine that the exiting thread is the last thread and must call exit, but that's easy to do since it's a single-threaded context with application signals blocked.
* lift child restrictions after multi-threaded forkRich Felker2020-11-112-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | as the outcome of Austin Group tracker issue #62, future editions of POSIX have dropped the requirement that fork be AS-safe. this allows but does not require implementations to synchronize fork with internal locks and give forked children of multithreaded parents a partly or fully unrestricted execution environment where they can continue to use the standard library (per POSIX, they can only portably use AS-safe functions). up until recently, taking this allowance did not seem desirable. however, commit 8ed2bd8bfcb4ea6448afb55a941f4b5b2b0398c0 exposed the extent to which applications and libraries are depending on the ability to use malloc and other non-AS-safe interfaces in MT-forked children, by converting latent very-low-probability catastrophic state corruption into predictable deadlock. dealing with the fallout has been a huge burden for users/distros. while it looks like most of the non-portable usage in applications could be fixed given sufficient effort, at least some of it seems to occur in language runtimes which are exposing the ability to run unrestricted code in the child as part of the contract with the programmer. any attempt at fixing such contracts is not just a technical problem but a social one, and is probably not tractable. this patch extends the fork function to take locks for all libc singletons in the parent, and release or reset those locks in the child, so that when the underlying fork operation takes place, the state protected by these locks is consistent and ready for the child to use. locking is skipped in the case where the parent is single-threaded so as not to interfere with legacy AS-safety property of fork in single-threaded programs. lock order is mostly arbitrary, but the malloc locks (including bump allocator in case it's used) must be taken after the locks on any subsystems that might use malloc, and non-AS-safe locks cannot be taken while the thread list lock is held, imposing a requirement that it be taken last.
* convert malloc use under libc-internal locks to use internal allocatorRich Felker2020-11-111-0/+5
| | | | | | | | | | | | | this change lifts undocumented restrictions on calls by replacement mallocs to libc functions that might take these locks, and sets the stage for lifting restrictions on the child execution environment after multithreaded fork. care is taken to #define macros to replace all four functions (malloc, calloc, realloc, free) even if not all of them will be used, using an undefined symbol name for the ones intended not to be used so that any inadvertent future use will be caught at compile time rather than directed to the wrong implementation.
* fix erroneous pthread_cond_wait mutex waiter count logic due to typoRich Felker2020-10-301-1/+1
| | | | introduced in commit 27b2fc9d6db956359727a66c262f1e69995660aa.
* fix missing-wake regression in pthread_cond_waitRich Felker2020-10-301-0/+5
| | | | | | | | | | | | | | | the reasoning in commit 2d0bbe6c788938d1332609c014eeebc1dff966ac was not entirely correct. while it's true that setting the waiters flag ensures that the next unlock will perform a wake, it's possible that the wake is consumed by a mutex waiter that has no relationship with the condvar wait queue being processed, which then takes the mutex. when that thread subsequently unlocks, it sees no waiters, and leaves the rest of the condvar queue stuck. bring back the waiter count adjustment, but skip it for PI mutexes, for which a successful lock-after-waiting always sets the waiters bit. if future changes are made to bring this same waiters-bit contract to all lock types, this can be reverted.
* fix sem_close unmapping of still-referenced semaphoreRich Felker2020-10-281-3/+5
| | | | | | | | | | | | | sem_open is required to return the same sem_t pointer for all references to the same named semaphore when it's opened more than once in the same process. thus we keep a table of all the mapped semaphores and their reference counts. the code path for sem_close checked the reference count, but then proceeded to unmap the semaphore regardless of whether the count had reached zero. add an immediate unlock-and-return for the nonzero refcnt case so the property of performing the munmap syscall after releasing the lock can be preserved.
* fix pthread_cond_wait paired with with priority-inheritance mutexRich Felker2020-10-261-6/+5
| | | | | | | | | | | | | | | | | | | | | pthread_cond_wait arranged for requeued waiters to wake when the mutex is unlocked by temporarily adjusting the mutex's waiter count. commit 54ca677983d47529bab8752315ac1a2b49888870 broke this when introducing PI mutexes by repurposing the waiter count field of the mutex structure. since then, for PI mutexes, the waiter count adjustment was misinterpreted by the mutex locking code as indicating that the mutex is non a non-recoverable state. it would be possible to special-case PI mutexes here, but instead just drop all adjustment of the waiters count, and instead use the lock word waiters bit for all mutex types. since the mutex is either held by the caller or in unrecoverable state at the time the bit is set, it will necessarily still be set at the time of any subsequent valid unlock operation, and this will produce the desired effect of waking the next waiter. if waiter counts are entirely dropped at some point in the future this code should still work without modification.
* drop use of pthread_once in mutexattr kernel support testsRich Felker2020-10-142-21/+18
| | | | | | | | this makes the code slightly smaller and eliminates these functions from relevance to possible future changes to multithreaded fork. the barrier of a_store isn't technically needed here, but a_store is used anyway for internal consistency of the memory model.
* avoid set*id/setrlimit misbehavior and hang in vforked/cloned childRich Felker2020-09-171-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | taking the deprecated/dropped vfork spec strictly, doing pretty much anything but execve in the child is wrong and undefined. however, these are commonly needed operations to setup the child state before exec, and historical implementations tolerated them. for single-threaded parents, these operations already worked as expected in the vforked child. however, due to the need for __synccall to synchronize id/resource limit changes among all threads, calling these functions in the vforked child of a multithreaded parent caused a misdirected broadcast signaling of all threads in the parent. these signals could kill the parent entirely if the synccall signal handler had never been installed in the parent, or could be ignored if it had, or could signal/kill one or more utterly wrong processes if the parent already terminated (due to vfork semantics, only possible via fatal signal) and the parent tids were recycled. in any case, the expected number of semaphore posts would never happen, so the child would permanently hang (with all signals blocked) waiting for them. to mitigate this, and also make the normal usage case work as intended, treat the condition where the caller's actual tid does not match the tid in its thread structure as single-threaded, and bypass the entire synccall broadcast operation.
* fix i386 __set_thread_area fallbackRich Felker2020-08-301-0/+1
| | | | | | | | this code is only needed for pre-2.6 kernels, which are not actually supported anyway, and was never tested. the fallback path using SYS_modify_ldt failed to clear the upper bits of %eax (all ones due to SYS_set_thread_area's return value being an error) before modifying %al to attempt a new syscall.
* remove redundant pthread struct members repeated for layout purposesRich Felker2020-08-271-1/+1
| | | | | | | dtv_copy, canary2, and canary_at_end existed solely to match multiple ABI and asm-accessed layouts simultaneously. now that pthread_arch.h can be included before struct __pthread is defined, the struct layout can depend on macros defined by pthread_arch.h.
* fix async-cancel-safety of pthread_cancelRich Felker2020-07-061-1/+4
| | | | | | | | | | | | | the previous commit addressing async-signal-safety issues around pthread_kill did not fully fix pthread_cancel, which is also required (albeit rather irrationally) to be async-cancel-safe. without blocking implementation-internal signals, it's possible that, when async cancellation is enabled, a cancel signal sent by another thread interrupts pthread_kill while the killlock for a targeted thread is held. as a result, the calling thread will terminate due to cancellation without ever unlocking the targeted thread's killlock, and thus the targeted thread will be unable to exit.
* make thread killlock async-signal-safe for pthread_killRich Felker2020-07-065-5/+18
| | | | | | | | | | pthread_kill is required to be AS-safe. that requirement can't be met if the target thread's killlock can be taken in contexts where application-installed signal handlers can run. block signals around use of this lock in all pthread_* functions which target a tid, and reorder blocking/unblocking of signals in pthread_exit so that they're blocked whenever the killlock is held.
* restore lock-skipping for processes that return to single-threaded stateRich Felker2020-05-222-5/+7
| | | | | | | | | the design used here relies on the barrier provided by the first lock operation after the process returns to single-threaded state to synchronize with actions by the last thread that exited. by storing the intent to change modes in the same object used to detect whether locking is needed, it's possible to avoid an extra (possibly costly) memory load after the lock is taken.
* don't use libc.threads_minus_1 as relaxed atomic for skipping locksRich Felker2020-05-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | after all but the last thread exits, the next thread to observe libc.threads_minus_1==0 and conclude that it can skip locking fails to synchronize with any changes to memory that were made by the last-exiting thread. this can produce data races. on some archs, at least x86, memory synchronization is unlikely to be a problem; however, with the inline locks in malloc, skipping the lock also eliminated the compiler barrier, and caused code that needed to re-check chunk in-use bits after obtaining the lock to reuse a stale value, possibly from before the process became single-threaded. this in turn produced corruption of the heap state. some uses of libc.threads_minus_1 remain, especially for allocation of new TLS in the dynamic linker; otherwise, it could be removed entirely. it's made non-volatile to reflect that the remaining accesses are only made under lock on the thread list. instead of libc.threads_minus_1, libc.threaded is now used for skipping locks. the difference is that libc.threaded is permanently true once an additional thread has been created. this will produce some performance regression in processes that are mostly single-threaded but occasionally creating threads. in the future it may be possible to bring back the full lock-skipping, but more care needs to be taken to produce a safe design.
* reorder thread list unlink in pthread_exit after all locksRich Felker2020-05-221-8/+11
| | | | | | | | | since the backend for LOCK() skips locking if single-threaded, it's unsafe to make the process appear single-threaded before the last use of lock. this fixes potential unsynchronized access to a linked list via __dl_thread_cleanup.
* harden thread start with failed scheduling against broken __cloneRich Felker2019-09-131-1/+1
| | | | | | | | | | | | | | | | | | commit 8a544ee3a2a75af278145b09531177cab4939b41 introduced a dependency of the failure path for explicit scheduling at thread creation on __clone's handling of the start function returning, which should result in SYS_exit. as noted in commit 05870abeaac0588fb9115cfd11f96880a0af2108, the arm version of __clone was broken in this case. in the past, the mips version was also broken; it was fixed in commit 8b2b61e0001281be0dcd3dedc899bf187172fecb. since this code path is pretty much entirely untested (previously only reachable in applications that call the public clone() and return from the start function) and consists of fragile per-arch asm, don't assume it works, at least not until it's been thoroughly tested. instead make the SYS_exit syscall from the start function's failure path.
* fix arm __a_barrier_oldkuser when built as thumbRich Felker2019-09-111-2/+2
| | | | | | | | | | | as noted in commit 05870abeaac0588fb9115cfd11f96880a0af2108, mov lr,pc is not a valid method for saving the return address in code that might be built as thumb. this one is unlikely to matter, since any ISA level that has thumb2 should also have native implementations of atomics that don't involve kuser_helper, and the affected code is only used on very old kernels to begin with.
* fix code path where child function returns in arm __clone built as thumbRich Felker2019-09-111-7/+3
| | | | | | | | | | | | | | mov lr,pc is not a valid way to save the return address in thumb mode since it omits the thumb bit. use a chain of bl and bx to emulate blx. this could be avoided by converting to a .S file with preprocessor conditions to use blx if available, but the time cost here is dominated by the syscall anyway. while making this change, also remove the remnants of support for pre-bx ISA levels. commit 9f290a49bf9ee247d540d3c83875288a7991699c removed the hack from the parent code paths, but left the unnecessary code in the child. keeping it would require rewriting two code paths rather than one, and is useless for reasons described in that commit.
* synchronously clean up pthread_create failure due to scheduling errorsRich Felker2019-09-061-13/+18
| | | | | | | | | | | | | | | | | | | previously, when pthread_create failed due to inability to set explicit scheduling according to the requested attributes, the nascent thread was detached and made responsible for its own cleanup via the standard pthread_exit code path. this left it consuming resources potentially well after pthread_create returned, in a way that the application could not see or mitigate, and unnecessarily exposed its existence to the rest of the implementation via the global thread list. instead, attempt explicit scheduling early and reuse the failure path for __clone failure if it fails. the nascent thread's exit futex is not needed for unlocking the thread list, since the thread calling pthread_create holds the thread list lock the whole time, so it can be repurposed to ensure the thread has finished exiting. no pthread_exit is needed, and freeing the stack, if needed, can happen just as it would if __clone failed.
* set explicit scheduling for new thread from calling thread, not selfRich Felker2019-09-061-21/+12
| | | | | | | | | | | | | | | if setting scheduling properties succeeds, the new thread may end up with lower priority than the caller, and may be unable to continue running due to another intermediate-priority thread. this produces a priority inversion situation for the thread calling pthread_create, since it cannot return until the new thread reports success. originally, the parent was responsible for setting the new thread's priority; commits b8742f32602add243ee2ce74d804015463726899 and 40bae2d32fd6f3ffea437fa745ad38a1fe77b27e changed it as part of trimming down the pthread structure. since then, commit 04335d9260c076cf4d9264bd93dd3b06c237a639 partly reversed the changes, but did not switch responsibilities back. do that now.
* fix unsynchronized decrement of thread count on pthread_create errorRich Felker2019-09-061-1/+2
| | | | | | | commit 8f11e6127fe93093f81a52b15bb1537edc3fc8af wrongly documented that all changes to libc.threads_minus_1 were guarded by the thread list lock, but the decrement for failed SYS_clone took place after the thread list lock was released.
* in arm cancellation point asm, don't unnecessarily preserve link registerPatrick Oppenlander2019-08-061-4/+4
| | | | | | | The only reason we needed to preserve the link register was because we were using a branch-link instruction to branch to __cp_cancel. Replacing this with a branch means we can avoid the save/restore as the link register is no longer modified.
* fix missing declarations for pthread_join extensions in source fileRich Felker2019-08-021-0/+1
| | | | | | | | per policy, define the feature test macro to get declarations for the pthread_tryjoin_np and pthread_timedjoin_np functions. in the past this has been only for checking; with 32-bit archs getting 64-bit time_t it will also be necessary for symbols to get redirected correctly.
* remove x32 syscall timespec fixup hacksRich Felker2019-07-292-43/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | the x32 syscall interfaces treat timespec's tv_nsec member as 64-bit despite the API type being long and long being 32-bit in the ABI. this is no problem for syscalls that store timespecs to userspace as results, but caused uninitialized padding to be misinterpreted as the high bits in syscalls that take timespecs as input. since the beginning of the port, we've dealt with this situation with hacks in syscall_arch.h, and injected between __syscall_cp_c and __syscall_cp_asm, to special-case the syscall numbers that involve timespecs as inputs and copy them to a form suitable to pass to the kernel. commit 40aa18d55ab763e69ad16d0cf1cebea708ffde47 set the stage for removal of these hacks by letting us treat the "normal" x32 syscalls dealing with timespec as if they're x32's "time64" syscalls, effectively making x32 ax "time64-only 32-bit arch" like riscv32 will be when it's added. since then, all users of syscalls that x32's syscall_arch.h had hacks for have been updated to use time64 syscalls, so the hacks can be removed. there are still at least a few other timespec-related syscalls broken on x32, which were overlooked when the x32 hacks were done or added later. these include at least recvmmsg, adjtimex/clock_adjtime, and timerfd_settime, and they will be fixed independently later on.
* futex wait operations: add time64 syscall support, decouple 32-bit time_tRich Felker2019-07-282-3/+41
| | | | | | | | | | | | | | | | | | | | | thanks to the original factorization using the __timedwait function, there are no FUTEX_WAIT calls anywhere else, giving us a single point of change to make nearly all the timed thread primitives time64-ready. the one exception is the FUTEX_LOCK_PI command for PI mutex timedlock. I haven't tried to make these two points share code, since they have different fallbacks (no non-private fallback needed for PI since PI was added later) and FUTEX_LOCK_PI isn't a cancellation point (thus allowing the whole code path to inline into pthread_mutex_timedlock). as for other changes in this series, the time64 syscall is used only if it's the only one defined for the arch, or if the requested timeout does not fit in 32 bits. on current 32-bit archs where time_t is a 32-bit type, this makes it statically unreachable. on 64-bit archs, there are only superficial changes to the code after preprocessing. on current 32-bit archs, the time is passed via an intermediate copy to remove the assumption that time_t is a 32-bit type.
* refactor thrd_sleep and nanosleep in terms of clock_nanosleepRich Felker2019-07-271-1/+2
| | | | | | | | | | | for namespace-safety with thrd_sleep, this requires an alias, which is also added. this eliminates all but one direct call point for nanosleep syscalls, and arranges that 64-bit time_t conversion logic will only need to exist in one file rather than three. as a bonus, clock_nanosleep with CLOCK_REALTIME and empty flags is now implemented as SYS_nanosleep, thereby working on older kernels that may lack POSIX clocks functionality.
* add riscv64 architecture supportRich Felker2019-06-144-0/+76
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Author: Alex Suykov <alex.suykov@gmail.com> Author: Aric Belsito <lluixhi@gmail.com> Author: Drew DeVault <sir@cmpwn.com> Author: Michael Clark <mjc@sifive.com> Author: Michael Forney <mforney@mforney.org> Author: Stefan O'Rear <sorear2@gmail.com> This port has involved the work of many people over several years. I have tried to ensure that everyone with substantial contributions has been credited above; if any omissions are found they will be noted later in an update to the authors/contributors list in the COPYRIGHT file. The version committed here comes from the riscv/riscv-musl repo's commit 3fe7e2c75df78eef42dcdc352a55757729f451e2, with minor changes by me for issues found during final review: - a_ll/a_sc atomics are removed (according to the ISA spec, lr/sc are not safe to use in separate inline asm fragments) - a_cas[_p] is fixed to be a memory barrier - the call from the _start assembly into the C part of crt1/ldso is changed to allow for the possibility that the linker does not place them nearby each other. - DTP_OFFSET is defined correctly so that local-dynamic TLS works - reloc.h LDSO_ARCH logic is simplified and made explicit. - unused, non-functional crti/n asm files are removed. - an empty .sdata section is added to crt1 so that the __global_pointer reference is resolvable. - indentation style errors in some asm files are fixed.
* remove external __syscall function and last remaining usersRich Felker2019-04-101-1/+1
| | | | | | | | | | | | | | the weak version of __syscall_cp_c was using a tail call to __syscall to avoid duplicating the 6-argument syscall code inline in small static-linked programs, but now that __syscall no longer exists, the inline expansion is no longer duplication. the syscall.h machinery suppported up to 7 syscall arguments, only via an external __syscall function, but we presently have no syscall call points that actually make use of that many, and the kernel only defines 7-argument calling conventions for arm, powerpc (32-bit), and sh. if it turns out we need them in the future, they can easily be added.
* overhaul i386 syscall mechanism not to depend on external asm sourceRich Felker2019-04-101-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | this is the first part of a series of patches intended to make __syscall fully self-contained in the object file produced using syscall.h, which will make it possible for crt1 code to perform syscalls. the (confusingly named) i386 __vsyscall mechanism, which this commit removes, was introduced before the presence of a valid thread pointer was mandatory; back then the thread pointer was setup lazily only if threads were used. the intent was to be able to perform syscalls using the kernel's fast entry point in the VDSO, which can use the sysenter (Intel) or syscall (AMD) instruction instead of int $128, but without inlining an access to the __syscall global at the point of each syscall, which would incur a significant size cost from PIC setup everywhere. the mechanism also shuffled registers/calling convention around to avoid spills of call-saved registers, and to avoid allocating ebx or ebp via asm constraints, since there are plenty of broken-but-supported compiler versions which are incapable of allocating ebx with -fPIC or ebp with -fno-omit-frame-pointer. the new mechanism preserves the properties of avoiding spills and avoiding allocation of ebx/ebp in constraints, but does it inline, using some fairly simple register shuffling, and uses a field of the thread structure rather than global data for the vdso-provided syscall code address. for now, the external __syscall function is refactored not to use the old __vsyscall so it can be kept, but the intent is to remove it too.
* fix harmless-by-chance typo in priority inheritance mutex codeRich Felker2019-04-011-1/+1
| | | | | | | commit 54ca677983d47529bab8752315ac1a2b49888870 inadvertently introduced bitwise and where logical and was intended. since the right-hand operand is always 0 or -1 whenever the left-hand operand is nonzero, the behavior happened to be equivalent.
* implement priority inheritance mutexesRich Felker2019-03-314-8/+93
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | priority inheritance is a feature to mitigate priority inversion situations, where a execution of a medium-priority thread can unboundedly block forward progress of a high-priority thread when a lock it needs is held by a low-priority thread. the natural way to do priority inheritance would be with a simple futex flag to donate the calling thread's priority to a target thread while it waits on the futex. unfortunately, linux does not offer such an interface, but instead insists on implementing the whole locking protocol in kernelspace with special futex commands that exist solely for the purpose of doing PI mutexes. this would require the entire "trylock" logic to be duplicated in the timedlock code path for PI mutexes, since, once the previous lock holder releases the lock and the futex call returns, the lock is already held by the caller. obviously such code duplication is undesirable. instead, I've made the PI timedlock success path set the mutex lock count to -1, which can be thought of as "not yet complete", since a lock count of 0 is "locked, with no recursive references". a simple branch in a non-hot path of pthread_mutex_trylock can then see and act on this state, skipping past the code that would check and take the lock to the same code path that runs after the lock is obtained for a non-PI mutex. because we're forced to let the kernel perform the actual lock and unlock operations whenever the mutex is contended, we have to patch things up when it does the wrong thing: 1. the lock operation is not aware of whether the mutex is error-checking, so it will always fail with EDEADLK rather than deadlocking. 2. the lock operation is not aware of whether the mutex is robust, so it will successfully obtain mutexes in the owner-died state even if they're non-robust, whereas this operation should deadlock. 3. the unlock operation always sets the lock value to zero, whereas for robust mutexes, we want to set it to a special value indicating that the mutex obtained after its owner died was unlocked without marking it consistent, so that future operations all fail with ENOTRECOVERABLE. the first of these is easy to solve, just by performing a futex wait on a dummy futex address to simulate deadlock or ETIMEDOUT as appropriate. but problems 2 and 3 interact in a nasty way. to solve problem 2, we need to back out the spurious success. but if waiters are present -- which we can't just ignore, because even if we don't want to wake them, the calling thread is incorrectly inheriting their priorities -- this requires using the kernel's unlock operation, which will zero the lock value, thereby losing the "owner died with lock held" state. to solve these problems, we overload the mutex's waiters field, which is unused for PI mutexes since they don't call the normal futex wait functions, as an indicator that the PI mutex is permanently non-lockable. originally I wanted to use the count field, but there is one code path that needs to access this flag without synchronization: trylock's CAS failure path needs to be able to decide whether to fail with EBUSY or ENOTRECOVERABLE, the waiters field is already treated as a relaxed-order atomic in our memory model, so this works out nicely.
* clean up access to mutex type in pthread_mutex_trylockRich Felker2019-03-291-2/+2
| | | | | | | there was no point in masking off the pshared bit when first loading the type, since every subsequent access involves a mask anyway. not masking it may avoid a subsequent load to check the pshared flag, and it's just simpler.
* fix data race choosing next key slot in pthread_key_createRich Felker2019-03-211-1/+1
| | | | | | | | | commit 84d061d5a31c9c773e29e1e2b1ffe8cb9557bc58 wrongly moved the access to the global next_key outside of the scope of the lock. the error manifested as spurious failure to find an available key slot under concurrent calls to pthread_key_create, since the stopping condition could be met after only a small number of slots were examined.
* fix namespace violation in dependencies of mtx_lockRich Felker2019-03-131-1/+1
| | | | | | commit 2de29bc994029b903a366b8a4a9f8c3c3ee2be90 left behind one reference to pthread_mutex_trylock. fixing this also improves code generation due to the namespace-safe version being hidde.
* add membarrier syscall wrapper, refactor dynamic tls install to use itRich Felker2019-02-221-2/+2
| | | | | | | | | | the motivation for this change is twofold. first, it gets the fallback logic out of the dynamic linker, improving code readability and organization. second, it provides application code that wants to use the membarrier syscall, which depends on preregistration of intent before the process becomes multithreaded unless unbounded latency is acceptable, with a symbol that, when linked, ensures that this registration happens.
* make thread list lock a recursive lockRich Felker2019-02-221-11/+21
| | | | | | this is a prerequisite for factoring the membarrier fallback code into a function that can be called from a context with the thread list already locked or independently.
* install dynamic tls synchronously at dlopen, streamline accessRich Felker2019-02-183-14/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | previously, dynamic loading of new libraries with thread-local storage allocated the storage needed for all existing threads at load-time, precluding late failure that can't be handled, but left installation in existing threads to take place lazily on first access. this imposed an additional memory access and branch on every dynamic tls access, and imposed a requirement, which was not actually met, that the dynamic tlsdesc asm functions preserve all call-clobbered registers before calling C code to to install new dynamic tls on first access. the x86[_64] versions of this code wrongly omitted saving and restoring of fpu/vector registers, assuming the compiler would not generate anything using them in the called C code. the arm and aarch64 versions saved known existing registers, but failed to be future-proof against expansion of the register file. now that we track live threads in a list, it's possible to install the new dynamic tls for each thread at dlopen time. for the most part, synchronization is not needed, because if a thread has not synchronized with completion of the dlopen, there is no way it can meaningfully request access to a slot past the end of the old dtv, which remains valid for accessing slots which already existed. however, it is necessary to ensure that, if a thread sees its new dtv pointer, it sees correct pointers in each of the slots that existed prior to the dlopen. my understanding is that, on most real-world coherency architectures including all the ones we presently support, a built-in consume order guarantees this; however, don't rely on that. instead, the SYS_membarrier syscall is used to ensure that all threads see the stores to the slots of their new dtv prior to the installation of the new dtv. if it is not supported, the same is implemented in userspace via signals, using the same mechanism as __synccall. the __tls_get_addr function, variants, and dynamic tlsdesc asm functions are all updated to remove the fallback paths for claiming new dynamic tls, and are now all branch-free.
* fix data race between new pthread_key_delete and dtor executionRich Felker2019-02-171-2/+4
| | | | | | | | | | | | access to clear the entry in each thread's tsd array for the key being deleted was not synchronized with __pthread_tsd_run_dtors. I probably made this mistake from a mistaken belief that the thread list lock was held during the latter, which of course is not possible since it executes application code in a still-live-thread context. while we're at it, expand the interval during which signals are blocked to cover taking the write lock on key_lock, so that a signal at an inopportune time doesn't block forward progress of readers.
* introduce namespace-safe rwlock aliases; use in pthread_key_createRich Felker2019-02-168-20/+34
| | | | | | | commit 84d061d5a31c9c773e29e1e2b1ffe8cb9557bc58 inadvertently introduced namespace violations by using the pthread-namespace rwlock functions in pthread_key_create, which is in turn used for C11 tss. fix that and possible future uses of rwlocks elsewhere.
* rewrite pthread_key_delete to use global thread listRich Felker2019-02-162-75/+19
| | | | | | | | | | | | with the availability of the thread list, there is no need to mark tsd key slots dirty and clean them up only when a free slot can't be found. instead, directly iterate threads and clear any value associated with the key being deleted. no synchronization is necessary for the clearing, since there is no way the slot can be accessed without having synchronized with the creation of a new key occupying the same slot, which is already sequenced after and synchronized with the deletion of the old key.
* rewrite __synccall in terms of global thread listRich Felker2019-02-162-123/+59
| | | | | | | | | | | | | | | | the __synccall mechanism provides stop-the-world synchronous execution of a callback in all threads of the process. it is used to implement multi-threaded setuid/setgid operations, since Linux lacks them at the kernel level, and for some other less-critical purposes. this change eliminates dependency on /proc/self/task to determine the set of live threads, which in addition to being an unwanted dependency and a potential point of resource-exhaustion failure, turned out to be inaccurate. test cases provided by Alexey Izbyshev showed that it could fail to reflect newly created threads. due to how the presignaling phase worked, this usually yielded a deadlock if hit, but in the worst case it could also result in threads being silently missed (allowed to continue running without executing the callback).
* track all live threads in an AS-safe, fully-consistent linked listRich Felker2019-02-154-38/+81
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | the hard problem here is unlinking threads from a list when they exit without creating a window of inconsistency where the kernel task for a thread still exists and is still executing instructions in userspace, but is not reflected in the list. the magic solution here is getting rid of per-thread exit futex addresses (set_tid_address), and instead using the exit futex to unlock the global thread list. since pthread_join can no longer see the thread enter a detach_state of EXITED (which depended on the exit futex address pointing to the detach_state), it must now observe the unlocking of the thread list lock before it can unmap the joined thread and return. it doesn't actually have to take the lock. for this, a __tl_sync primitive is offered, with a signature that will allow it to be enhanced for quick return even under contention on the lock, if needed. for now, the exiting thread always performs a futex wake on its detach_state. a future change could optimize this out except when there is already a joiner waiting. initial/dynamic variants of detached state no longer need to be tracked separately, since the futex address is always set to the global list lock, not a thread-local address that could become invalid on detached thread exit. all detached threads, however, must perform a second sigprocmask syscall to block implementation-internal signals, since locking the thread list with them already blocked is not permissible. the arch-independent C version of __unmapself no longer needs to take a lock or setup its own futex address to release the lock, since it must necessarily be called with the thread list lock already held, guaranteeing exclusive access to the temporary stack. changes to libc.threads_minus_1 no longer need to be atomic, since they are guarded by the thread list lock. it is largely vestigial at this point, and can be replaced with a cheaper boolean indicating whether the process is multithreaded at some point in the future.
* always block signals for starting new threads, refactor start argsRich Felker2019-02-152-56/+56
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | whether signals need to be blocked at thread start, and whether unblocking is necessary in the entry point function, has historically depended on intricacies of the cancellation design and on whether there are scheduling operations to perform on the new thread before its successful creation can be committed. future changes to track an AS-safe list of live threads will require signals to be blocked whenever changes are made to the list, so ... prior to commits b8742f32602add243ee2ce74d804015463726899 and 40bae2d32fd6f3ffea437fa745ad38a1fe77b27e, a signal mask for the entry function to restore was part of the pthread structure. it was removed to trim down the size of the structure, which both saved a small amount of stack space and improved code generation on archs where small immediate displacements are less costly than arbitrary ones, by limiting the range of offsets between the base of the thread structure, its members, and the thread pointer. these commits moved the saved mask to a special structure used only when special scheduling was needed, in which case the pthread_create caller and new thread had to synchronize with each other and could use this memory to pass a mask. this commit partially reverts the above two commits, but instead of putting the mask back in the pthread structure, it moves all "start argument" members out of the pthread structure, trimming it down further, and puts them in a separate structure passed on the new thread's stack. the code path for explicit scheduling of the new thread is also changed to synchronize with the calling thread in such a way to avoid spurious futex wakes.
* redesign robust mutex states to eliminate data races on type fieldRich Felker2019-02-124-12/+23
| | | | | | | | | | | | | | | | | | | | | | | in order to implement ENOTRECOVERABLE, the implementation has traditionally used a bit of the mutex type field to indicate that it's recovered after EOWNERDEAD and will go into ENOTRECOVERABLE state if pthread_mutex_consistent is not called before unlocking. while it's only the thread that holds the lock that needs access to this information (except possibly for the sake of pthread_mutex_consistent choosing between EINVAL and EPERM for erroneous calls), the change to the type field is formally a data race with all other threads that perform any operation on the mutex. no individual bits race, and no write races are possible, so things are "okay" in some sense, but it's still not good. this patch moves the recovery/consistency state to the mutex owner/lock field which is rightfully mutable. bit 30, the same bit the kernel uses with a zero owner to indicate that the previous owner died holding the lock, is now used with a nonzero owner to indicate that the mutex is held but has not yet been marked consistent. note that the kernel ABI also reserves bit 29 not to appear in any tid, so the sentinel value we use for ENOTRECOVERABLE, 0x7fffffff, does not clash with any tid plus bit 30.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-22 13:08:44 +0000