about summary refs log tree commit diff
path: root/malloc/arena.c
Commit message (Collapse)AuthorAgeFilesLines
* Update copyright dates with scripts/update-copyrights.Joseph Myers2020-01-011-1/+1
|
* Prefer https to http for gnu.org and fsf.org URLsPaul Eggert2019-09-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Also, change sources.redhat.com to sourceware.org. This patch was automatically generated by running the following shell script, which uses GNU sed, and which avoids modifying files imported from upstream: sed -ri ' s,(http|ftp)(://(.*\.)?(gnu|fsf|sourceware)\.org($|[^.]|\.[^a-z])),https\2,g s,(http|ftp)(://(.*\.)?)sources\.redhat\.com($|[^.]|\.[^a-z]),https\2sourceware.org\4,g ' \ $(find $(git ls-files) -prune -type f \ ! -name '*.po' \ ! -name 'ChangeLog*' \ ! -path COPYING ! -path COPYING.LIB \ ! -path manual/fdl-1.3.texi ! -path manual/lgpl-2.1.texi \ ! -path manual/texinfo.tex ! -path scripts/config.guess \ ! -path scripts/config.sub ! -path scripts/install-sh \ ! -path scripts/mkinstalldirs ! -path scripts/move-if-change \ ! -path INSTALL ! -path locale/programs/charmap-kw.h \ ! -path po/libc.pot ! -path sysdeps/gnu/errlist.c \ ! '(' -name configure \ -execdir test -f configure.ac -o -f configure.in ';' ')' \ ! '(' -name preconfigure \ -execdir test -f preconfigure.ac ';' ')' \ -print) and then by running 'make dist-prepare' to regenerate files built from the altered files, and then executing the following to cleanup: chmod a+x sysdeps/unix/sysv/linux/riscv/configure # Omit irrelevant whitespace and comment-only changes, # perhaps from a slightly-different Autoconf version. git checkout -f \ sysdeps/csky/configure \ sysdeps/hppa/configure \ sysdeps/riscv/configure \ sysdeps/unix/sysv/linux/csky/configure # Omit changes that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/powerpc/powerpc64/ppc-mcount.S: trailing lines git checkout -f \ sysdeps/powerpc/powerpc64/ppc-mcount.S \ sysdeps/unix/sysv/linux/s390/s390-64/syscall.S # Omit change that caused a pre-commit check to fail like this: # remote: *** error: sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S: last line does not end in newline git checkout -f sysdeps/sparc/sparc64/multiarch/memcpy-ultra3.S
* Add glibc.malloc.mxfast tunableDJ Delorie2019-08-091-0/+2
| | | | | | | | | | | | * elf/dl-tunables.list: Add glibc.malloc.mxfast. * manual/tunables.texi: Document it. * malloc/malloc.c (do_set_mxfast): New. (__libc_mallopt): Call it. * malloc/arena.c: Add mxfast tunable. * malloc/tst-mxfast.c: New. * malloc/Makefile: Add it. Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* Remove do_set_mallopt_check prototypeH.J. Lu2019-04-231-1/+0
| | | | | | | | | Remove do_set_mallopt_check prototype since it is unused. * malloc/arena.c (do_set_mallopt_check): Removed. Reviewed-by: Carlos O'Donell <carlos@redhat.com> Reviewed-by: DJ Delorie <dj@redhat.com>
* Update copyright dates with scripts/update-copyrights.Joseph Myers2019-01-011-1/+1
| | | | | | | * All files with FSF copyright notices: Update copyright dates using scripts/update-copyrights. * locale/programs/charmap-kw.h: Regenerated. * locale/programs/locfile-kw.h: Likewise.
* malloc: Convert the unlink macro to the unlink_chunk functionFlorian Weimer2018-11-121-2/+2
| | | | | | | This commit is in preparation of turning the macro into a proper function. The output arguments of the macro were in fact unused. Also clean up uses of __builtin_expect.
* Run thread shutdown functions in an explicit orderFlorian Weimer2018-06-261-3/+2
| | | | | | | This removes the __libc_thread_subfreeres hook in favor of explict calls. Reviewed-by: Carlos O'Donell <carlos@redhat.com>
* Update copyright dates with scripts/update-copyrights.Joseph Myers2018-01-011-1/+1
| | | | | | | * All files with FSF copyright notices: Update copyright dates using scripts/update-copyrights. * locale/programs/charmap-kw.h: Regenerated. * locale/programs/locfile-kw.h: Likewise.
* malloc: Call tcache destructor in arena_thread_freeresFlorian Weimer2017-11-231-0/+5
| | | | | | | | | It does not make sense to register separate cleanup functions for arena and tcache since they're always going to be called together. Call the tcache cleanup function from within arena_thread_freeres since it at least makes the order of those cleanups clear in the code. Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
* Improve malloc initialization sequenceWilco Dijkstra2017-10-171-9/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | The current malloc initialization is quite convoluted. Instead of sometimes calling malloc_consolidate from ptmalloc_init, call malloc_init_state early so that the main_arena is always initialized. The special initialization can now be removed from malloc_consolidate. This also fixes BZ #22159. Check all calls to malloc_consolidate and remove calls that are redundant initialization after ptmalloc_init, like in int_mallinfo and __libc_mallopt (but keep the latter as consolidation is required for set_max_fast). Update comments to improve clarity. Remove impossible initialization check from _int_malloc, fix assert in do_check_malloc_state to ensure arena->top != 0. Fix the obvious bugs in do_check_free_chunk and do_check_remalloced_chunk to enable single threaded malloc debugging (do_check_malloc_state is not thread safe!). [BZ #22159] * malloc/arena.c (ptmalloc_init): Call malloc_init_state. * malloc/malloc.c (do_check_free_chunk): Fix build bug. (do_check_remalloced_chunk): Fix build bug. (do_check_malloc_state): Add assert that checks arena->top. (malloc_consolidate): Remove initialization. (int_mallinfo): Remove call to malloc_consolidate. (__libc_mallopt): Clarify why malloc_consolidate is needed.
* malloc: Remove the internal_function attributeFlorian Weimer2017-08-311-7/+0
|
* malloc: Remove corrupt arena flagFlorian Weimer2017-08-301-18/+2
| | | | | This is no longer needed because we now abort immediately once heap corruption is detected.
* malloc: Remove check_action variable [BZ #21754]Florian Weimer2017-08-301-8/+3
| | | | | | | | | | | Clean up calls to malloc_printerr and trim its argument list. This also removes a few bits of work done before calling malloc_printerr (such as unlocking operations). The tunable/environment variable still enables the lightweight additional malloc checking, but mallopt (M_CHECK_ACTION) no longer has any effect.
* Add per-thread cache to mallocDJ Delorie2017-07-061-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | * config.make.in: Enable experimental malloc option. * configure.ac: Likewise. * configure: Regenerate. * manual/install.texi: Document it. * INSTALL: Regenerate. * malloc/Makefile: Likewise. * malloc/malloc.c: Add per-thread cache (tcache). (tcache_put): New. (tcache_get): New. (tcache_thread_freeres): New. (tcache_init): New. (__libc_malloc): Use cached chunks if available. (__libc_free): Initialize tcache if needed. (__libc_realloc): Likewise. (__libc_calloc): Likewise. (_int_malloc): Prefill tcache when appropriate. (_int_free): Likewise. (do_set_tcache_max): New. (do_set_tcache_count): New. (do_set_tcache_unsorted_limit): New. * manual/probes.texi: Document new probes. * malloc/arena.c: Add new tcache tunables. * elf/dl-tunables.list: Likewise. * manual/tunables.texi: Document them. * NEWS: Mention the per-thread cache.
* tunables: Clean up hooks to get and set tunablesSiddhesh Poyarekar2017-06-071-18/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The TUNABLE_SET_VALUE and family of macros (and my later attempt to add a TUNABLE_GET) never quite went together very well because the overall interface was not clearly defined. This patch is an attempt to do just that. This patch consolidates the API to two simple sets of macros, TUNABLE_GET* and TUNABLE_SET*. If TUNABLE_NAMESPACE is defined, TUNABLE_GET takes just the tunable name, type and a (optionally NULL) callback function to get the value of the tunable. The callback function, if non-NULL, is called if the tunable was externally set (i.e. via GLIBC_TUNABLES or any future mechanism). For example: val = TUNABLE_GET (check, int32_t, check_callback) returns the value of the glibc.malloc.check tunable (assuming TUNABLE_NAMESPACE is set to malloc) as an int32_t into VAL after calling check_callback. Likewise, TUNABLE_SET can be used to set the value of the tunable, although this is currently possible only in the dynamic linker before it relocates itself. For example: TUNABLE_SET (check, int32_t, 2) will set glibc.malloc.check to 2. Of course, this is not possible since we set (or read) glibc.malloc.check long after it is relocated. To access or set a tunable outside of TUNABLE_NAMESPACE, use the TUNABLE_GET_FULL and TUNABLE_SET_FULL macros, which have the following prototype: TUNABLE_GET_FULL (glibc, tune, hwcap_mask, uint64_t, NULL) TUNABLE_SET_FULL (glibc, tune, hwcap_mask, uint64_t, 0xffff) In future the tunable list may get split into mutable and immutable tunables where mutable tunables can be modified by the library and userspace after relocation as well and TUNABLE_SET will be more useful than it currently is. However whenever we actually do that split, we will have to ensure that the mutable tunables are protected with locks. * elf/Versions (__tunable_set_val): Rename to __tunable_get_val. * elf/dl-tunables.c: Likewise. (do_tunable_update_val): New function. (__tunable_set_val): New function. (__tunable_get_val): Call CB only if the tunable was externally initialized. (tunables_strtoul): Replace strval with initialized. * elf/dl-tunables.h (strval): Replace with a bool initialized. (TUNABLE_ENUM_NAME, TUNABLE_ENUM_NAME1): Adjust names to prevent collision. (__tunable_set_val): New function. (TUNABLE_GET, TUNABLE_GET_FULL): New macros. (TUNABLE_SET, TUNABLE_SET_FULL): Likewise. (TUNABLE_SET_VAL): Remove. (TUNABLE_SET_VAL_WITH_CALLBACK): Likewise. * README.tunables: Document the new macros. * malloc/arena.c (ptmalloc_init): Adjust.
* Fix getting tunable values on big-endian (BZ #21109)Siddhesh Poyarekar2017-02-081-4/+4
| | | | | | | The code to set value passed a tunable_val_t, which when cast to int32_t on big-endian gives the wrong value. Instead, use tunable_val_t.numval instead, which can then be safely cast into int32_t.
* Update copyright dates with scripts/update-copyrights.Joseph Myers2017-01-011-1/+1
|
* Add framework for tunablesSiddhesh Poyarekar2016-12-311-0/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The tunables framework allows us to uniformly manage and expose global variables inside glibc as switches to users. tunables/README has instructions for glibc developers to add new tunables. Tunables support can be enabled by passing the --enable-tunables configure flag to the configure script. This patch only adds a framework and does not pose any limitations on how tunable values are read from the user. It also adds environment variables used in malloc behaviour tweaking to the tunables framework as a PoC of the compatibility interface. * manual/install.texi: Add --enable-tunables option. * INSTALL: Regenerate. * README.tunables: New file. * Makeconfig (CPPFLAGS): Define TOP_NAMESPACE. (before-compile): Generate dl-tunable-list.h early. * config.h.in: Add HAVE_TUNABLES. * config.make.in: Add have-tunables. * configure.ac: Add --enable-tunables option. * configure: Regenerate. * csu/init-first.c (__libc_init_first): Move __libc_init_secure earlier... * csu/init-first.c (LIBC_START_MAIN):... to here. Include dl-tunables.h, libc-internal.h. (LIBC_START_MAIN) [!SHARED]: Initialize tunables for static binaries. * elf/Makefile (dl-routines): Add dl-tunables. * elf/Versions (ld): Add __tunable_set_val to GLIBC_PRIVATE namespace. * elf/dl-support (_dl_nondynamic_init): Unset MALLOC_CHECK_ only when !HAVE_TUNABLES. * elf/rtld.c (process_envvars): Likewise. * elf/dl-sysdep.c [HAVE_TUNABLES]: Include dl-tunables.h (_dl_sysdep_start): Call __tunables_init. * elf/dl-tunable-types.h: New file. * elf/dl-tunables.c: New file. * elf/dl-tunables.h: New file. * elf/dl-tunables.list: New file. * malloc/tst-malloc-usable-static.c: New test case. * malloc/Makefile (tests-static): Add it. * malloc/arena.c [HAVE_TUNABLES]: Include dl-tunables.h. Define TUNABLE_NAMESPACE. (DL_TUNABLE_CALLBACK (set_mallopt_check)): New function. (DL_TUNABLE_CALLBACK_FNDECL): New macro. Use it to define callback functions. (ptmalloc_init): Set tunable values. * scripts/gen-tunables.awk: New file. * sysdeps/mach/hurd/dl-sysdep.c: Include dl-tunables.h. (_dl_sysdep_start): Call __tunables_init.
* malloc: Use accessors for chunk metadata accessFlorian Weimer2016-10-281-3/+3
| | | | | This change allows us to change the encoding of these struct members in a centralized fashion.
* malloc: Manual part of conversion to __libc_lockFlorian Weimer2016-09-211-3/+3
| | | | | This removes the old mutex_t-related definitions from malloc-machine.h, too.
* malloc: Automated part of conversion to __libc_lockFlorian Weimer2016-09-061-26/+26
|
* malloc: Preserve arena free list/thread count invariant [BZ #20370]Florian Weimer2016-08-021-5/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | It is necessary to preserve the invariant that if an arena is on the free list, it has thread attach count zero. Otherwise, when arena_thread_freeres sees the zero attach count, it will add it, and without the invariant, an arena could get pushed to the list twice, resulting in a cycle. One possible execution trace looks like this: Thread 1 examines free list and observes it as empty. Thread 2 exits and adds its arena to the free list, with attached_threads == 0). Thread 1 selects this arena in reused_arena (not from the free list). Thread 1 increments attached_threads and attaches itself. (The arena remains on the free list.) Thread 1 exits, decrements attached_threads, and adds the arena to the free list. The final step creates a cycle in the usual way (by overwriting the next_free member with the former list head, while there is another list item pointing to the arena structure). tst-malloc-thread-exit exhibits this issue, but it was only visible with a debugger because the incorrect fix in bug 19243 removed the assert from get_free_list.
* malloc: Avoid premature fallback to mmap [BZ #20284]Florian Weimer2016-06-211-6/+4
| | | | | | | Before this change, the while loop in reused_arena which avoids returning a corrupt arena would never execute its body if the selected arena were not corrupt. As a result, result == begin after the loop, and the function returns NULL, triggering fallback to mmap.
* Revert __malloc_initialize_hook symbol poisoningFlorian Weimer2016-06-201-1/+1
| | | | | | It turns out the Emacs-internal malloc implementation uses __malloc_* symbols. If glibc poisons them in <stdc-pre.h>, Emacs will no longer compile.
* malloc: Remove __malloc_initialize_hook from the API [BZ #19564]Florian Weimer2016-06-101-1/+3
| | | | | | | __malloc_initialize_hook is interposed by application code, so the usual approach to define a compatibility symbol does not work. This commit adds a new mechanism based on #pragma GCC poison in <stdc-predef.h>.
* malloc: Add missing internal_function attributes on function definitionsFlorian Weimer2016-04-141-0/+3
| | | | Fixes build on i386 after commit 29d794863cd6e03115d3670707cc873a9965ba92.
* malloc: Remove malloc hooks from fork handlerFlorian Weimer2016-04-141-117/+8
| | | | | | | The fork handler now runs so late that there is no risk anymore that other fork handlers in the same thread use malloc, so it is no longer necessary to install malloc hooks which made a subset of malloc functionality available to the thread that called fork.
* malloc: Run fork handler as late as possible [BZ #19431]Florian Weimer2016-04-141-41/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, a thread M invoking fork would acquire locks in this order: (M1) malloc arena locks (in the registered fork handler) (M2) libio list lock A thread F invoking flush (NULL) would acquire locks in this order: (F1) libio list lock (F2) individual _IO_FILE locks A thread G running getdelim would use this order: (G1) _IO_FILE lock (G2) malloc arena lock After executing (M1), (F1), (G1), none of the threads can make progress. This commit changes the fork lock order to: (M'1) libio list lock (M'2) malloc arena locks It explicitly encodes the lock order in the implementations of fork, and does not rely on the registration order, thus avoiding the deadlock.
* malloc: Remove NO_THREADSFlorian Weimer2016-02-191-3/+0
| | | | | No functional change. It was not possible to build without threading support before.
* malloc: Remove arena_mem variableFlorian Weimer2016-02-191-6/+0
| | | | The computed value is never used. The accesses were data races.
* Update copyright dates with scripts/update-copyrights.Joseph Myers2016-01-041-1/+1
|
* malloc: Update comment for list_lockFlorian Weimer2015-12-231-3/+4
|
* malloc: Fix list_lock/arena lock deadlock [BZ #19182]Florian Weimer2015-12-211-15/+51
| | | | | | | | | | | | | | | | | | * malloc/arena.c (list_lock): Document lock ordering requirements. (free_list_lock): New lock. (ptmalloc_lock_all): Comment on free_list_lock. (ptmalloc_unlock_all2): Reinitialize free_list_lock. (detach_arena): Update comment. free_list_lock is now needed. (_int_new_arena): Use free_list_lock around detach_arena call. Acquire arena lock after list_lock. Add comment, including FIXME about incorrect synchronization. (get_free_list): Switch to free_list_lock. (reused_arena): Acquire free_list_lock around detach_arena call and attached threads counter update. Add two FIXMEs about incorrect synchronization. (arena_thread_freeres): Switch to free_list_lock. * malloc/malloc.c (struct malloc_state): Update comments to mention free_list_lock.
* malloc: Fix attached thread reference count handling [BZ #19243]Florian Weimer2015-12-161-4/+8
| | | | | | | | | | | | | | | | reused_arena can increase the attached thread count of arenas on the free list. This means that the assertion that the reference count is zero is incorrect. In this case, the reference count initialization is incorrect as well and could cause arenas to be put on the free list too early (while they still have attached threads). * malloc/arena.c (get_free_list): Remove assert and adjust reference count handling. Add comment about reused_arena interaction. (reused_arena): Add comments abount get_free_list interaction. * malloc/tst-malloc-thread-exit.c: New file. * malloc/Makefile (tests): Add tst-malloc-thread-exit. (tst-malloc-thread-exit): Link against libpthread.
* Replace MUTEX_INITIALIZER with _LIBC_LOCK_INITIALIZER in generic codeFlorian Weimer2015-11-241-1/+1
| | | | | | | | | | * sysdeps/mach/hurd/libc-lock.h (_LIBC_LOCK_INITIALIZER): Define. (__libc_lock_define_initialized): Use it. * sysdeps/nptl/libc-lockP.h (_LIBC_LOCK_INITIALIZER): Define. * malloc/arena.c (list_lock): Use _LIBC_LOCK_INITIALIZER. * malloc/malloc.c (main_arena): Likewise. * sysdeps/generic/malloc-machine.h (MUTEX_INITIALIZER): Remove. * sysdeps/nptl/malloc-machine.h (MUTEX_INITIALIZER): Remove.
* malloc: Prevent arena free_list from turning cyclic [BZ #19048]Florian Weimer2015-10-281-5/+65
| | | | | | | | | | | | | | | | | [BZ# 19048] * malloc/malloc.c (struct malloc_state): Update comment. Add attached_threads member. (main_arena): Initialize attached_threads. * malloc/arena.c (list_lock): Update comment. (ptmalloc_lock_all, ptmalloc_unlock_all): Likewise. (ptmalloc_unlock_all2): Reinitialize arena reference counts. (deattach_arena): New function. (_int_new_arena): Initialize arena reference count and deattach replaced arena. (get_free_list, reused_arena): Update reference count and deattach replaced arena. (arena_thread_freeres): Update arena reference count and only put unreferenced arenas on the free list.
* malloc: Rewrite with explicit TLS access using __threadFlorian Weimer2015-10-171-29/+19
|
* malloc: Consistently apply trim_threshold to all heaps (Bug 17195)Carlos O'Donell2015-10-071-2/+8
| | | | | | | | | | | | | | | | | | | | | | | | In the per-thread arenas we apply trim_threshold-based checks to the extra space between the pad and the top_area. This isn't quite accurate and instead we should be harmonizing with the way in which trim_treshold is applied everywhere else like sysrtim and _int_free. The trimming check should be based on the size of the top chunk and only the size of the top chunk. The following patch harmonizes the trimming and make it consistent for the main arena and thread arenas. In the old code a large padding request might have meant that trimming was not triggered. Now trimming is considered first based on the chunk, then the pad is subtracted, and the remainder trimmed. This is how all the other trimmings operate. I didn't measure the performance difference of this change because it corrects what I consider to be a behavioural anomaly. We'll need some profile driven optimization to make this code better, and even there Ondrej and others have better ideas on how to speedup malloc. Tested on x86_64 with no regressions. Already reviewed by Siddhesh Poyarekar and Mel Gorman here and discussed here: https://sourceware.org/ml/libc-alpha/2015-05/msg00002.html
* Don't use the main arena in retry path if it is corruptSiddhesh Poyarekar2015-08-241-0/+4
| | | | | | | | | If allocation on a non-main arena fails, the main arena is used without checking to see if it is corrupt. Add a check that avoids the main arena if it is corrupt. * malloc/arena.c (arena_get_retry): Don't use main_arena if it is corrupt.
* Drop unused first argument from arena_get2Siddhesh Poyarekar2015-08-241-5/+3
| | | | | | | | | | | | The arena pointer in the first argument to arena_get2 was used in the old days before per-thread arenas. They're unused now and hence can be dropped. ChangeLog: * malloc/arena.c (arena_get2): Drop unused argument. (arena_lock): Adjust. (arena_get_retry): Likewise.
* malloc: Do not corrupt the top of a threaded heap if top chunk is MINSIZE ↵Mel Gorman2015-06-261-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | [BZ #18502] mksquashfs was reported in openSUSE to be causing segmentation faults when creating installation images. Testing showed that mksquashfs sometimes failed and could be reproduced within 10 attempts. The core dump looked like the heap top was corrupted and was pointing to an unmapped area. In other cases, this has been due to an application corrupting glibc structures but mksquashfs appears to be fine in this regard. The problem is that heap_trim is "growing" the top into unmapped space. If the top chunk == MINSIZE then top_area is -1 and this check does not behave as expected due to a signed/unsigned comparison if (top_area <= pad) return 0; The next calculation extra = ALIGN_DOWN(top_area - pad, pagesz) calculates extra as a negative number which also is unnoticed due to a signed/unsigned comparison. We then call shrink_heap(heap, negative_number) which crashes later. This patch adds a simple check against MINSIZE to make sure extra does not become negative. It adds a cast to hint to the reader that this is a signed vs unsigned issue. Without the patch, mksquash fails within 10 attempts. With it applied, it completed 1000 times without error. The standard test suite "make check" showed no changes in the summary of test results.
* Avoid deadlock in malloc on backtrace (BZ #16159)Siddhesh Poyarekar2015-05-191-4/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the malloc subsystem detects some kind of memory corruption, depending on the configuration it prints the error, a backtrace, a memory map and then aborts the process. In this process, the backtrace() call may result in a call to malloc, resulting in various kinds of problematic behavior. In one case, the malloc it calls may detect a corruption and call backtrace again, and a stack overflow may result due to the infinite recursion. In another case, the malloc it calls may deadlock on an arena lock with the malloc (or free, realloc, etc.) that detected the corruption. In yet another case, if the program is linked with pthreads, backtrace may do a pthread_once initialization, which deadlocks on itself. In all these cases, the program exit is not as intended. This is avoidable by marking the arena that malloc detected a corruption on, as unusable. The following patch does that. Features of this patch are as follows: - A flag is added to the mstate struct of the arena to indicate if the arena is corrupt. - The flag is checked whenever malloc functions try to get a lock on an arena. If the arena is unusable, a NULL is returned, causing the malloc to use mmap or try the next arena. - malloc_printerr sets the corrupt flag on the arena when it detects a corruption - free does not concern itself with the flag at all. It is not important since the backtrace workflow does not need free. A free in a parallel thread may cause another corruption, but that's not new - The flag check and set are not atomic and may race. This is fine since we don't care about contention during the flag check. We want to make sure that the malloc call in the backtrace does not trip on itself and all that action happens in the same thread and not across threads. I verified that the test case does not show any regressions due to this patch. I also ran the malloc benchmarks and found an insignificant difference in timings (< 2%). * malloc/Makefile (tests): New test case tst-malloc-backtrace. * malloc/arena.c (arena_lock): Check if arena is corrupt. (reused_arena): Find a non-corrupt arena. (heap_trim): Pass arena to unlink. * malloc/hooks.c (malloc_check_get_size): Pass arena to malloc_printerr. (top_check): Likewise. (free_check): Likewise. (realloc_check): Likewise. * malloc/malloc.c (malloc_printerr): Add arena argument. (unlink): Likewise. (munmap_chunk): Adjust. (ARENA_CORRUPTION_BIT): New macro. (arena_is_corrupt): Likewise. (set_arena_corrupt): Likewise. (sysmalloc): Use mmap if there are no usable arenas. (_int_malloc): Likewise. (__libc_malloc): Don't fail if arena_get returns NULL. (_mid_memalign): Likewise. (__libc_calloc): Likewise. (__libc_realloc): Adjust for additional argument to malloc_printerr. (_int_free): Likewise. (malloc_consolidate): Likewise. (_int_realloc): Likewise. (_int_memalign): Don't touch corrupt arenas. * malloc/tst-malloc-backtrace.c: New test case.
* malloc: Consistently apply trim_threshold to all heaps [BZ #17195]Mel Gorman2015-04-021-3/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Trimming heaps is a balance between saving memory and the system overhead required to update page tables and discard allocated pages. The malloc option M_TRIM_THRESHOLD is a tunable that users are meant to use to decide where this balance point is but it is only applied to the main arena. For scalability reasons, glibc malloc has per-thread heaps but these are shrunk with madvise() if there is one page free at the top of the heap. In some circumstances this can lead to high system overhead if a thread has a control flow like while (data_to_process) { buf = malloc(large_size); do_stuff(); free(buf); } For a large size, the free() will call madvise (pagetable teardown, page free and TLB flush) every time followed immediately by a malloc (fault, kernel page alloc, zeroing and charge accounting). The kernel overhead can dominate such a workload. This patch allows the user to tune when madvise gets called by applying the trim threshold to the per-thread heaps and using similar logic to the main arena when deciding whether to shrink. Alternatively if the dynamic brk/mmap threshold gets adjusted then the new values will be obeyed by the per-thread heaps. Bug 17195 was a test case motivated by a problem encountered in scientific applications written in python that performance badly due to high page fault overhead. The basic operation of such a program was posted by Julian Taylor https://sourceware.org/ml/libc-alpha/2015-02/msg00373.html With this patch applied, the overhead is eliminated. All numbers in this report are in seconds and were recorded by running Julian's program 30 times. pyarray glibc madvise 2.21 v2 System min 1.81 ( 0.00%) 0.00 (100.00%) System mean 1.93 ( 0.00%) 0.02 ( 99.20%) System stddev 0.06 ( 0.00%) 0.01 ( 88.99%) System max 2.06 ( 0.00%) 0.03 ( 98.54%) Elapsed min 3.26 ( 0.00%) 2.37 ( 27.30%) Elapsed mean 3.39 ( 0.00%) 2.41 ( 28.84%) Elapsed stddev 0.14 ( 0.00%) 0.02 ( 82.73%) Elapsed max 4.05 ( 0.00%) 2.47 ( 39.01%) glibc madvise 2.21 v2 User 141.86 142.28 System 57.94 0.60 Elapsed 102.02 72.66 Note that almost a minutes worth of system time is eliminted and the program completes 28% faster on average. To illustrate the problem without python this is a basic test-case for the worst case scenario where every free is a madvise followed by a an alloc /* gcc bench-free.c -lpthread -o bench-free */ static int num = 1024; void __attribute__((noinline,noclone)) dostuff (void *p) { } void *worker (void *data) { int i; for (i = num; i--;) { void *m = malloc (48*4096); dostuff (m); free (m); } return NULL; } int main() { int i; pthread_t t; void *ret; if (pthread_create (&t, NULL, worker, NULL)) exit (2); if (pthread_join (t, &ret)) exit (3); return 0; } Before the patch, this resulted in 1024 calls to madvise. With the patch applied, madvise is called twice because the default trim threshold is high enough to avoid this. This a more complex case where there is a mix of frees. It's simply a different worker function for the test case above void *worker (void *data) { int i; int j = 0; void *free_index[num]; for (i = num; i--;) { void *m = malloc ((i % 58) *4096); dostuff (m); if (i % 2 == 0) { free (m); } else { free_index[j++] = m; } } for (; j >= 0; j--) { free(free_index[j]); } return NULL; } glibc 2.21 calls malloc 90305 times but with the patch applied, it's called 13438. Increasing the trim threshold will decrease the number of times it's called with the option of eliminating the overhead. ebizzy is meant to generate a workload resembling common web application server workloads. It is threaded with a large working set that at its core has an allocation, do_stuff, free loop that also hits this case. The primary metric of the benchmark is records processed per second. This is running on my desktop which is a single socket machine with an I7-4770 and 8 cores. Each thread count was run for 30 seconds. It was only run once as the performance difference is so high that the variation is insignificant. glibc 2.21 patch threads 1 10230 44114 threads 2 19153 84925 threads 4 34295 134569 threads 8 51007 183387 Note that the saving happens to be a concidence as the size allocated by ebizzy was less than the default threshold. If a different number of chunks were specified then it may also be necessary to tune the threshold to compensate This is roughly quadrupling the performance of this benchmark. The difference in system CPU usage illustrates why. ebizzy running 1 thread with glibc 2.21 10230 records/s 306904 real 30.00 s user 7.47 s sys 22.49 s 22.49 seconds was spent in the kernel for a workload runinng 30 seconds. With the patch applied ebizzy running 1 thread with patch applied 44126 records/s 1323792 real 30.00 s user 29.97 s sys 0.00 s system CPU usage was zero with the patch applied. strace shows that glibc running this workload calls madvise approximately 9000 times a second. With the patch applied madvise was called twice during the workload (or 0.06 times per second). 2015-02-10 Mel Gorman <mgorman@suse.de> [BZ #17195] * malloc/arena.c (free): Apply trim threshold to per-thread heaps as well as the main arena.
* Use alignment macros, pagesize and powerof2.Carlos O'Donell2015-02-171-4/+4
| | | | | | | | | | | | | | We are replacing all of the bespoke alignment code with ALIGN_UP, ALIGN_DOWN, PTR_ALIGN_UP, and PTR_ALIGN_DOWN. This cleans up malloc/malloc.c, malloc/arena.c, and elf/dl-reloc.c. It also makes all the code consistently use pagesize, and powerof2 as required. Code size is reduced with the removal of precomputed pagemask, and use of pagesize instead. No measurable difference in performance. No regressions on x86_64.
* Update copyright dates with scripts/update-copyrights.Joseph Myers2015-01-021-1/+1
|
* Use glibc_likely instead __builtin_expect.Ondřej Bílka2014-02-101-4/+4
|
* Remove THREAD_STATS.Ondřej Bílka2014-02-101-20/+0
| | | | | A THREAD_STATS macro duplicates gathering information that could be obtained by systemtap probes instead.
* Reformat malloc to gnu style.Ondřej Bílka2014-01-021-373/+418
|
* Update copyright notices with scripts/update-copyrightsAllan McRae2014-01-011-1/+1
|
* Drop PER_THREAD conditionals from malloc.Ondřej Bílka2013-12-101-76/+0
|
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-10-31 05:35:47 +0000