about summary refs log tree commit diff
Commit message (Collapse)AuthorAgeFilesLines
...
* x86-64: Add strcpy family functions with 256-bit EVEXH.J. Lu2021-03-299-3/+1339
| | | | | | Update ifunc-strcpy.h to select the function optimized with 256-bit EVEX instructions using YMM16-YMM31 registers to avoid RTM abort with usable AVX512VL and AVX512BW since VZEROUPPER isn't needed at function exit.
* x86-64: Add ifunc-avx2.h functions with 256-bit EVEXH.J. Lu2021-03-2924-17/+2995
| | | | | | | | | | Update ifunc-avx2.h, strchr.c, strcmp.c, strncmp.c and wcsnlen.c to select the function optimized with 256-bit EVEX instructions using YMM16-YMM31 registers to avoid RTM abort with usable AVX512VL, AVX512BW and BMI2 since VZEROUPPER isn't needed at function exit. For strcmp/strncmp, prefer AVX2 strcmp/strncmp if Prefer_AVX2_STRCMP is set.
* x86: Set Prefer_No_VZEROUPPER and add Prefer_AVX2_STRCMPH.J. Lu2021-03-293-2/+21
| | | | | | | | | 1. Set Prefer_No_VZEROUPPER if RTM is usable to avoid RTM abort triggered by VZEROUPPER inside a transactionally executing RTM region. 2. Since to compare 2 32-byte strings, 256-bit EVEX strcmp requires 2 loads, 3 VPCMPs and 2 KORDs while AVX2 strcmp requires 1 load, 2 VPCMPEQs, 1 VPMINU and 1 VPMOVMSKB, AVX2 strcmp is faster than EVEX strcmp. Add Prefer_AVX2_STRCMP to prefer AVX2 strcmp family functions.
* add workload traces for missing functions (double format)Paul Zimmermann2021-03-2915-4/+13231
| | | | | | | | | | This patch adds workload traces for all double format functions where such files are missing. For each function, a set of 1000 random values is generated at random using SageMath, such that the output values are meaningful (for example avoiding too large inputs for exp10 where the output would be +Inf). More details about the generated values are given at the beginning of each file. Reviewed-by: Siddhesh Poyarekar <siddhesh@sourceware.org>
* linux: Add y2106 support on utimensat testsAdhemerval Zanella2021-03-294-166/+119
| | | | | | | | | | | | The tests are refactored to use a common skeleton that handles whether the underlying filesystem supports 64 bit time, skips 64 bit time tests when the TU only supports 32 bit, and also skip 64 bit time tests larger than 32 unsigned int (y2106) if the system does not support it (MIPSn64 on kernels without statx support). Checked on x86_64-linux-gnu and i686-linux-gnu. I also checked on a mips64el-linux-gnu with 4.1.4 and 5.10.0-4-5kc-malta kernel to verify if the y2106 are indeed skipped.
* support: Add support_path_support_time64_valueAdhemerval Zanella2021-03-292-9/+12
| | | | It allows to check for support on arbitrary timestamp values.
* linux: Use statx for MIPSn64Adhemerval Zanella2021-03-293-33/+29
| | | | | | | | | | | MIPSn64 kernel ABI for legacy stat uses unsigned 32 bit for second timestamp, which limits the maximum value to y2106. This patch make mips64 use statx as for 32-bit architectures. Thie __cp_stat64_t64_statx is open coded, its usage is solely on fstatat64 and it avoid the need to redefine the name for mips64 (which will call __cp_stat64_statx since its does not use __stat64_t64 internally).
* linux: Disable fstatat64 fallback if __ASSUME_STATX is definedAdhemerval Zanella2021-03-291-16/+40
| | | | | | | | | | | | If the minimum kernel supports statx there is no need to call the fallback stat legacy syscalls. The statx is also called on compat xstat syscall, but different than the fstatat it calls no fallback and it is assumed to be always present. Checked on powerpc-linux-gnu (with and without --enable-kernel=4.11) and on powerpc64-linux-gnu.
* linux: Implement fstatat with __fstatat64_time64Adhemerval Zanella2021-03-291-38/+11
| | | | | | | | | It makes fstatat use __NR_statx, which fix the s390 issue with missing nanoxsecond support on compat stat syscalls (at least on recent kernels) and limits the statx call to only one function (which simplifies the __ASSUME_STATX support). Checked on i686-linux-gnu and on powerpc-linux-gnu.
* x86: Properly disable XSAVE related features [BZ #27605]H.J. Lu2021-03-292-0/+56
| | | | | | | 1. Support GLIBC_TUNABLES=glibc.cpu.hwcaps=-XSAVE. 2. Disable all features which depend on XSAVE: a. If OSXSAVE is disabled by glibc tunables. Or b. If both XSAVE and XSAVEC aren't usable.
* test-strnlen.c: Check that strnlen won't go beyond the maximum lengthH.J. Lu2021-03-271-0/+30
| | | | | Place strings ending at page boundary without the null byte. If an implementation goes beyond EXP_LEN, it will trigger the segfault.
* test-strnlen.c: Initialize wchar_t string with wmemset [BZ #27655]H.J. Lu2021-03-271-1/+3
| | | | Use wmemset to initialize wchar_t string.
* nptl: Remove unused rulesAdhemerval Zanella2021-03-261-5/+0
| | | | The wrapper are not exported by libpthread.
* nptl: Remove __libc_allocate_rtsig, __libc_current_sigrtmax, and ↵Adhemerval Zanella2021-03-2636-183/+10
| | | | | | | | __libc_current_sigrtmin The libc version is identical and built with same flags. Checked on x86_64-linux-gnu.
* nptl: Move sigaction to libcAdhemerval Zanella2021-03-2633-95/+3
| | | | | | The libc version is identical and built with same flags. Checked on x86_64-linux-gnu.
* nptl: Remove pthread raise implementationAdhemerval Zanella2021-03-2634-81/+0
| | | | | | | | | | The Linux version already target the current thread by using tgkill along with getpid and gettid. For arm, libpthread does not do a intra PLT since it will call the raise from libc. Checked on x86_64-linux-gnu.
* nptl: Move pthread_kill to libcAdhemerval Zanella2021-03-2664-32/+75
| | | | | | A new 2.34 version is also provided. Checked on x86_64-linux-gnu.
* nptl: Remove sigsuspend from pthread-compat-wrappersAdhemerval Zanella2021-03-261-7/+0
| | | | The symbol is not exported by libpthread.
* nptl: Remove msgrcv from pthread-compat-wrappersAdhemerval Zanella2021-03-261-2/+0
| | | | The symbol is not exported by libpthread.
* nptl: Remove msgsnd from pthread-compat-wrappersAdhemerval Zanella2021-03-261-2/+1
| | | | The symbol is not exported by libpthread.
* nptl: Remove pwrite from libpthreadAdhemerval Zanella2021-03-2641-94/+38
| | | | | | | The libc version is identical and built with same flags, it is also uses as the default version. Checked on x86_64-linux-gnu.
* nptl: Remove pread from libpthreadAdhemerval Zanella2021-03-2641-94/+39
| | | | | | | The libc version is identical and built with same flags, it is also uses as the default version. Checked on x86_64-linux-gnu.
* nptl: Remove open from libpthreadAdhemerval Zanella2021-03-2640-127/+15
| | | | | | | The libc version is identical and built with same flags. The libc version is set as the default version. Checked on x86_64-linux-gnu.
* nptl: Remove lseek from libpthreadAdhemerval Zanella2021-03-2640-95/+13
| | | | | | | | | | | The libc version is identical and built with same flags. The libc version is set as the default version. The libpthread compat symbol requires to mask it when building the loader object otherwise ld might complain about a missing versioned symbol (as for alpha). Checked on x86_64-linux-gnu.
* nptl: Remove send from libpthreadAdhemerval Zanella2021-03-2640-76/+21
| | | | | | | | | | | | | The libc version is identical and built with same flags. Both aarch64 and nios2 also requires to export __send and tt was done previously with the HAVE_INTERNAL_SEND_SYMBOL (which forced the symbol creation). All __send callers are internal to libc and the original issue that required the symbol export was due a missing libc_hidden_def. So a compat symbol is added for __send and the libc_hidden_def is defined regardless. Checked on x86_64-linux-gnu and i686-linux-gnu.
* Add OTHER_SHLIB_COMPATAdhemerval Zanella2021-03-261-0/+9
| | | | | | It is similar to SHLIB_COMPAT, but allows to check versions from other libraries. It is used to move compat symbol from other libraries to libc.
* misc: Fix tst-select timeout handling (BZ#27648)Adhemerval Zanella2021-03-261-30/+55
| | | | | | | | Instead of polling the stderr, create two pipes and fork to check if child timeout as expected similar to tst-pselect.c. Also lower the timeout value. Checked on x86_64-linux-gnu.
* malloc: Ensure mtag code path in checked_request2size is coldSzabolcs Nagy2021-03-261-2/+7
| | | | | | | | | This is a workaround (hack) for a gcc optimization issue (PR 99551). Without this the generated code may evaluate the expression in the cold path which causes performance regression for small allocations in the memory tagging disabled (common) case. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Remove unnecessary tagging around _mid_memalignSzabolcs Nagy2021-03-261-8/+2
| | | | | | | | The internal _mid_memalign already returns newly tagged memory. (__libc_memalign and posix_memalign already relied on this, this patch fixes the other call sites.) Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Rename chunk2rawmemSzabolcs Nagy2021-03-262-43/+43
| | | | | | | | | | | The previous patch ensured that all chunk to mem computations use chunk2rawmem, so now we can rename it to chunk2mem, and in the few cases where the tag of mem is relevant chunk2mem_tag can be used. Replaced tag_at (chunk2rawmem (x)) with chunk2mem_tag (x). Renamed chunk2rawmem to chunk2mem. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Use chunk2rawmem throughoutSzabolcs Nagy2021-03-262-26/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The difference between chunk2mem and chunk2rawmem is that the latter does not get the memory tag for the returned pointer. It turns out chunk2rawmem almost always works: The input of chunk2mem is a chunk pointer that is untagged so it can access the chunk header. All memory that is not user allocated heap memory is untagged, which in the current implementation means that it has the 0 tag, but this patch does not rely on the tag value. The patch relies on that chunk operations are either done on untagged chunks or without doing memory access to the user owned part. Internal interface contracts: sysmalloc: Returns untagged memory. _int_malloc: Returns untagged memory. _int_free: Takes untagged memory. _int_memalign: Returns untagged memory. _int_realloc: Takes and returns tagged memory. So only _int_realloc and functions outside this list need care. Alignment checks do not need the right tag and tcache works with untagged memory. tag_at was kept in realloc after an mremap, which is not strictly necessary, since the pointer is only used to retag the memory, but this way the tag is guaranteed to be different from the old tag. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Use different tag after mremapSzabolcs Nagy2021-03-261-1/+1
| | | | | | | | The comment explained why different tag is used after mremap, but for that correctly tagged pointer should be passed to tag_new_usable. Use chunk2mem to get the tag. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Use memsize instead of CHUNK_AVAILABLE_SIZESzabolcs Nagy2021-03-262-26/+24
| | | | | | | | | | | | | | | This is a pure refactoring change that does not affect behaviour. The CHUNK_AVAILABLE_SIZE name was unclear, the memsize name tries to follow the existing convention of mem denoting the allocation that is handed out to the user, while chunk is its internally used container. The user owned memory for a given chunk starts at chunk2mem(p) and the size is memsize(p). It is not valid to use on dumped heap chunks. Moved the definition next to other chunk and mem related macros. Reviewed-by: DJ Delorie <dj@redhat.com>
* aarch64: Optimize __libc_mtag_tag_zero_regionSzabolcs Nagy2021-03-261-16/+80
| | | | | | | | This is a target hook for memory tagging, the original was a naive implementation. Uses the same algorithm as __libc_mtag_tag_region, but with instructions that also zero the memory. This was not benchmarked on real cpu, but expected to be faster than the naive implementation.
* aarch64: Optimize __libc_mtag_tag_regionSzabolcs Nagy2021-03-261-18/+80
| | | | | | | | This is a target hook for memory tagging, the original was a naive implementation. The optimized version relies on "dc gva" to tag 64 bytes at a time for large allocations and optimizes small cases without adding too many branches. This was not benchmarked on real cpu, but expected to be faster than the naive implementation.
* aarch64: inline __libc_mtag_new_tagSzabolcs Nagy2021-03-263-41/+11
| | | | | This is a common operation when heap tagging is enabled, so inline the instructions instead of using an extern call.
* aarch64: inline __libc_mtag_address_get_tagSzabolcs Nagy2021-03-263-39/+10
| | | | | | | | | | | | This is a common operation when heap tagging is enabled, so inline the instruction instead of using an extern call. The .inst directive is used instead of the name of the instruction (or acle intrinsics) because malloc.c is not compiled for armv8.5-a+memtag architecture, runtime cpu support detection is used. Prototypes are removed from the comments as they were not always correct.
* malloc: Use mtag_enabled instead of USE_MTAGSzabolcs Nagy2021-03-262-12/+8
| | | | | | | | | Use the runtime check where possible: it should not cause slow down in the !USE_MTAG case since then mtag_enabled is constant false, but it allows compiling the tagging logic so it's less likely to break or diverge when developers only test the !USE_MTAG case. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Use branches instead of mtag_granule_maskSzabolcs Nagy2021-03-262-21/+14
| | | | | | | | | | | | | The branches may be better optimized since mtag_enabled is widely used. Granule size larger than a chunk header is not supported since then we cannot have both the chunk header and user area granule aligned. To fix that for targets with large granule, the chunk layout has to change. So code that attempted to handle the granule mask generally was changed. This simplified CHUNK_AVAILABLE_SIZE and the logic in malloc_usable_size. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Change calloc when tagging is disabledSzabolcs Nagy2021-03-261-6/+4
| | | | | | | | | | | | When glibc is built with memory tagging support (USE_MTAG) but it is not enabled at runtime (mtag_enabled) then unconditional memset was used even though that can be often avoided. This is for performance when tagging is supported but not enabled. The extra check should have no overhead: tag_new_zero_region already had a runtime check which the compiler can now optimize away. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Only support zeroing and not arbitrary memset with mtagSzabolcs Nagy2021-03-265-26/+21
| | | | | | | | | | The memset api is suboptimal and does not provide much benefit. Memory tagging only needs a zeroing memset (and only for memory that's sized and aligned to multiples of the tag granule), so change the internal api and the target hooks accordingly. This is to simplify the implementation of the target hook. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Use global flag instead of function pointer dispatch for mtagSzabolcs Nagy2021-03-262-52/+39
| | | | | | | | | | | | | | | | | | A flag check can be faster than function pointers because of how branch prediction and speculation works and it can also remove a layer of indirection when there is a mismatch between the malloc internal tag_* api and __libc_mtag_* target hooks. Memory tagging wrapper functions are moved to malloc.c from arena.c and the logic now checks mmap_enabled. The definition of tag_new_usable is moved after chunk related definitions. This refactoring also allows using mtag_enabled checks instead of USE_MTAG ifdefs when memory tagging support only changes code logic when memory tagging is enabled at runtime. Note: an "if (false)" code block is optimized away even at -O0 by gcc. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Refactor TAG_ macros to avoid indirectionSzabolcs Nagy2021-03-263-56/+51
| | | | | | | | | | | This does not change behaviour, just removes one layer of indirection in the internal memory tagging logic. Use tag_ and mtag_ prefixes instead of __tag_ and __mtag_ since these are all symbols with internal linkage, private to malloc.c, so there is no user namespace pollution issue. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Ensure the generic mtag hooks are not usedSzabolcs Nagy2021-03-261-10/+31
| | | | | | | | | | | | | | | | | | Use inline functions instead of macros, because macros can cause unused variable warnings and type conversion issues. We assume these functions may appear in the code but only in dead code paths (hidden by a runtime check), so it's important that they can compile with correct types, but if they are actually used that should be an error. Currently the hooks are only used when USE_MTAG is true which only happens on aarch64 and then the aarch64 specific code is used not this generic header. However followup refactoring will allow the hooks to be used with !USE_MTAG. Note: the const qualifier in the comment was wrong: changing tags is a write operation. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Avoid taggig mmaped memory on freeSzabolcs Nagy2021-03-261-3/+4
| | | | | | | | | | Either the memory belongs to the dumped area, in which case we don't want to tag (the dumped area has the same tag as malloc internal data so tagging is unnecessary, but chunks there may not have the right alignment for the tag granule), or the memory will be unmapped immediately (and thus tagging is not useful). Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Simplify __mtag_tag_new_usableSzabolcs Nagy2021-03-261-5/+0
| | | | | | | | | The chunk cannot be a dumped one here. The only non-obvious cases are free and realloc which may be called on a dumped area chunk, but in both cases it can be verified that tagging is already avoided for dumped area chunks. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Move MTAG_MMAP_FLAGS definitionSzabolcs Nagy2021-03-262-7/+2
| | | | | | | This is only used internally in malloc.c, the extern declaration was wrong, __mtag_mmap_flags has internal linkage. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Fix a potential realloc issue with memory taggingSzabolcs Nagy2021-03-261-7/+7
| | | | | | | | | | | | At an _int_free call site in realloc the wrong size was used for tag clearing: the chunk header of the next chunk was also cleared which in practice may work, but logically wrong. The tag clearing is moved before the memcpy to save a tag computation, this avoids a chunk2mem. Another chunk2mem is removed because newmem does not have to be recomputed. Whitespaces got fixed too. Reviewed-by: DJ Delorie <dj@redhat.com>
* malloc: Fix a realloc crash with heap tagging [BZ 27468]Szabolcs Nagy2021-03-261-1/+3
| | | | | | | | | | | | _int_free must be called with a chunk that has its tag reset. This was missing in a rare case that could crash when heap tagging is enabled: when in a multi-threaded process the current arena runs out of memory during realloc, but another arena still has space to finish the realloc then _int_free was called without clearing the user allocation tags. Fixes bug 27468. Reviewed-by: DJ Delorie <dj@redhat.com>
* S390: Also check vector support in memmove ifunc-selector [BZ #27511]Stefan Liebler2021-03-264-6/+15
| | | | | | | | | | | | | | | The arch13 memmove variant is currently selected by the ifunc selector if the Miscellaneous-Instruction-Extensions Facility 3 facility bit is present, but the function is also using vector instructions. If the vector support is not present, one is receiving an operation exception. Therefore this patch also checks for vector support in the ifunc selector and in ifunc-impl-list.c. Just to be sure, the configure check is now also testing an arch13 vector instruction and an arch13 Miscellaneous-Instruction-Extensions Facility 3 instruction.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-07-09 00:41:28 +0000