x86: Harden printf against non-normal long double values (bug 26649)

The behavior of isnan/__builtin_isnan on bit patterns that do not
correspond to something that the CPU would produce from valid inputs
is currently under-defined in the toolchain. (The GCC built-in and
glibc disagree.)

The isnan check in PRINTF_FP_FETCH in stdio-common/printf_fp.c
assumes the GCC behavior that returns true for non-normal numbers
which are not specified as NaN. (The glibc implementation returns
false for such numbers.)

At present, passing non-normal numbers to __mpn_extract_long_double
causes this function to produce irregularly shaped multi-precision
integers, triggering undefined behavior in __printf_fp_l.

With GCC 10 and glibc 2.32, this behavior is not visible because
__builtin_isnan is used, which avoids calling
__mpn_extract_long_double in this case.  This commit updates the
implementation of __mpn_extract_long_double so that regularly shaped
multi-precision integers are produced in this case, avoiding
undefined behavior in __printf_fp_l.

1 files changed, 4 insertions, 0 deletions

diff --git a/sysdeps/x86/Makefile b/sysdeps/x86/Makefile
index c369faf00d..081cc72e93 100644
--- a/sysdeps/x86/Makefile
+++ b/sysdeps/x86/Makefile
@@ -11,6 +11,10 @@ tests += tst-get-cpu-features tst-get-cpu-features-static \
 tests-static += tst-get-cpu-features-static
 endif
 
+ifeq ($(subdir),math)
+tests += tst-ldbl-nonnormal-printf
+endif # $(subdir) == math
+
 ifeq ($(subdir),setjmp)
 gen-as-const-headers += jmp_buf-ssp.sym
 sysdep_routines += __longjmp_cancel