x86: Harden printf against non-normal long double values (bug 26649)

The behavior of isnan/__builtin_isnan on bit patterns that do not
correspond to something that the CPU would produce from valid inputs
is currently under-defined in the toolchain. (The GCC built-in and
glibc disagree.)

The isnan check in PRINTF_FP_FETCH in stdio-common/printf_fp.c
assumes the GCC behavior that returns true for non-normal numbers
which are not specified as NaN. (The glibc implementation returns
false for such numbers.)

At present, passing non-normal numbers to __mpn_extract_long_double
causes this function to produce irregularly shaped multi-precision
integers, triggering undefined behavior in __printf_fp_l.

With GCC 10 and glibc 2.32, this behavior is not visible because
__builtin_isnan is used, which avoids calling
__mpn_extract_long_double in this case.  This commit updates the
implementation of __mpn_extract_long_double so that regularly shaped
multi-precision integers are produced in this case, avoiding
undefined behavior in __printf_fp_l.

1 files changed, 8 insertions, 0 deletions

diff --git a/sysdeps/x86/ldbl2mpn.c b/sysdeps/x86/ldbl2mpn.c
index ec8464eef7..23afedfb67 100644
--- a/sysdeps/x86/ldbl2mpn.c
+++ b/sysdeps/x86/ldbl2mpn.c
@@ -115,6 +115,14 @@ __mpn_extract_long_double (mp_ptr res_ptr, mp_size_t size,
 	   && res_ptr[N - 1] == 0)
     /* Pseudo zero.  */
     *expt = 0;
+  else
+    /* Unlike other floating point formats, the most significant bit
+       is explicit and expected to be set for normal numbers.  Set it
+       in case it is cleared in the input.  Otherwise, callers will
+       not be able to produce the expected multi-precision integer
+       layout by shifting.  */
+    res_ptr[N - 1] |= (mp_limb_t) 1 << (LDBL_MANT_DIG - 1
+					- ((N - 1) * BITS_PER_MP_LIMB));
 
   return N;
 }