Fix nextafter overflow in non-default rounding modes (bug 16677).

ISO C requires the result of nextafter to be independent of the
rounding mode, even when underflow or overflow occurs.  This patch
fixes the bug in various nextafter implementations that, having done
an overflowing computation to force an overflow exception (correct),
they then return the result of that computation rather than an
infinity computed some other way (incorrect, when the overflowing
result of arithmetic with that sign and rounding mode is finite but
the correct result is infinite) - generally by falling through to
existing code to return a value that in fact is correct for this case
(but was computed by an integer increment and so without generating
the exceptions required).  Having fixed the bug, the previously
deferred conversion of nextafter testing in libm-test.inc to
ALL_RM_TEST is also included.

Tested x86_64 and x86; also spot-checked results of nextafter tests
for powerpc32 and mips64 to test the ldbl-128ibm and ldbl-128
changes.  (The m68k change is untested.)

	[BZ #16677]
	* math/s_nextafter.c (__nextafter): Do not return value from
	overflowing computation.
	* sysdeps/i386/fpu/s_nextafterl.c (__nextafterl): Likewise.
	* sysdeps/ieee754/flt-32/s_nextafterf.c (__nextafterf): Likewise.
	* sysdeps/ieee754/ldbl-128/s_nextafterl.c (__nextafterl):
	Likewise.
	* sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c (__nextafterl):
	Likewise.
	* sysdeps/m68k/m680x0/fpu/s_nextafterl.c (__nextafterl): Likewise.
	* math/libm-test.inc (nextafter_test): Use ALL_RM_TEST.

1 files changed, 10 insertions, 4 deletions

diff --git a/sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c b/sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c
index c050944c0c..30b1540a88 100644
--- a/sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c
+++ b/sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c
@@ -66,8 +66,11 @@ long double __nextafterl(long double x, long double y)
 	       long double with a 106 bit mantissa, and nextafterl
 	       is insane with variable precision.  So to make
 	       nextafterl sane we assume 106 bit precision.  */
-	    if((hx==0xffefffffffffffffLL)&&(lx==0xfc8ffffffffffffeLL))
-	      return x+x;	/* overflow, return -inf */
+	    if((hx==0xffefffffffffffffLL)&&(lx==0xfc8ffffffffffffeLL)) {
+	      u = x+x;	/* overflow, return -inf */
+	      math_force_eval (u);
+	      return y;
+	    }
 	    if (hx >= 0x7ff0000000000000LL) {
 	      u = 0x1.fffffffffffff7ffffffffffff8p+1023L;
 	      return u;
@@ -93,8 +96,11 @@ long double __nextafterl(long double x, long double y)
 	    }
 	    return x - u;
 	} else {				/* x < y, x += ulp */
-	    if((hx==0x7fefffffffffffffLL)&&(lx==0x7c8ffffffffffffeLL))
-	      return x+x;	/* overflow, return +inf */
+	    if((hx==0x7fefffffffffffffLL)&&(lx==0x7c8ffffffffffffeLL)) {
+	      u = x+x;	/* overflow, return +inf */
+	      math_force_eval (u);
+	      return y;
+	    }
 	    if ((uint64_t) hx >= 0xfff0000000000000ULL) {
 	      u = -0x1.fffffffffffff7ffffffffffff8p+1023L;
 	      return u;