printf: fix edge case where hex float precision was not honored

commit cfa0a54c082d41db6446638eed1d57f163434092 attempted to fix
rounding on archs where long double is not 80-bit (where LDBL_MANT_DIG
is not zero mod four), but failed to address the edge case where
rounding was skipped because LDBL_MANT_DIG/4 rounded down in the
comparison against the requested precision.

the rounding logic based on hex digit count is difficult to understand
and not well-motivated, so rather than try to fix it, replace it with
an explicit calculation in terms of number of bits to be kept, without
any truncating division operations. based on patch by Peter Ammon, but
with scalbn to apply the rounding exponent since the value will not
generally fit in any integer type. scalbn is used instead of scalbnl
to avoid pulling in the latter unnecessarily, since the value is an
exact power of two whose exponent range is bounded by LDBL_MANT_DIG, a
small integer.

1 files changed, 2 insertions, 9 deletions

diff --git a/src/stdio/vfprintf.c b/src/stdio/vfprintf.c
index 497c5e19..dc648e7e 100644
--- a/src/stdio/vfprintf.c
+++ b/src/stdio/vfprintf.c
@@ -211,18 +211,11 @@ static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t)
 	if (y) e2--;
 
 	if ((t|32)=='a') {
-		long double round = 8.0;
-		int re;
-
 		if (t&32) prefix += 9;
 		pl += 2;
 
-		if (p<0 || p>=LDBL_MANT_DIG/4-1) re=0;
-		else re=LDBL_MANT_DIG/4-1-p;
-
-		if (re) {
-			round *= 1<<(LDBL_MANT_DIG%4);
-			while (re--) round*=16;
+		if (p>=0 && p<(LDBL_MANT_DIG-1+3)/4) {
+			double round = scalbn(1, LDBL_MANT_DIG-1-(p*4));
 			if (*prefix=='-') {
 				y=-y;
 				y-=round;