malloc: Fix for infinite loop in memalign/posix_memalign.

A very large alignment argument passed to mealign/posix_memalign
causes _int_memalign to enter an infinite loop. Limit the maximum
alignment value to the maximum representable power of two to
prevent this from happening.

Changelog:

2013-10-30  Will Newton  <will.newton@linaro.org>

	[BZ #16038]
	* malloc/hooks.c (memalign_check): Limit alignment to the
	maximum representable power of two.
	* malloc/malloc.c (__libc_memalign): Likewise.
	* malloc/tst-memalign.c (do_test): Add test for very
	large alignment values.
	* malloc/tst-posix_memalign.c (do_test): Likewise.

1 files changed, 8 insertions, 0 deletions

diff --git a/malloc/hooks.c b/malloc/hooks.c
index 3f663bb6b2..1dbe93f383 100644
--- a/malloc/hooks.c
+++ b/malloc/hooks.c
@@ -361,6 +361,14 @@ memalign_check(size_t alignment, size_t bytes, const void *caller)
   if (alignment <= MALLOC_ALIGNMENT) return malloc_check(bytes, NULL);
   if (alignment <  MINSIZE) alignment = MINSIZE;
 
+  /* If the alignment is greater than SIZE_MAX / 2 + 1 it cannot be a
+     power of 2 and will cause overflow in the check below.  */
+  if (alignment > SIZE_MAX / 2 + 1)
+    {
+      __set_errno (EINVAL);
+      return 0;
+    }
+
   /* Check for overflow.  */
   if (bytes > SIZE_MAX - alignment - MINSIZE)
     {