Move malloc hooks into a compat DSO

Remove all malloc hook uses from core malloc functions and move it
into a new library libc_malloc_debug.so.  With this, the hooks now no
longer have any effect on the core library.

libc_malloc_debug.so is a malloc interposer that needs to be preloaded
to get hooks functionality back so that the debugging features that
depend on the hooks, i.e. malloc-check, mcheck and mtrace work again.
Without the preloaded DSO these debugging features will be nops.
These features will be ported away from hooks in subsequent patches.

Similarly, legacy applications that need hooks functionality need to
preload libc_malloc_debug.so.

The symbols exported by libc_malloc_debug.so are maintained at exactly
the same version as libc.so.

Finally, static binaries will no longer be able to use malloc
debugging features since they cannot preload the debugging DSO.

Reviewed-by: Carlos O'Donell <carlos@redhat.com>
Tested-by: Carlos O'Donell <carlos@redhat.com>

1 files changed, 8 insertions, 1 deletions

diff --git a/Rules b/Rules
index ba13598df6..b1137afe71 100644
--- a/Rules
+++ b/Rules
@@ -279,10 +279,17 @@ endif
 
 # All malloc-check tests will be run with MALLOC_CHECK_=3
 define malloc-check-ENVS
-$(1)-malloc-check-ENV = MALLOC_CHECK_=3
+$(1)-malloc-check-ENV = MALLOC_CHECK_=3 \
+			LD_PRELOAD=$(common-objpfx)/malloc/libc_malloc_debug.so
 endef
 $(foreach t,$(tests-malloc-check),$(eval $(call malloc-check-ENVS,$(t))))
 
+# mcheck tests need the debug DSO to support -lmcheck.
+define mcheck-ENVS
+$(1)-mcheck-ENV = LD_PRELOAD=$(common-objpfx)/malloc/libc_malloc_debug.so
+endef
+$(foreach t,$(tests-mcheck),$(eval $(call mcheck-ENVS,$(t))))
+
 ifneq "$(strip $(tests) $(tests-internal) $(xtests) $(test-srcs))" ""
 # These are the implicit rules for making test outputs
 # from the test programs and whatever input files are present.