Support -fsanitize=address with --enable-asan

When --enable-asan is provided to configure then build all user
space components with fsanitize=address.  For kernel support
use the Linux KASAN feature instead.

https://github.com/google/sanitizers/wiki/AddressSanitizer

When using gcc version 4.8 any test case which intentionally
generates a core dump will fail when using --enable-asan.
The default behavior is to disable core dumps and only newer
versions allow this behavior to be controled at run time with
the ASAN_OPTIONS environment variable.

Additionally, this patch includes some build system cleanup.

* Rules.am updated to set the minimum AM_CFLAGS, AM_CPPFLAGS,
  and AM_LDFLAGS.  Any additional flags should be added on a
  per-Makefile basic.  The --enable-debug and --enable-asan
  options apply to all user space binaries and libraries.

* Compiler checks consolidated in always-compiler-options.m4
  and renamed for consistency.

* -fstack-check compiler flag was removed, this functionality
  is provided by asan when configured with --enable-asan.

* Split DEBUG_CFLAGS in to DEBUG_CFLAGS, DEBUG_CPPFLAGS, and
  DEBUG_LDFLAGS.

* Moved default kernel build flags in to module/Makefile.in and
  split in to ZFS_MODULE_CFLAGS and ZFS_MODULE_CPPFLAGS.  These
  flags are set with the standard ccflags-y kbuild mechanism.

* -Wframe-larger-than checks applied only to binaries or
  libraries which include source files which are built in
  both user space and kernel space.  This restriction is
  relaxed for user space only utilities.

* -Wno-unused-but-set-variable applied only to libzfs and
  libzpool.  The remaining warnings are the result of an
  ASSERT using a variable when is always declared.

* -D_POSIX_PTHREAD_SEMANTICS and -D__EXTENSIONS__ dropped
  because they are Solaris specific and thus not needed.

* Ensure $GDB is defined as gdb by default in zloop.sh.

Signed-off-by: DHE <[email protected]>
Signed-off-by: Brian Behlendorf <[email protected]>
Closes #7027 

5 files changed, 8 insertions, 3 deletions

diff --git a/tests/zfs-tests/tests/functional/checksum/Makefile.am b/tests/zfs-tests/tests/functional/checksum/Makefile.am
index 8132ea1d6..cacb3c927 100644
--- a/tests/zfs-tests/tests/functional/checksum/Makefile.am
+++ b/tests/zfs-tests/tests/functional/checksum/Makefile.am
@@ -1,4 +1,5 @@
 include $(top_srcdir)/config/Rules.am
+
 AM_CPPFLAGS += -I$(top_srcdir)/include
 LDADD = $(top_srcdir)/lib/libicp/libicp.la
 
diff --git a/tests/zfs-tests/tests/functional/cli_root/zfs/zfs_002_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zfs/zfs_002_pos.ksh
index 27a429db8..b21b6c657 100755
--- a/tests/zfs-tests/tests/functional/cli_root/zfs/zfs_002_pos.ksh
+++ b/tests/zfs-tests/tests/functional/cli_root/zfs/zfs_002_pos.ksh
@@ -93,6 +93,7 @@ if is_linux; then
 	ulimit -c unlimited
 	echo "$corepath/core.zfs" >/proc/sys/kernel/core_pattern
 	echo 0 >/proc/sys/kernel/core_uses_pid
+	export ASAN_OPTIONS="abort_on_error=1:disable_coredump=0"
 else
 	log_must coreadm -p ${corepath}/core.%f
 fi
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_002_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_002_pos.ksh
index a45e4a883..d4abaa5ff 100755
--- a/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_002_pos.ksh
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_002_pos.ksh
@@ -89,6 +89,7 @@ if is_linux; then
 	ulimit -c unlimited
 	echo "$corepath/core.zpool" >/proc/sys/kernel/core_pattern
 	echo 0 >/proc/sys/kernel/core_uses_pid
+	export ASAN_OPTIONS="abort_on_error=1:disable_coredump=0"
 else
 	coreadm -p ${corepath}/core.%f
 fi
diff --git a/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_003_pos.ksh b/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_003_pos.ksh
index 8bfbb5fd8..0f04f0c04 100755
--- a/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_003_pos.ksh
+++ b/tests/zfs-tests/tests/functional/cli_root/zpool/zpool_003_pos.ksh
@@ -61,9 +61,10 @@ log_mustnot zpool freeze fakepool
 [[ -f core ]] && log_must rm -f core
 
 if is_linux; then
-        ulimit -c unlimited
-        echo "core" >/proc/sys/kernel/core_pattern
-        echo 0 >/proc/sys/kernel/core_uses_pid
+	ulimit -c unlimited
+	echo "core" >/proc/sys/kernel/core_pattern
+	echo 0 >/proc/sys/kernel/core_uses_pid
+	export ASAN_OPTIONS="abort_on_error=1:disable_coredump=0"
 fi
 
 ZFS_ABORT=1; export ZFS_ABORT
diff --git a/tests/zfs-tests/tests/functional/hkdf/Makefile.am b/tests/zfs-tests/tests/functional/hkdf/Makefile.am
index 37fe2ed2c..d0a68f442 100644
--- a/tests/zfs-tests/tests/functional/hkdf/Makefile.am
+++ b/tests/zfs-tests/tests/functional/hkdf/Makefile.am
@@ -1,4 +1,5 @@
 include $(top_srcdir)/config/Rules.am
+
 AM_CPPFLAGS += -I$(top_srcdir)/include
 AM_CPPFLAGS += -I$(top_srcdir)/lib/libspl/include
 LDADD = $(top_srcdir)/lib/libicp/libicp.la