Travis CI additions and cleanup

Convert Travis build configuration to a single var instead of 4 tuple.
Makes it much easier to review the builds in the Travis web UI.

Adds sanitizer builds for Clang on both Linux and OS X. Clang is a different
compiler from GCC and its sanitizers may catch things GCC does not.
I have no idea if Apple's Clang has some magic sanitizer sauce stock
LLVM does not, so maybe sanitizer build on OS X can be skipped.

Adds Linux cross compile targets for ARM32, ARM64, PPC64, and MinGW x86
using the cross compiler available in Trusty. All of them build and are
set up to run through qemu/wine. All of the tests currently fail and
so are marked as expected fail in the Travis matrix.

The ARM test runs seem to have thread problems; ARM32 thread creation just fails
with an exception, as if pthreads was disabled. All other tests pass ok for ARM32.

On Aarch64, it looks like there is a hard crash the first time the library tries
creating a thread. Both of these might be due to statically linking the binary?

I have been unable to convince Ubuntu's qemu-ppc64 to execute binaries compiled by
Ubuntu's ppc64 cross compiler. I'm downloading an Ubuntu ISO to try this in a VM.

Running under Wine exposes several issues, both in Wine and Botan. Many functions are
stubs and it appears that entropy collection fails as a result. This triggers a bug
in the FFI tests which causes a crash there.

A pox on time zones; _mkgmtime is a MSVC extension and is not available on MinGW GCC.
Add a last resort call that just uses the localzone variant instead.

Adds valgrind target, remove a bogus poison in pubkey.cpp (it was effectively
asserting that all of RSA was const time which is sadly not true at all).

Moves -Wshadow to maintainer mode for GCC - GCC 4.8 has a noisy variant of -Wshadow
which warns if a parameter masks a function name, but this comes up all the time
in constructors. Later GCCs no longer warn about this (even with -Wshadow), so the
warnings are never fixed, but they cause noise in CI output and hide interesting
warnings like

warning: vec_lvsl is deprecated for little endian; use assignment for unaligned loads and stores [-Wdeprecated]
          __vector unsigned char perm = vec_lvsl(0, static_cast<u32bit*>(nullptr));

2 files changed, 166 insertions, 71 deletions

diff --git a/src/scripts/ci/travis/build.sh b/src/scripts/ci/travis/build.sh
index 8cdb6ccc9..ce1ac18af 100755
--- a/src/scripts/ci/travis/build.sh
+++ b/src/scripts/ci/travis/build.sh
@@ -2,97 +2,167 @@
 set -ev
 which shellcheck > /dev/null && shellcheck "$0" # Run shellcheck on this if available
 
-if [ "$BUILD_MODE" = "static" ]; then
-    CFG_FLAGS=(--disable-shared --via-amalgamation)
-elif [ "$BUILD_MODE" = "shared" ] || [ "$BUILD_MODE" = "sonarqube" ]; then
-    CFG_FLAGS=()
+MAKE_PREFIX=""
+TEST_PREFIX=""
+TEST_EXE=./botan-test
+CFG_FLAGS=(--prefix=/tmp/botan-installation --cc=$CC --os=$TRAVIS_OS_NAME)
+
+CFG_FLAGS+=(--with-pkcs11)
+
+CC_BIN=$CXX
+
+if [ "$BUILD_MODE" = "static" ] || [ "$BUILD_MODE" = "mini-static" ]; then
+    CFG_FLAGS+=(--disable-shared --via-amalgamation)
+elif [ "$BUILD_MODE" = "shared" ] || [ "$BUILD_MODE" = "mini-shared" ]; then
+    # No special flags required for shared lib build
+    CFG_FLAGS+=()
+elif [ "$BUILD_MODE" = "sonarqube" ]; then
+    # No special flags required
+    CFG_FLAGS+=()
+elif [ "$BUILD_MODE" = "parallel" ]; then
+
+    if [ "$CC" = "gcc" ]; then
+        CFG_FLAGS+=(--with-cilkplus)
+    else
+        CFG_FLAGS+=(--with-openmp)
+    fi
+
 elif [ "$BUILD_MODE" = "coverage" ]; then
-    CFG_FLAGS=(--with-coverage)
+    CFG_FLAGS+=(--with-coverage)
 elif [ "$BUILD_MODE" = "sanitizer" ]; then
-    CFG_FLAGS=(--with-sanitizers)
+    export ASAN_OPTIONS=detect_leaks=0
+    CFG_FLAGS+=(--with-sanitizers)
+elif [ "$BUILD_MODE" = "valgrind" ]; then
+    CFG_FLAGS+=(--with-valgrind --with-debug-info)
+    TEST_PREFIX="valgrind --track-origins=yes --error-exitcode=9 -v"
 fi
 
-if [ "$MODULES" = "min" ]; then
-    CFG_FLAGS+=(--minimized-build --enable-modules=base)
+if [ "$BUILD_MODE" = "mini-static" ] || [ "$BUILD_MODE" = "mini-shared" ]; then
+    CFG_FLAGS+=(--minimized-build --enable-modules="base,dev_random,system_rng,sha2_32,sha2_64,aes")
+elif [ "$BUILD_MODE" = "valgrind" ]; then
+    # Valgrind on Travis on full build takes too long and the job is killed
+    # Prune to the most important stuff
+    CFG_FLAGS+=(--module-policy=modern --enable-modules=tls)
+
+elif [ "${BUILD_MODE:0:5}" != "cross" ]; then
+    # Only use external libraries when compiling natively
+    CFG_FLAGS+=(--with-bzip2 --with-lzma --with-sqlite --with-zlib)
+
+    # Avoid OpenSSL when using dynamic checkers...
+    if [ "$BUILD_MODE" != "sanitizer" ] && [ "$BUILD_MODE" != "valgrind" ]; then
+        CFG_LFAGS+=(--with-openssl)
+    fi
 fi
 
-if [ "$BOOST" = "y" ]; then
+if [ "$TRAVIS_OS_NAME" = "osx" ] && [ "${BUILD_MODE:0:5}" != "cross" ]; then
+    # The Boost-specific codepaths are tested via the OS X CI
     CFG_FLAGS+=(--with-boost)
 fi
 
-CFG_FLAGS+=(--with-pkcs11 --prefix=/tmp/botan-installation)
+if [ "${BUILD_MODE:0:6}" = "cross-" ]; then
+    CFG_FLAGS+=(--disable-shared)
 
-# enable ccache
-if [ "$BUILD_MODE" != "sonarqube" ]; then
-    ccache --max-size=100M
-    ccache --show-stats
-    export CXX="ccache $CXX"
-fi
+    if [ "$TRAVIS_OS_NAME" = "osx" ]; then
+        MAKE_PREFIX="xcrun --sdk iphoneos"
+        if [ "$BUILD_MODE" = "cross-arm32" ]; then
+            CFG_FLAGS+=(--cpu=armv7 --cc-abi-flags="-arch armv7 -arch armv7s -stdlib=libc++")
+        elif [ "$BUILD_MODE" = "cross-arm64" ]; then
+            CFG_FLAGS+=(--cpu=armv8-a --cc-abi-flags="-arch arm64 -stdlib=libc++")
+        fi
+    elif [ "$TRAVIS_OS_NAME" = "linux" ]; then
+        CFG_FLAGS+=(--cc-abi-flags="-static-libstdc++")
 
-# configure
-if [ "$TARGETOS" = "ios32" ]; then
-    ./configure.py "${CFG_FLAGS[@]}" --cpu=armv7 --cc=clang --cc-bin="$CXX" \
-        --cc-abi-flags="-arch armv7 -arch armv7s -stdlib=libc++"
+        if [ "$BUILD_MODE" = "cross-arm32" ]; then
+            CC_BIN=arm-linux-gnueabihf-g++-4.8
+            TEST_PREFIX="qemu-arm -L /usr/arm-linux-gnueabihf/"
+            CFG_FLAGS+=(--cpu=armv7)
+        elif [ "$BUILD_MODE" = "cross-arm64" ]; then
+            CC_BIN=aarch64-linux-gnu-g++-4.8
+            TEST_PREFIX="qemu-aarch64 -L /usr/aarch64-linux-gnu/"
+            CFG_FLAGS+=(--cpu=armv8-a)
+        elif [ "$BUILD_MODE" = "cross-ppc32" ]; then
+            CC_BIN=powerpc-linux-gnu-g++-4.8
+            TEST_PREFIX="qemu-ppc -L /usr/powerpc-linux-gnu/"
+            CFG_FLAGS+=(--cpu=ppc32)
+        elif [ "$BUILD_MODE" = "cross-ppc64" ]; then
+            CC_BIN=powerpc64le-linux-gnu-g++-4.8
+            TEST_PREFIX="qemu-ppc64 -L /usr/powerpc64le-linux-gnu/"
+            CFG_FLAGS+=(--cpu=ppc64)
+        elif [ "$BUILD_MODE" = "cross-win32" ]; then
+            CC_BIN=i686-w64-mingw32-g++
+            # No test prefix needed, PE executes as usual with Wine installed
+            CFG_FLAGS+=(--cpu=x86_32 --os=windows --cc-abi-flags="-static")
+            TEST_EXE=./botan-test.exe
+        fi
+    fi
+fi
 
-elif [ "$TARGETOS" = "ios64" ]; then
-    ./configure.py "${CFG_FLAGS[@]}" --cpu=armv8-a --cc=clang --cc-bin="$CXX" \
-        --cc-abi-flags="-arch arm64 -stdlib=libc++"
+CFG_FLAGS+=(--cc-bin="ccache $CC_BIN")
 
-else
-    ./configure.py "${CFG_FLAGS[@]}" --cc="$CC" --cc-bin="$CXX" \
-        --with-bzip2 --with-lzma --with-openssl --with-sqlite --with-zlib
+if [ "$BUILD_MODE" = "sonarqube" ]; then
+   MAKE_PREFIX="./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir bw-outputs"
 fi
 
-# build
-if [ "${TARGETOS:0:3}" = "ios" ]; then
-    xcrun --sdk iphoneos make -j 2
-elif [ "$BUILD_MODE" = "sonarqube" ]; then
-    ./build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir bw-outputs make -j 2
-else
-    make -j 2
-fi
+# configure
+./configure.py "${CFG_FLAGS[@]}"
 
-# Show post-build ccache stats
-if [ "$BUILD_MODE" != "sonarqube" ]; then
-    ccache --show-stats
-fi
+# pre-build ccache stats
+ccache --show-stats
+
+# build!
+echo $MAKE_PREFIX make -j $BUILD_JOBS
+time $MAKE_PREFIX make -j $BUILD_JOBS
+
+# post-build ccache stats
+ccache --show-stats
 
 # Run SonarQube analysis
-if [ "$TRAVIS_BRANCH" = "master" ] && [ "$TRAVIS_PULL_REQUEST" = "false" ] && [ "$BUILD_MODE" = "sonarqube" ]; then
-    # => This will run a full analysis of the project and push results to the SonarQube server.
-    #
-    # Analysis is done only on master so that build of branches don't push analyses to the same project and therefore "pollute" the results
-    echo "Starting analysis by SonarQube..."
-    sonar-scanner -Dsonar.login=$SONAR_TOKEN
-fi
 
-# PR analysis deactivated at least until custom quality profiles can be created
-#elif [ "$TRAVIS_PULL_REQUEST" != "false" ] && [ -n "${GITHUB_TOKEN-}" ]  && [ "$BUILD_MODE" = "sonarqube" ]; then
-    # => This will analyse the PR and display found issues as comments in the PR, but it won't push results to the SonarQube server
-    #
-    # For security reasons environment variables are not available on the pull requests
-    # coming from outside repositories
-    # http://docs.travis-ci.com/user/pull-requests/#Security-Restrictions-when-testing-Pull-Requests
-    # That's why the analysis does not need to be executed if the variable GITHUB_TOKEN is not defined.
-#    echo "Starting Pull Request analysis by SonarQube..."
-#    sonar-scanner -Dsonar.login=$SONAR_TOKEN \
-#    -Dsonar.analysis.mode=preview \
-#    -Dsonar.github.oauth=$GITHUB_TOKEN \
-#    -Dsonar.github.repository=$TRAVIS_REPO_SLUG \
-#    -Dsonar.github.pullRequest=$TRAVIS_PULL_REQUEST
-#fi
-# When neither on master branch nor on a non-external pull request => nothing to do
-
-if [ "$MODULES" != "min" ] && [ "${TARGETOS:0:3}" != "ios" ] && [ "$BUILD_MODE" != "sonarqube" ]; then
-    ./botan-test
+if [ "$BUILD_MODE" = "sonarqube" ]; then
+
+    if [ "$TRAVIS_BRANCH" = "master" ] && [ "$TRAVIS_PULL_REQUEST" = "false" ]; then
+       # => This will run a full analysis of the project and push results to the SonarQube server.
+       #
+       # Analysis is done only on master so that build of branches don't push analyses to the same project and therefore "pollute" the results
+       echo "Starting analysis by SonarQube..."
+       sonar-scanner "-Dsonar.login=$SONAR_TOKEN"
+
+    # PR analysis deactivated at least until custom quality profiles can be created
+    elif false && [ "$TRAVIS_PULL_REQUEST" != "false" ] && [ -n "${GITHUB_TOKEN-}" ]; then
+        # => This will analyse the PR and display found issues as comments in the PR, but it won't push results to the SonarQube server
+        #
+        # For security reasons environment variables are not available on the pull requests
+        # coming from outside repositories
+        # http://docs.travis-ci.com/user/pull-requests/#Security-Restrictions-when-testing-Pull-Requests
+        # That's why the analysis does not need to be executed if the variable GITHUB_TOKEN is not defined.
+        echo "Starting Pull Request analysis by SonarQube..."
+        sonar-scanner -Dsonar.login=$SONAR_TOKEN \
+                      -Dsonar.analysis.mode=preview \
+                      -Dsonar.github.oauth=$GITHUB_TOKEN \
+                      -Dsonar.github.repository=$TRAVIS_REPO_SLUG \
+                      -Dsonar.github.pullRequest=$TRAVIS_PULL_REQUEST
+    fi
+       # When neither on master branch nor on a non-external pull request => nothing to do
+    fi
+
+if [ "$BUILD_MODE" == "sonarqube" ] || \
+       ( [ "${BUILD_MODE:0:5}" == "cross" ] && [ "$TRAVIS_OS_NAME" == "osx" ] ); then
+    echo "Running tests disabled on this build type"
+else
+    echo Running $TEST_PREFIX $TEST_EXE
+    time $TEST_PREFIX $TEST_EXE
 fi
 
-if [ "$MODULES" != "min" ] && [ "$BUILD_MODE" = "shared" ] && [ "$TARGETOS" = "native" ]
+# Run Python tests (need shared libs)
+if [ "$BUILD_MODE" = "shared" ]
 then
-    python2 --version
-    python3 --version
-    LD_LIBRARY_PATH=. python2 src/python/botan.py
-    LD_LIBRARY_PATH=. python3 src/python/botan.py
+    # TODO: find all things in PATH that begin with python- and execute them :)
+    for py in python2 python3
+    do
+        $py --version
+        LD_LIBRARY_PATH=. $py src/python/botan.py
+    done
 fi
 
+# Test make install
 make install
diff --git a/src/scripts/ci/travis/install.sh b/src/scripts/ci/travis/install.sh
index a9d38ed80..987438127 100755
--- a/src/scripts/ci/travis/install.sh
+++ b/src/scripts/ci/travis/install.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 set -ev
 which shellcheck > /dev/null && shellcheck "$0" # Run shellcheck on this if available
 
@@ -18,13 +18,38 @@ if [ "$BUILD_MODE" = "sonarqube" ]; then
     unzip build-wrapper-linux-x86.zip
 fi
 
+if [ "$TRAVIS_OS_NAME" = "linux" ]; then
+    if [ "$BUILD_MODE" = "valgrind" ] || [ "${BUILD_MODE:0:5}" = "cross" ]; then
+        sudo apt-get -qq update
+
+        if [ "$BUILD_MODE" = "valgrind" ]; then
+            sudo apt-get install valgrind
+        elif [ "$BUILD_MODE" = "cross-arm32" ]; then
+            sudo apt-get install g++-4.8-arm-linux-gnueabihf libc6-dev-armhf-cross qemu-user
+        elif [ "$BUILD_MODE" = "cross-arm64" ]; then
+            sudo apt-get install g++-4.8-aarch64-linux-gnu libc6-dev-arm64-cross qemu-user
+        elif [ "$BUILD_MODE" = "cross-ppc32" ]; then
+            sudo apt-get install g++-4.8-powerpc-linux-gnu libc6-dev-powerpc-cross qemu-user
+        elif [ "$BUILD_MODE" = "cross-ppc64" ]; then
+            sudo apt-get install g++-4.8-powerpc64le-linux-gnu libc6-dev-ppc64el-cross qemu-user
+        elif [ "$BUILD_MODE" = "cross-win32" ]; then
+            sudo apt-get install g++-mingw-w64-i686 mingw-w64-i686-dev
+
+            # See https://github.com/travis-ci/travis-ci/issues/6460
+            sudo dpkg --add-architecture i386
+            sudo apt-get -qq update # have to update again due to adding i386 above
+            sudo apt-get install wine
+        fi
+    fi
+fi
+
 if [ "$TRAVIS_OS_NAME" = "osx" ]; then
     # Workaround for https://github.com/Homebrew/homebrew/issues/42553
     brew update || brew update
 
     brew install ccache
 
-    if [ "$TARGETOS" = "native" ]; then
+    if [ "$BUILD_MODE" != "cross-arm32" ] && [ "$BUILD_MODE" != "cross-arm64" ]; then
         brew install xz
         brew install python # python2
         brew install python3