In compression wrappers add an overflow check before calling malloc

If malloc fails, don't save the size that was attempted. Otherwise a
failing malloc followed by a free(nullptr) would zero a block of
memory equal to the failed allocation starting from the null address.

It's not clear if zlib,bzip2,lzma expect the return of the malloc
function to be zero but LZMA at least seems to read from it before
writing. Zero it.

1 files changed, 21 insertions, 3 deletions

diff --git a/src/lib/compression/compression.cpp b/src/lib/compression/compression.cpp
index fc2c6192a..178de245f 100644
--- a/src/lib/compression/compression.cpp
+++ b/src/lib/compression/compression.cpp
@@ -14,10 +14,28 @@ namespace Botan {
 
 void* Compression_Alloc_Info::do_malloc(size_t n, size_t size)
    {
-   const size_t total_sz = n * size;
+   const size_t total_size = n * size;
+
+   BOTAN_ASSERT_EQUAL(total_size / size, n, "Overflow check");
+
+   // TODO maximum length check here?
+
+   void* ptr = std::malloc(total_size);
+
+   /*
+   * Return null rather than throwing here as we are being called by a
+   * C library and it may not be possible for an exception to unwind
+   * the call stack from here. The compression library is expecting a
+   * function written in C and a null return on error, which it will
+   * send upwards to the compression wrappers.
+   */
+
+   if(ptr)
+      {
+      std::memset(ptr, 0, total_size);
+      m_current_allocs[ptr] = total_size;
+      }
 
-   void* ptr = std::malloc(total_sz);
-   m_current_allocs[ptr] = total_sz;
    return ptr;
    }