Fix heap overflow in ECC point multiplication

If affine coordinates larger than the prime modulus were given,
a later multiplication could overflow the size of an allocated
output buffer, which was sized based on the size of the prime.

This will cause an overflow into either the system heap or if the
mlock/mmap pool allocator is in use, then into the adjacent key
material stored in the pool.

Reported by Alex Gaynor who found it with AFL

Also fix a one word overwrite in P-521 reduction. Found with AFL

1 files changed, 5 insertions, 0 deletions

diff --git a/src/lib/math/mp/mp_karat.cpp b/src/lib/math/mp/mp_karat.cpp
index c7f179191..9135fdd6a 100644
--- a/src/lib/math/mp/mp_karat.cpp
+++ b/src/lib/math/mp/mp_karat.cpp
@@ -256,6 +256,9 @@ void bigint_mul(word z[], size_t z_size, word workspace[],
                 const word x[], size_t x_size, size_t x_sw,
                 const word y[], size_t y_size, size_t y_sw)
    {
+   // checking that z_size >= x_sw + y_sw without overflow
+   BOTAN_ASSERT(z_size > x_sw && z_size > y_sw && z_size-x_sw >= y_sw, "Output size is sufficient");
+
    if(x_sw == 1)
       {
       bigint_linmul3(z, y, y_sw, x[0]);
@@ -312,6 +315,8 @@ void bigint_mul(word z[], size_t z_size, word workspace[],
 void bigint_sqr(word z[], size_t z_size, word workspace[],
                 const word x[], size_t x_size, size_t x_sw)
    {
+   BOTAN_ASSERT(z_size/2 >= x_sw, "Output size is sufficient");
+
    if(x_sw == 1)
       {
       bigint_linmul3(z, x, x_sw, x[0]);