Fix heap overflow in ECC point multiplication

If affine coordinates larger than the prime modulus were given,
a later multiplication could overflow the size of an allocated
output buffer, which was sized based on the size of the prime.

This will cause an overflow into either the system heap or if the
mlock/mmap pool allocator is in use, then into the adjacent key
material stored in the pool.

Reported by Alex Gaynor who found it with AFL

Also fix a one word overwrite in P-521 reduction. Found with AFL

1 files changed, 2 insertions, 1 deletions

diff --git a/src/lib/math/ec_gfp/curve_nistp.cpp b/src/lib/math/ec_gfp/curve_nistp.cpp
index bbc11ff21..6a98d9588 100644
--- a/src/lib/math/ec_gfp/curve_nistp.cpp
+++ b/src/lib/math/ec_gfp/curve_nistp.cpp
@@ -72,7 +72,8 @@ void redc_p521(BigInt& x, secure_vector<word>& ws)
 
    x.mask_bits(521);
 
-   bigint_add3(x.mutable_data(), x.data(), p_words, ws.data(), p_words);
+   word carry = bigint_add3_nc(x.mutable_data(), x.data(), p_words, ws.data(), p_words);
+   BOTAN_ASSERT_EQUAL(carry, 0, "Final final carry in P-521 reduction");
 
    normalize(prime_p521(), x, ws, 1);
    }