propagate from branch 'net.randombit.botan' (head dd33b7150f3f49d795e4eb962d8d41d1ada58d8d)

            to branch 'net.randombit.botan.ssl' (head 1452205423962b664263fd2a35149122dfc94d37)

9 files changed, 172 insertions, 114 deletions

diff --git a/doc/api.tex b/doc/api.tex
index 556e76aa0..dc920d07b 100644
--- a/doc/api.tex
+++ b/doc/api.tex
@@ -12,7 +12,7 @@
 
 \title{\textbf{Botan API Reference}}
 \author{}
-\date{2009/2/19}
+\date{2010/02/05}
 
 \newcommand{\filename}[1]{\texttt{#1}}
 \newcommand{\manpage}[2]{\texttt{#1}(#2)}
@@ -2621,53 +2621,124 @@ degree of applicability.
 
 \subsection{S2K Algorithms}
 
-There are various procedures (usually fairly ad-hoc) for turning a passphrase
-into a (mostly) arbitrary length key for a symmetric cipher. A general
-interface for such algorithms is presented in \filename{s2k.h}. The main
-function is \function{derive\_key}, which takes a passphrase, and the desired
-length of the output key, and returns a key of that length, deterministically
-produced from the passphrase. If an algorithm can't produce a key of that size,
-it will throw an exception (most notably, PKCS \#5's PBKDF1 can only produce
-strings between 1 and $n$ bytes, where $n$ is the output size of the underlying
-hash function).
-
-Most such algorithms allow the use of a ``salt'', which provides some extra
-randomness and helps against dictionary attacks on the passphrase. Simply call
-\function{change\_salt} (there are variations of it for most of the ways you
-might wish to specify a salt, check the header for details) with a block of
-random data. You can also have the class generate a new salt for you with
-\function{new\_random\_salt}; the salt that was generated can be retrieved with
-\function{current\_salt}.
-
-Additionally some algorithms allow you to set some sort of iteration
-count, which will make the algorithm take longer to compute the final
-key (reducing the speed of brute-force attacks of various kinds). This
-can be changed with the \function{set\_iterations} function. Most
-standards recommend an iteration count of at least 1000. Currently
-defined S2K algorithms are ``PBKDF1(digest)'', ``PBKDF2(digest)'', and
-``OpenPGP-S2K(digest)''; you can retrieve any of these using the
-\function{get\_s2k}, found in \filename{lookup.h}. As of this writing,
-``PBKDF2(SHA-256)'' with 10000 iterations and an 8 byte salt is
-recommend for new applications.
+There are various procedures (usually fairly ad-hoc) for turning a
+passphrase into a (mostly) arbitrary length key for a symmetric
+cipher. A general interface for such algorithms is presented in
+\filename{s2k.h}. The main function is \function{derive\_key}, which
+takes a passphrase, a salt, an iteration count, and the desired length
+of the output key, and returns a key of that length, deterministically
+produced from the passphrase and salt. If an algorithm can't produce a
+key of that size, it will throw an exception (most notably, PKCS \#5's
+PBKDF1 can only produce strings between 1 and $n$ bytes, where $n$ is
+the output size of the underlying hash function).
+
+The purpose of the iteration count is to make the algorithm take
+longer to compute the final key (reducing the speed of brute-force
+attacks of various kinds). Most standards recommend an iteration count
+of at least 10000. Currently defined S2K algorithms are
+``PBKDF1(digest)'', ``PBKDF2(digest)'', and ``OpenPGP-S2K(digest)'';
+you can retrieve any of these using the \function{get\_s2k}, found in
+\filename{lookup.h}. As of this writing, ``PBKDF2(SHA-256)'' with
+10000 iterations and a 16 byte salt is recommend for new applications.
 
 \subsubsection{OpenPGP S2K}
 
-There are some oddities about OpenPGP's S2K algorithms that are documented
-here. For one thing, it uses the iteration count in a strange manner; instead
-of specifying how many times to iterate the hash, it tells how many
-\emph{bytes} should be hashed in total (including the salt). So the exact
-iteration count will depend on the size of the salt (which is fixed at 8 bytes
-by the OpenPGP standard, though the implementation will allow any salt size)
-and the size of the passphrase.
-
-To get what OpenPGP calls ``Simple S2K'', set iterations to 0 (the default for
-OpenPGP S2K), and do not specify a salt. To get ``Salted S2K'', again leave the
-iteration count at 0, but give an 8-byte salt. ``Salted and Iterated S2K''
-requires an 8-byte salt and some iteration count (this should be significantly
-larger than the size of the longest passphrase that might reasonably be used;
-somewhere from 1024 to 65536 would probably be about right). Using both a
-reasonably sized salt and a large iteration count is highly recommended to
-prevent password guessing attempts.
+There are some oddities about OpenPGP's S2K algorithms that are
+documented here. For one thing, it uses the iteration count in a
+strange manner; instead of specifying how many times to iterate the
+hash, it tells how many \emph{bytes} should be hashed in total
+(including the salt). So the exact iteration count will depend on the
+size of the salt (which is fixed at 8 bytes by the OpenPGP standard,
+though the implementation will allow any salt size) and the size of
+the passphrase.
+
+To get what OpenPGP calls ``Simple S2K'', set iterations to 0, and do
+not specify a salt. To get ``Salted S2K'', again leave the iteration
+count at 0, but give an 8-byte salt. ``Salted and Iterated S2K''
+requires an 8-byte salt and some iteration count (this should be
+significantly larger than the size of the longest passphrase that
+might reasonably be used; somewhere from 1024 to 65536 would probably
+be about right). Using both a reasonably sized salt and a large
+iteration count is highly recommended to prevent password guessing
+attempts.
+
+\subsection{Password Hashing}
+
+Storing passwords for user authentication purposes in plaintext is the
+simplest but least secure method; when an attacker compromises the
+database in which the passwords are stored, they immediately gain
+access to all of them. Often passwords are reused among multiple
+services or machines, meaning once a password to a single service is
+known an attacker has a substantial head start on attacking other
+machines.
+
+The general approach is to store, instead of the password, the output
+of a one way function of the password. Upon receiving an
+authentication request, the authenticator can recompute the one way
+function and compare the value just computed with the one that was
+stored. If they match, then the authentication request succeeds. But
+when an attacker gains access to the database, they only have the
+output of the one way function, not the original password.
+
+Common hash functions such as SHA-256 are one way, but used alone they
+have problems for this purpose. What an attacker can do, upon gaining
+access to such a stored password database, is hash common dictionary
+words and other possible passwords, storing them in a list. Then he
+can search through his list; if a stored hash and an entry in his list
+match, then he has found the password. Even worse, this can happen
+\emph{offline}: an attacker can begin hashing common passwords days,
+months, or years before ever gaining access to the database. In
+addition, if two users choose the same password, the one way function
+output will be the same for both of them, which will be visible upon
+inspection of the database.
+
+There are two solutions to these problems: salting and
+iteration. Salting refers to including, along with the password, a
+randomly chosen value which perturbs the one way function. Salting can
+reduce the effectivness of offline dictionary generation (because for
+each potential password, an attacker would have to compute the one way
+function output for all possible salts - with a large enough salt,
+this can make the problem quite difficult). It also prevents the same
+password from producing the same output, as long as the salts do not
+collide. With a large salt (say 80 to 128 bits) this will be quite
+unlikely. Iteration refers to the general technique of forcing
+multiple one way function evaluations when computing the output, to
+slow down the operation. For instance if hashing a single password
+requires running SHA-256 100,000 times instead of just once, that will
+slow down user authentication by a factor of 100,000, but user
+authentication happens quite rarely, and usually there are more
+expensive operations that need to occur anyway (network and database
+I/O, etc). On the other hand, an attacker who is attempting to break a
+database full of stolen password hashes will be seriously
+inconvenienced by a factor of 100,000 slowdown; they will be able to
+only test at a rate of .0001\% of what they would without iterations
+(or, equivalently, will require 100,000 times as many zombie botnet
+hosts).
+
+There are many different ways of doing this password hashing
+operation, with common ones including Unix's crypt (which is based on
+DES) and OpenBSD's bcrypt (based on Blowfish). Other variants using
+MD5 or SHA-256 are also in use on various systems.
+
+Botan provides a technique called passhash9, in
+\filename{passhash9.h}, which is based on PBKDF2. Two functions are
+provided in this header, \function{generate\_passhash9} and
+\function{check\_passhash9}.  The generate function takes the password
+to hash, a \type{RandomNumberGenerator}, and a work factor, which
+tells how many iterations to compute. The default work factor is 10
+(which means 100,000 iterations), but any non-zero value is accepted.
+The check function takes a password and a passhash9 output and checks
+if the password is the same as the one that was used to generate the
+passhash9 output, returning a boolean true (same) or false (not same).
+An example can be found in \filename{doc/examples/passhash.cpp}.
+
+Passhash9 currently uses HMAC(SHA-1) for the underlying PBKDF2
+psuedo-random function, but can be extended to use different
+algorithms in the future if necessary. For instance using a PRF based
+on Blowfish (a block cipher that requires 4 KiB of RAM for efficient
+execution) could be used to make hardware-based password cracking more
+expensive (this was one motivation for Blowfish's use in the bcrypt
+hashing scheme, in fact).
 
 \subsection{Checksums}
 
diff --git a/doc/examples/cpuid.cpp b/doc/examples/cpuid.cpp
index ba499051f..8adc8be6c 100644
--- a/doc/examples/cpuid.cpp
+++ b/doc/examples/cpuid.cpp
@@ -28,7 +28,6 @@ int main()
    print_if_feature("SSE4.2", CPUID::has_sse42());
 
    print_if_feature("AES-NI", CPUID::has_aes_intel());
-   print_if_feature("AES-VIA", CPUID::has_aes_via());
 
    print_if_feature("AltiVec", CPUID::has_altivec());
    }
diff --git a/doc/examples/decrypt.cpp b/doc/examples/decrypt.cpp
index ebab5d804..de261b5f3 100644
--- a/doc/examples/decrypt.cpp
+++ b/doc/examples/decrypt.cpp
@@ -106,12 +106,22 @@ int main(int argc, char* argv[])
       const u32bit iv_len = block_size_of(algo);
 
       std::auto_ptr<S2K> s2k(get_s2k("PBKDF2(SHA-1)"));
-      s2k->set_iterations(8192);
-      s2k->change_salt(b64_decode(salt_str));
 
-      SymmetricKey bc_key = s2k->derive_key(key_len, "BLK" + passphrase);
-      InitializationVector iv = s2k->derive_key(iv_len, "IVL" + passphrase);
-      SymmetricKey mac_key = s2k->derive_key(16, "MAC" + passphrase);
+      const u32bit PBKDF2_ITERATIONS = 8192;
+
+      SecureVector<byte> salt = b64_decode(salt_str);
+
+      SymmetricKey bc_key = s2k->derive_key(key_len, "BLK" + passphrase,
+                                            &salt[0], salt.size(),
+                                            PBKDF2_ITERATIONS);
+
+      InitializationVector iv = s2k->derive_key(iv_len, "IVL" + passphrase,
+                                                &salt[0], salt.size(),
+                                                PBKDF2_ITERATIONS);
+
+      SymmetricKey mac_key = s2k->derive_key(16, "MAC" + passphrase,
+                                             &salt[0], salt.size(),
+                                             PBKDF2_ITERATIONS);
 
       Pipe pipe(new Base64_Decoder,
                 get_cipher(algo + "/CBC", bc_key, iv, DECRYPTION),
diff --git a/doc/examples/encrypt.cpp b/doc/examples/encrypt.cpp
index f903c2f24..4999fa086 100644
--- a/doc/examples/encrypt.cpp
+++ b/doc/examples/encrypt.cpp
@@ -125,17 +125,26 @@ int main(int argc, char* argv[])
       AutoSeeded_RNG rng;
 
       std::auto_ptr<S2K> s2k(get_s2k("PBKDF2(SHA-1)"));
-      s2k->set_iterations(8192);
-      s2k->new_random_salt(rng, 8);
 
-      SymmetricKey bc_key = s2k->derive_key(key_len, "BLK" + passphrase);
-      InitializationVector iv = s2k->derive_key(iv_len, "IVL" + passphrase);
-      SymmetricKey mac_key = s2k->derive_key(16, "MAC" + passphrase);
+      SecureVector<byte> salt(8);
+      rng.randomize(&salt[0], salt.size());
+
+      const u32bit PBKDF2_ITERATIONS = 8192;
+
+      SymmetricKey bc_key = s2k->derive_key(key_len, "BLK" + passphrase,
+                                            &salt[0], salt.size(),
+                                            PBKDF2_ITERATIONS);
+      InitializationVector iv = s2k->derive_key(iv_len, "IVL" + passphrase,
+                                                &salt[0], salt.size(),
+                                                PBKDF2_ITERATIONS);
+      SymmetricKey mac_key = s2k->derive_key(16, "MAC" + passphrase,
+                                             &salt[0], salt.size(),
+                                             PBKDF2_ITERATIONS);
 
       // Just to be all fancy we even write a (simple) header.
       out << "-------- ENCRYPTED FILE --------" << std::endl;
       out << algo << std::endl;
-      out << b64_encode(s2k->current_salt()) << std::endl;
+      out << b64_encode(salt) << std::endl;
 
       Pipe pipe(new Fork(
                    new Chain(new MAC_Filter("HMAC(SHA-1)", mac_key),
diff --git a/doc/examples/encrypt2.cpp b/doc/examples/encrypt2.cpp
index dac2f8314..41f4fb478 100644
--- a/doc/examples/encrypt2.cpp
+++ b/doc/examples/encrypt2.cpp
@@ -26,16 +26,21 @@ int main()
 
    PKCS5_PBKDF2 pbkdf2(new HMAC(new SHA_160));
 
-   pbkdf2.set_iterations(4096);
-   pbkdf2.new_random_salt(rng, 8);
-   SecureVector<byte> the_salt = pbkdf2.current_salt();
+   const u32bit PBKDF2_ITERATIONS = 8192;
 
-   SecureVector<byte> master_key = pbkdf2.derive_key(48, passphrase).bits_of();
+   SecureVector<byte> salt(8);
+   rng.randomize(&salt[0], salt.size());
+
+   SecureVector<byte> master_key = pbkdf2.derive_key(48, passphrase,
+                                                     &salt[0], salt.size(),
+                                                     PBKDF2_ITERATIONS).bits_of();
 
    KDF* kdf = get_kdf("KDF2(SHA-1)");
 
    SymmetricKey key = kdf->derive_key(20, master_key, "cipher key");
+
    SymmetricKey mac_key = kdf->derive_key(20, master_key, "hmac key");
+
    InitializationVector iv = kdf->derive_key(8, master_key, "cipher iv");
 
    Pipe pipe(new Fork(
@@ -50,7 +55,7 @@ int main()
                 )
       );
 
-   outfile.write((const char*)the_salt.begin(), the_salt.size());
+   outfile.write((const char*)salt.begin(), salt.size());
 
    pipe.start_msg();
    infile >> pipe;
diff --git a/doc/examples/passhash.cpp b/doc/examples/passhash.cpp
index 24f7ff674..586c28c3f 100644
--- a/doc/examples/passhash.cpp
+++ b/doc/examples/passhash.cpp
@@ -5,17 +5,11 @@
 */
 
 #include <botan/botan.h>
-#include <botan/pbkdf2.h>
-#include <botan/hmac.h>
-#include <botan/sha160.h>
+#include <botan/passhash9.h>
 #include <iostream>
 #include <memory>
 
-using namespace Botan;
-
-std::string password_hash(const std::string& pass,
-                          RandomNumberGenerator& rng);
-bool password_hash_ok(const std::string& pass, const std::string& hash);
+#include <stdio.h>
 
 int main(int argc, char* argv[])
    {
@@ -33,14 +27,14 @@ int main(int argc, char* argv[])
 
       if(argc == 2)
          {
-         AutoSeeded_RNG rng;
+         Botan::AutoSeeded_RNG rng;
 
          std::cout << "H('" << argv[1] << "') = "
-                   << password_hash(argv[1], rng) << '\n';
+                   << Botan::generate_passhash9(argv[1], rng) << '\n';
          }
       else
          {
-         bool ok = password_hash_ok(argv[1], argv[2]);
+         bool ok = Botan::check_passhash9(argv[1], argv[2]);
          if(ok)
             std::cout << "Password and hash match\n";
          else
@@ -54,39 +48,3 @@ int main(int argc, char* argv[])
       }
    return 0;
    }
-
-std::string password_hash(const std::string& pass,
-                          RandomNumberGenerator& rng)
-   {
-   PKCS5_PBKDF2 kdf(new HMAC(new SHA_160));
-
-   kdf.set_iterations(10000);
-   kdf.new_random_salt(rng, 6); // 48 bits
-
-   Pipe pipe(new Base64_Encoder);
-   pipe.start_msg();
-   pipe.write(kdf.current_salt());
-   pipe.write(kdf.derive_key(12, pass).bits_of());
-   pipe.end_msg();
-
-   return pipe.read_all_as_string();
-   }
-
-bool password_hash_ok(const std::string& pass, const std::string& hash)
-   {
-   Pipe pipe(new Base64_Decoder);
-   pipe.start_msg();
-   pipe.write(hash);
-   pipe.end_msg();
-
-   SecureVector<byte> hash_bin = pipe.read_all();
-
-   PKCS5_PBKDF2 kdf(new HMAC(new SHA_160));
-
-   kdf.set_iterations(10000);
-   kdf.change_salt(hash_bin, 6);
-
-   SecureVector<byte> cmp = kdf.derive_key(12, pass).bits_of();
-
-   return same_mem(cmp.begin(), hash_bin.begin() + 6, 12);
-   }
diff --git a/doc/examples/row_encryptor.cpp b/doc/examples/row_encryptor.cpp
index 17f44ce7b..7c234105d 100644
--- a/doc/examples/row_encryptor.cpp
+++ b/doc/examples/row_encryptor.cpp
@@ -66,10 +66,9 @@ void Row_Encryptor::init(const std::string& passphrase)
    {
    std::auto_ptr<S2K> s2k(get_s2k("PBKDF2(SHA-160)"));
 
-   s2k->set_iterations(10000);
-   s2k->change_salt(&s2k_salt[0], s2k_salt.size());
-
-   SecureVector<byte> key = s2k->derive_key(32, passphrase).bits_of();
+   SecureVector<byte> key = s2k->derive_key(32, passphrase,
+                                            &s2k_salt[0], s2k_salt.size(),
+                                            10000).bits_of();
 
    /*
     Save pointers to the EAX objects so we can change the IV as needed
diff --git a/doc/license.txt b/doc/license.txt
index 9d1067cc1..599bf7f63 100644
--- a/doc/license.txt
+++ b/doc/license.txt
@@ -1,6 +1,6 @@
 Botan (http://botan.randombit.net/) is distributed under these terms:
 
-Copyright (C) 1999-2009 Jack Lloyd
+Copyright (C) 1999-2010 Jack Lloyd
               2001 Peter J Jones
               2004-2007 Justin Karneges
               2005 Matthew Gregan
@@ -14,6 +14,7 @@ Copyright (C) 1999-2009 Jack Lloyd
               2007 Manuel Hartl
               2007 Christoph Ludwig
               2007 Patrick Sona
+              2010 Olivier de Gaalon
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
diff --git a/doc/log.txt b/doc/log.txt
index c729179b8..847bf8596 100644
--- a/doc/log.txt
+++ b/doc/log.txt
@@ -1,10 +1,16 @@
 
 * 1.9.4-dev, ????-??-??
+ - Add SIMD implementation of Noekeon
  - Add SSE2 implementation of IDEA
  - Perform XTS encryption and decryption in parallel where possible
  - Perform CBC decryption in parallel where possible
+ - Add SQLite3 db encryption codec, contributed by Olivier de Gaalon
  - Add a block cipher cascade construction
+ - Add support for password hashing for authentication (passhash9.h)
  - Add support for Win32 high resolution system timers
+ - Changed S2K interface: derive_key now takes salt, iteration count
+ - Fix crash in GMP_Engine if library is shutdown and reinitialized
+ - Fix an invalid memory read in MD4
  - Remove Timer class entirely
  - Switch default PKCS #8 encryption algorithm from 3DES to AES-256
  - New option --gen-amalgamation for creating a SQLite-style amalgamation