Merge master into this branch, resolving conflicts with #457/#576

which recently landed on master.

10 files changed, 443 insertions, 176 deletions

diff --git a/doc/contributing.rst b/doc/contributing.rst
index a0b8daec1..64b609af5 100644
--- a/doc/contributing.rst
+++ b/doc/contributing.rst
@@ -38,7 +38,6 @@ Library Layout
 * ``entropy`` has various entropy sources
 * ``asn1`` is the DER encoder/decoder
 * ``cert/x509`` is X.509 certificates, PKCS #10 requests, OCSP
-* ``cert/cvc`` is Card Verifiable Certificates (ePassport credentials)
 * ``tls`` contains the TLS implementation
 * ``filters`` is a filter/pipe API for data transforms
 * ``compression`` has the compression wrappers (zlib, bzip2, lzma)
@@ -113,9 +112,11 @@ Sending patches
 ========================================
 
 All contributions should be submitted as pull requests via GitHub
-(https://github.com/randombit/botan). If you are planning a large change email
-the mailing list or open a discussion ticket on github before starting out to
-make sure you are on the right path.
+(https://github.com/randombit/botan). If you are planning a large
+change email the mailing list or open a discussion ticket on github
+before starting out to make sure you are on the right path. And once
+you have something written, free to open a [WIP] PR for early review
+and comment.
 
 If possible please sign your git commits using a PGP key.
 See https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work for
@@ -137,6 +138,21 @@ Also, try building and testing it on whatever hardware you have handy,
 especially non-x86 platforms, or especially C++11 compilers other than the
 regularly tested GCC, Clang, and Visual Studio compilers.
 
+Git Usage
+========================================
+
+Do *NOT* merge ``master`` into your topic branch, this creates
+needless commits and noise in history. Instead, as needed, rebase your
+branch against master (``git rebase -i master``) and force push the
+branch to update the PR. If the GitHub PR page does not report any
+merge conflicts and nobody asks you to rebase, you don't need to
+rebase.
+
+Try to keep your history clean and use rebase to squash your commits
+as needed. If your diff is less than roughly 100 lines, it should
+probably be a single commit. Only split commits as needed to help with
+review/understanding of the change.
+
 External Dependencies
 ========================================
 
diff --git a/doc/credits.rst b/doc/credits.rst
index 6d62b6380..290067491 100644
--- a/doc/credits.rst
+++ b/doc/credits.rst
@@ -15,6 +15,13 @@ snail-mail address (S), and Bitcoin address (B).
   D: documentation editing
   S: Oregon, USA
 
+  N: Simon Cogliani
+  E: [email protected]
+  W: https://www.tanker.io/
+  P: EA73 D0AF 5A81 A61A 8931  C2CA C9AB F2E4 3820 4F25
+  D: Getting keystream of ChaCha
+  S: Paris, France
+
   N: Martin Doering
   E: [email protected]
   D: GF(p) arithmetic
@@ -60,6 +67,13 @@ snail-mail address (S), and Bitcoin address (B).
   D: Locking in Algo_Registry for Windows OS
   S: Slovenia
 
+  N: René Korthaus
+  E: [email protected]
+  W: https://www.sirrix.com
+  P: C196 FF9D 3DDC A5E7 F98C E745 9AD0 F9FA 587E 74D6
+  D: CI, ECGDSA, ECKCDSA
+  S: Bochum, Germany
+
   N: Adam Langley
   E: [email protected]
   D: Curve25519
@@ -104,3 +118,15 @@ snail-mail address (S), and Bitcoin address (B).
   W: https://www.kullo.net
   D: Build system
   S: Germany
+
+  N: Philipp Weber
+  E: [email protected]
+  W: https://sirrix.com/
+  D: KDF1-18033, ECIES
+  S: Saarland, Germany
+  
+  N: Daniel Neus
+  E: [email protected]
+  W: https://sirrix.com/
+  D: CI, PKCS#11, RdSeed, BSI module policy
+  S: Bochum, Germany
diff --git a/doc/deprecated.txt b/doc/deprecated.txt
new file mode 100644
index 000000000..d62954d1d
--- /dev/null
+++ b/doc/deprecated.txt
@@ -0,0 +1,34 @@
+Currently deprecated:
+
+- PRNGs X9.31 (no longer approved) and HMAC_RNG (non-standard)
+  Use HMAC_DRBG
+
+- ECB mode for block ciphers
+
+- Rabin-Williams signatures
+
+- Nyberg-Rueppel signatures
+
+- MARS, RC2, RC5, RC6, SAFER, TEA
+
+- ECB Cipher_Mode
+
+- MD2, HAS-160, RIPEMD-128
+
+- 3DES and SEED ciphersuites in TLS
+
+- DSA auth in TLS (not ECDSA)
+
+- Old (Google specific) ChaCha20 TLS ciphersuites
+
+- All built in ECC groups < 256 bits
+
+- All built in MODP groups < 2048 bits
+
+- All pre-created DSA groups
+
+- All support for BeOS/Haiku including BeOS entropy source
+
+- EGD entropy source
+
+- Unix process exec entropy source
diff --git a/doc/license.txt b/doc/license.txt
index f292a0d59..88b16651c 100644
--- a/doc/license.txt
+++ b/doc/license.txt
@@ -29,6 +29,8 @@ Copyright (C) 1999-2013,2014,2015,2016 Jack Lloyd
               2015,2016 Daniel Neus
               2015 Uri Blumenthal
               2015,2016 Kai Michaelis
+              2016 Simon Cogliani
+              2015,2016 Rohde & Schwarz Cybersecurity
               2016 Juraj Somorovsky
               2016 Christian Mainka
 All rights reserved.
diff --git a/doc/manual/rng.rst b/doc/manual/rng.rst
index 300570c3a..7eb229a5e 100644
--- a/doc/manual/rng.rst
+++ b/doc/manual/rng.rst
@@ -3,108 +3,77 @@
 Random Number Generators
 ========================================
 
-The random number generators provided in Botan are meant for creating
-keys, IVs, padding, nonces, and anything else that requires 'random'
-data. It is important to remember that the output of these classes
-will vary, even if they are supplied with the same seed (ie, two
-``Randpool`` objects with similar initial states will not produce the
-same output, because the value of high resolution timers is added to
-the state at various points).
-
-To create a random number generator, instantiate a ``AutoSeeded_RNG``
-object. This object will handle choosing the right algorithms from the
-set of enabled ones and doing seeding using OS specific
-routines. The main service a RandomNumberGenerator provides is, of
-course, random numbers:
+The base class ``RandomNumberGenerator`` is in the header ``botan/rng.h``.
 
-.. cpp:function:: byte RandomNumberGenerator::next_byte()
+The major interfaces are
+
+.. cpp:function:: void RandomNumberGenerator::randomize(byte* output_array, size_t length)
 
-  Generates a single random byte and returns it
+  Places *length* random bytes into the provided buffer.
 
-.. cpp:function:: void RandomNumberGenerator::randomize(byte* data, size_t length)
+.. cpp:function:: void RandomNumberGenerator::add_entropy(const byte* data, size_t length)
 
-  Places *length* bytes into the array pointed to by *data*
+  Incorporates provided data into the state of the PRNG, if at all
+  possible.  This works for most RNG types, including the system and
+  TPM RNGs. But if the RNG doesn't support this operation, the data is
+  dropped, no error is indicated.
 
-To ensure good quality output, a PRNG needs to be seeded with truly
-random data. Normally this is done for you. However it may happen that
-your application has access to data that is potentially unpredictable
-to an attacker. If so, use
+.. cpp:function:: void RandomNumberGenerator::randomize_with_input(byte* data, size_t length, \
+    const byte* ad, size_t ad_len)
 
-.. cpp:function:: void RandomNumberGenerator::add_entropy(const byte* data, \
-                                                          size_t length)
+  Like randomize, but first incorporates the additional input field
+  into the state of the RNG. The additional input could be anything which
+  parameterizes this request.
 
-which incorporates the data into the current randomness state. Don't
-worry about filtering the data or doing any kind of cryptographic
-preprocessing (such as hashing); the RNG objects in botan are designed
-such that you can feed them any arbitrary non-random or even
-maliciously chosen data - as long as at some point some of the seed
-data was good the output will be secure.
+.. cpp:function:: byte RandomNumberGenerator::next_byte()
 
+  Generates a single random byte and returns it. Note that calling this
+  function several times is much slower than calling ``randomize`` once
+  to produce multiple bytes at a time.
 
-Implementation Notes
+RNG Types
 ----------------------------------------
 
-Randpool
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The following RNG types are included
+
+HMAC_DRBG
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+HMAC DRBG is a random number generator designed by NIST and specified
+in SP 800-90A. It can be instantiated with any hash function but is
+typically used with SHA-256, SHA-384, or SHA-512.
 
-``Randpool`` is the primary PRNG within Botan. In recent versions all
-uses of it have been wrapped by an implementation of the X9.31 PRNG
-(see below). If for some reason you should have cause to create a PRNG
-instead of using the "global" one owned by the library, it would be
-wise to consider the same on the grounds of general caution; while
-``Randpool`` is designed with known attacks and PRNG weaknesses in
-mind, it is not an standard/official PRNG. The remainder of this
-section is a (fairly technical, though high-level) description of the
-algorithms used in this PRNG. Unless you have a specific interest in
-this subject, the rest of this section might prove somewhat
-uninteresting.
-
-``Randpool`` has an internal state called pool, which is 512 bytes
-long. This is where entropy is mixed into and extracted from. There is also a
-small output buffer (called buffer), which holds the data which has already
-been generated but has just not been output yet.
-
-It is based around a MAC and a block cipher (which are currently
-HMAC(SHA-256) and AES-256). Where a specific size is mentioned, it
-should be taken as a multiple of the cipher's block size. For example,
-if a 256-bit block cipher were used instead of AES, all the sizes
-internally would double. Every time some new output is needed, we
-compute the MAC of a counter and a high resolution timer. The
-resulting MAC is XORed into the output buffer (wrapping as needed),
-and the output buffer is then encrypted with AES, producing 16 bytes
-of output.
-
-After 8 blocks (or 128 bytes) have been produced, we mix the pool. To
-do this, we first rekey both the MAC and the cipher; the new MAC key
-is the MAC of the current pool under the old MAC key, while the new
-cipher key is the MAC of the current pool under the just-chosen MAC
-key. We then encrypt the entire pool in CBC mode, using the current
-(unused) output buffer as the IV. We then generate a new output
-buffer, using the mechanism described in the previous paragraph.
-
-To add randomness to the PRNG, we compute the MAC of the input and XOR
-the output into the start of the pool. Then we remix the pool and
-produce a new output buffer. The initial MAC operation should make it
-very hard for chosen inputs to harm the security of ``Randpool``, and
-as HMAC should be able to hold roughly 256 bits of state, it is
-unlikely that we are wasting much input entropy (or, if we are, it
-doesn't matter, because we have a very abundant supply).
+HMAC DRBG seems to be the most conservative generator of the NIST
+approved options.
+
+System_RNG
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In ``system_rng.h``, objects of ``System_RNG`` reference a single
+(process global) reference to the system PRNG (/dev/urandom or
+CryptGenRandom).
+
+AutoSeeded_RNG
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+This instantiates a new instance of a userspace PRNG, seeds it with
+a default entropy pool.
 
 ANSI X9.31
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+This generator is deprecated and will be removed in a future release.
+
 ``ANSI_X931_PRNG`` is the standard issue X9.31 Appendix A.2.4 PRNG,
 though using AES-256 instead of 3DES as the block cipher. This PRNG
 implementation has been checked against official X9.31 test vectors.
 
-Internally, the PRNG holds a pointer to another PRNG (typically
-Randpool). This internal PRNG generates the key and seed used by the
-X9.31 algorithm, as well as the date/time vectors. Each time an X9.31
-PRNG object receives entropy, it passes it along to the PRNG it is
-holding, and then pulls out some random bits to generate a new key and
-seed. This PRNG considers itself seeded as soon as the internal PRNG
-is seeded.
-
+Internally, the PRNG holds a pointer to another RNG object. This
+internal PRNG generates the key and seed used by the X9.31 algorithm,
+as well as the date/time vectors. Each time an X9.31 PRNG object
+receives entropy, it passes it along to the PRNG it is holding, and
+then pulls out some random bits to generate a new key and seed. This
+PRNG considers itself seeded as soon as the internal PRNG is seeded.
 
 Entropy Sources
 ---------------------------------
diff --git a/doc/manual/srp.rst b/doc/manual/srp.rst
index e3aace5ff..74b67d890 100644
--- a/doc/manual/srp.rst
+++ b/doc/manual/srp.rst
@@ -10,7 +10,7 @@ This verifier is based on a password, but the password cannot be
 easily derived from the verifier. Later, the client and server can
 perform an SRP exchange, in which
 
- .. warning::
+.. warning::
 
      While knowledge of the verifier does not easily allow an attacker
      to get the raw password, they could still use the verifier to
diff --git a/doc/manual/tls.rst b/doc/manual/tls.rst
index a10a4280c..6c1ca42f2 100644
--- a/doc/manual/tls.rst
+++ b/doc/manual/tls.rst
@@ -22,44 +22,62 @@ instance, by reading from a network socket) it passes that information
 to TLS using :cpp:func:`TLS::Channel::received_data`. If the data
 passed in results in some change in the state, such as a handshake
 completing, or some data or an alert being received from the other
-side, then a user provided callback will be invoked. If the reader is
-familiar with OpenSSL's BIO layer, it might be analagous to saying the
-only way of interacting with Botan's TLS is via a `BIO_mem` I/O
+side, then the appropriate user provided callback will be invoked.
+
+If the reader is familiar with OpenSSL's BIO layer, it might be analagous
+to saying the only way of interacting with Botan's TLS is via a `BIO_mem` I/O
 abstraction. This makes the library completely agnostic to how you
 write your network layer, be it blocking sockets, libevent, asio, a
-message queue, etc.
+message queue, lwIP on RTOS, some carrier pidgeons, etc.
 
-The callbacks for TLS have the signatures
+Starting in 1.11.31, the application callbacks are encapsulated as the class
+``TLS::Callbacks`` with the following members. The first four (``tls_emit_data``,
+``tls_record_received``, ``tls_alert``, and ``tls_session_established``) are
+mandatory for using TLS, all others are optional and provide additional
+information about the connection.
 
- .. cpp:function:: void output_fn(const byte data[], size_t data_len)
+ .. cpp:function:: void tls_emit_data(const byte data[], size_t data_len)
 
-    TLS requests that all bytes of *data* be queued up to send to the
-    counterparty. After this function returns, *data* will be
-    overwritten, so a copy of the input must be made if the callback
+    Mandatory. The TLS stack requests that all bytes of *data* be queued up to send to the
+    counterparty. After this function returns, the buffer containing *data* will
+    be overwritten, so a copy of the input must be made if the callback
     cannot send the data immediately.
 
- .. cpp:function:: void data_cb(const byte data[], size_t data_len)
+    As an example you could ``send`` to perform a blocking write on a socket,
+    or append the data to a queue managed by your application, and initiate
+    an asyncronous write.
+
+    For TLS all writes must occur *in the order requested*.
+    For DTLS this ordering is not strictly required, but is still recommended.
+
+ .. cpp:function:: void tls_record_received(uint64_t rec_no, const byte data[], size_t data_len)
+
+    Mandatory. Called once for each application_data record which is received, with the
+    matching (TLS level) record sequence number.
+
+    Currently empty records are ignored and do not instigate a callback,
+    but this may change in a future release.
+
+     As with ``tls_emit_data``, the array will be overwritten sometime after
+     the callback returns, so a copy should be made if needed.
 
-     Called whenever application data is received from the other side
-     of the connection, in which case *data* and *data_len* specify
-     the data received. This array will be overwritten sometime after
-     the callback returns, so again a copy should be made if need be.
+     For TLS the record number will always increase.
 
- .. cpp:function:: void alert_cb(Alert alert, const byte data[], size_t data_len)
+     For DTLS, it is possible to receive records with the `rec_no` field
+     field out of order or repeated. It is even possible (from a malicious or
+     faulty peer) to receive multiple copies of a single record with differing plaintexts.
 
-     Called when an alert is received. Normally, data is null and
-     data_len is 0, as most alerts have no associated data. However,
-     if TLS heartbeats (see :rfc:`6520`) were negotiated, and we
-     initiated a heartbeat, then if/when the other party responds,
-     ``alert_cb`` will be called with whatever data was included in
-     the heartbeat response (if any) along with a psuedo-alert value
-     of ``HEARTBEAT_PAYLOAD``.
+ .. cpp:function:: void tls_alert(Alert alert) 
 
- .. cpp:function:: bool handshake_cb(const TLS::Session& session)
+     Mandatory. Called when an alert is received from the peer. Note that alerts
+     received before the handshake is complete are not authenticated and
+     could have been inserted by a MITM attacker.
+     
+ .. cpp:function:: bool tls_session_established(const TLS::Session& session)
 
-     Called whenever a negotiation completes. This can happen more
-     than once on any connection. The *session* parameter provides
-     information about the session which was established.
+     Mandatory. Called whenever a negotiation completes. This can happen more
+     than once on any connection, if renegotiation occurs. The *session* parameter
+     provides information about the session which was just established.
 
      If this function returns false, the session will not be cached
      for later resumption.
@@ -68,8 +86,29 @@ The callbacks for TLS have the signatures
      exception which will send a close message to the counterparty and
      reset the connection state.
 
-You can of course use tools like ``std::bind`` to bind additional
-parameters to your callback functions.
+ .. cpp:function:: std::string tls_server_choose_app_protocol(const std::vector<std::string>& client_protos)
+
+     Optional. Called by the server when a client includes a list of protocols in the ALPN extension.
+     The server then choose which protocol to use, or "" to disable sending any ALPN response.
+     The default implementation returns the empty string all of the time, effectively disabling
+     ALPN responses.
+
+ .. cpp:function:: void tls_inspect_handshake_msg(const Handshake_Message&)
+
+     This callback is optional, and can be used to inspect all handshake messages
+     while the session establishment occurs.
+
+ .. cpp:function:: void tls_log_debug(const char*)
+
+     This callback is for exerimental purposes and currently unused. It may be
+     removed or modified in a future release.
+
+Versions from 1.11.0 to 1.11.30 did not have ``TLS::Callbacks` and instead
+used independent std::functions to pass the various callback functions.
+This interface is currently still included but is deprecated and will be removed
+in a future release. For the documentation for this interface, please check
+the docs for 1.11.30. This version of the manual only documents the new interface
+added in 1.11.31.
 
 TLS Channels
 ----------------------------------------
@@ -80,16 +119,6 @@ available:
 
 .. cpp:class:: TLS::Channel
 
-   .. cpp:type:: std::function<void (const byte[], size_t)> output_fn
-
-   .. cpp:type:: std::function<void (const byte[], size_t)> data_cb
-
-   .. cpp:type:: std::function<void (Alert, const byte[], size_t)> alert_cb
-
-   .. cpp:type:: std::function<bool (const Session&)> handshake_cb
-
-     Typedefs used in the code for the functions described above
-
    .. cpp:function:: size_t received_data(const byte buf[], size_t buf_size)
    .. cpp:function:: size_t received_data(const std::vector<byte>& buf)
 
@@ -194,10 +223,7 @@ TLS Clients
 .. cpp:class:: TLS::Client
 
    .. cpp:function:: Client( \
-         output_fn out, \
-         data_cb app_data_cb, \
-         alert_cb alert_cb, \
-         handshake_cb hs_cb, \
+         Callbacks& callbacks,
          Session_Manager& session_manager, \
          Credentials_Manager& creds, \
          const Policy& policy, \
@@ -211,29 +237,8 @@ TLS Clients
    Initialize a new TLS client. The constructor will immediately
    initiate a new session.
 
-   The *output_fn* callback will be called with output that
-   should be sent to the counterparty. For instance this will be
-   called immediately from the constructor after the client hello
-   message is constructed. An implementation of *output_fn* is
-   allowed to defer the write (for instance if writing when the
-   callback occurs would block), but should eventually write the data
-   to the counterparty *in order*.
-
-   The *data_cb* will be called with data sent by the counterparty
-   after it has been processed. The byte array and size_t represent
-   the plaintext value and size.
-
-   The *alert_cb* will be called when a protocol alert is received,
-   commonly with a close alert during connection teardown.
-
-   The *handshake_cb* function is called when a handshake
-   (either initial or renegotiation) is completed. The return value of
-   the callback specifies if the session should be cached for later
-   resumption. If the function for some reason desires to prevent the
-   connection from completing, it should throw an exception
-   (preferably a TLS::Exception, which can provide more specific alert
-   information to the counterparty). The :cpp:class:`TLS::Session`
-   provides information about the session that was just established.
+   The *callbacks* parameter specifies the various application callbacks
+   which pertain to this particular client connection.
 
    The *session_manager* is an interface for storing TLS sessions,
    which allows for session resumption upon reconnecting to a server.
@@ -285,34 +290,30 @@ TLS Servers
 .. cpp:class:: TLS::Server
 
    .. cpp:function:: Server( \
-         output_fn output, \
-         data_cb data_cb, \
-         alert_cb alert_cb, \
-         handshake_cb handshake_cb, \
+         Callbacks& callbacks,
          Session_Manager& session_manager, \
          Credentials_Manager& creds, \
          const Policy& policy, \
          RandomNumberGenerator& rng, \
-         next_protocol_fn next_proto = next_protocol_fn(), \
          bool is_datagram = false, \
          size_t reserved_io_buffer_size = 16*1024 \
          )
 
-The first 8 arguments as well as the final argument
+The first 5 arguments as well as the final argument
 *reserved_io_buffer_size*, are treated similiarly to the :ref:`client
 <tls_client>`.
 
-The (optional) argument, *proto_chooser*, is a function called if the
-client sent the ALPN extension to negotiate an application
-protocol. In that case, the function should choose a protocol to use
-and return it. Alternately it can throw an exception to abort the
-exchange; the ALPN specification says that if this occurs the alert
-should be of type `NO_APPLICATION_PROTOCOL`.
+If a client sends the ALPN extension, the ``callbacks`` function
+``tls_server_choose_app_protocol`` will be called and the result
+sent back to the client. If the empty string is returned, the server
+will not send an ALPN response. The function can also throw an exception
+to abort the handshake entirely, the ALPN specification says that if this
+occurs the alert should be of type `NO_APPLICATION_PROTOCOL`.
 
 The optional argument *is_datagram* specifies if this is a TLS or DTLS
 server; unlike clients, which know what type of protocol (TLS vs DTLS)
 they are negotiating from the start via the *offer_version*, servers
-would not until they actually received a hello without this parameter.
+would not until they actually received a client hello.
 
 Code for a TLS server using asio is in `src/cli/tls_proxy.cpp`.
 
@@ -516,7 +517,7 @@ policy settings from a file.
      Values without an explicit mode use old-style CBC with HMAC encryption.
 
      Default value: "AES-256/GCM", "AES-128/GCM", "ChaCha20Poly1305",
-     "AES-256/CCM", "AES-128/CCM", "AES-256/CCM-8", "AES-128/CCM-8",
+     "AES-256/CCM", "AES-128/CCM", "AES-256/CCM(8)", "AES-128/CCM(8)",
      "AES-256", "AES-128"
 
      Also allowed: "Camellia-256/GCM", "Camellia-128/GCM",
@@ -529,15 +530,18 @@ policy settings from a file.
 
      .. note::
 
-        The current ChaCha20Poly1305 ciphersuites are non-standard but
-        as of 2015 were implemented and deployed by Google and
-        elsewhere. Support will be changed to using IETF standard
-        ChaCha20Poly1305 ciphersuites when those are defined.
+        Before 1.11.30 only the non-standard ChaCha20Poly1305 ciphersuite
+        was implemented. The RFC 7905 ciphersuites are supported in 1.11.30
+        onwards.
 
      .. note::
 
         Support for the broken RC4 cipher was removed in 1.11.17
 
+     .. note::
+
+        SEED and 3DES are deprecated and will be removed in a future release.
+
  .. cpp:function:: std::vector<std::string> allowed_macs() const
 
      Returns the list of algorithms we are willing to use for
@@ -577,6 +581,10 @@ policy settings from a file.
 
      Also allowed (disabled by default): "" (meaning anonymous)
 
+     .. note::
+
+        DSA authentication is deprecated and will be removed in a future release.
+
  .. cpp:function:: std::vector<std::string> allowed_ecc_curves() const
 
      Return a list of ECC curves we are willing to use, in order of preference.
diff --git a/doc/news.rst b/doc/news.rst
index 2a9fe53a6..711f982a8 100644
--- a/doc/news.rst
+++ b/doc/news.rst
@@ -1,7 +1,128 @@
 Release Notes
 ========================================
 
-Version 1.11.30, Not Yet Released
+Version 1.11.31, 2016-08-30
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Fix undefined behavior in Curve25519 on platforms without a native 128-bit
+  integer type. This was known to produce incorrect results on 32-bit ARM
+  under Clang. GH #532 (CVE-2016-6878)
+
+* If X509_Certificate::allowed_usage was called with more than one Key_Usage
+  set in the enum value, the function would return true if *any* of the allowed
+  usages were set, instead of if *all* of the allowed usages are set.
+  GH #591 (CVE-2016-6879)
+
+* Incompatible changes in DLIES: Previously the input to the KDF was
+  the concatenation of the (ephemeral) public key and the secret value
+  derived by the key agreement operation. Now the input is only the
+  secret value obtained by the key agreement operation. That's how it
+  is specified in the original paper "DHIES: An encryption scheme
+  based on Diffie-Hellman Problem" or in BSI technical guideline
+  TR-02102-1 for example. In addition to the already present
+  XOR-encrypion/decryption mode it's now possible to use DLIES with a
+  block cipher.  Furthermore the order of the output was changed from
+  {public key, tag, ciphertext} to {public key, ciphertext, tag}. Both
+  modes are compatible with BouncyCastle.
+
+* Add initial PKCS #11 support (GH #507). Currently includes a low level
+  wrapper to all of PKCS #11 (p11.h) and high level code for RSA and ECDSA
+  signatures and hardware RNG access.
+
+* Add ECIES encryption scheme, compatible with BouncyCastle (GH #483)
+
+* Add ECKCDSA signature algorithm (GH #504)
+
+* Add KDF1 from ISO 18033 (GH #483)
+
+* Add FRP256v1 curve (GH #551)
+
+* Changes for userspace PRNGs HMAC_DRBG and HMAC_RNG (GH #520 and #593)
+
+  These RNGs now derive from Stateful_RNG which handles issues like periodic
+  reseeding and (on Unix) detecting use of fork. Previously these measures were
+  included only in HMAC_RNG.
+
+  Stateful_RNG allows reseeding from another RNG and/or a specified set of
+  entropy sources. For example it is possible to configure a HMAC_DRBG to reseed
+  using a PKCS #11 token RNG, the CPU's RDSEED instruction, and the system RNG
+  but disabling all other entropy polls.
+
+* AutoSeeded_RNG now uses NIST SP800-90a HMAC_DRBG(SHA-384). (GH #520)
+
+* On Windows and Unix systems, the system PRNG is used as the sole reseeding
+  source for a default AutoSeeded_RNG, completely skipping the standard entropy
+  polling code. New constructors allow specifying the reseed RNG and/or entropy
+  sources. (GH #520)
+
+* Add RDRAND_RNG which directly exposes the CPU RNG (GH #543)
+
+* Add PKCS #1 v1.5 id for SHA-512/256 (GH #554)
+
+* Add X509_Time::to_std_timepoint (GH #560)
+
+* Fix a bug in ANSI X9.23 padding mode, which returned one byte more
+  than the given block size (GH #529).
+
+* Fix bug in SipHash::clear, which did not reset all state (GH #547)
+
+* Fixes for FreeBSD (GH #517) and OpenBSD (GH #523). The compiler defaults
+  to Clang on FreeBSD now.
+
+* SonarQube static analysis integration (GH #592)
+
+* Switched Travis CI to Ubuntu 14.04 LTS (GH #592)
+
+* Added ARM32, ARM64, PPC32, PPC64, and MinGW x86 cross compile targets to Travis CI (GH #608)
+
+* Clean up in TLS ciphersuite handling (GH #583)
+
+* Threefish-512 AVX2 optimization work (GH #581)
+
+* Remove build configuration host and timestamp from build.h
+  This makes this header reproducible and allows using ccache's direct mode
+  (GH #586 see also #587)
+
+* Prevent building for x86-64 with x86-32 compiler and the reverse (GH #585)
+
+* Avoid build problem on 32-bit userspace ARMv8 (GH #563)
+
+* Refactor of internal MP headers (GH #549)
+
+* Avoid MSVC C4100 warning (GH #525)
+
+* Change botan.exe to botan-cli.exe on Windows to workaround VC issue (GH #584)
+
+* More tests for RSA-KEM (GH #538), DH (GH #556), EME (GH #553),
+  cipher mode padding (GH #529), CTS mode (GH #531),
+  KDF1/ISO18033 (GH #537), OctetString (GH #545), OIDs (GH #546),
+  parallel hash (GH #548), charset handling (GH #555),
+  BigInt (GH #558), HMAC_DRBG (GH #598 #600)
+
+* New deprecations. See the full list in doc/deprecated.txt
+
+  The X9.31 and HMAC_RNG RNGs are deprecated.
+  If you need a userspace PRNG, use HMAC_DRBG (or AutoSeeded_RNG
+  which is HMAC_DRBG with defaults).
+
+  Support for getting entropy from EGD is deprecated, and will be
+  removed in a future release. The developers believe that it is
+  unlikely that any modern system requires EGD and so the code is now
+  dead weight. If you rely on EGD support, you should contact the
+  developers by email or GitHub ASAP.
+
+  The TLS ciphersuites using 3DES and SEED are deprecated and will be
+  removed in a future release.
+
+  ECB mode Cipher_Mode is deprecated and will be removed in a future
+  release.
+
+  Support for BeOS/Haiku has not been tested in 5+ years and is in an
+  unknown state.  Unless reports are received of successful builds and
+  use on this platform, support for BeOS/Haiku will be removed in a
+  future release.
+
+Version 1.11.30, 2016-06-19
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 * In 1.11.23 a bug was introduced such that CBC-encrypted TLS packets
@@ -9,16 +130,48 @@ Version 1.11.30, Not Yet Released
   a MAC failure. Records like this are used by OpenSSL in TLS 1.0
   connections in order to randomize the IV.
 
+* A bug in GCM caused incorrect results if the 32-bit counter field
+  overflowed. This bug has no implications on the security but affects
+  interoperability.
+
+  With a 96-bit nonce, this could only occur if at least 2**32 128-bit
+  blocks (64 GiB) were encrypted. This actually exceeds the maximum
+  allowable length of a GCM plaintext; when messages longer than
+  2**32 - 2 blocks are encrypted, GCM loses its security properties.
+
+  In addition to 96-bit nonces, GCM also supports nonces of arbitrary
+  length using a different method which hashes the provided nonce
+  under the authentication key. When using such a nonce, the last 4
+  bytes of the resulting CTR input might be near the overflow
+  boundary, with the probability of incorrect overflow increasing with
+  longer messages. when encrypting 256 MiB of data under a random 128
+  bit nonce, an incorrect result would be produced about 1/256 of the
+  time. With 1 MiB texts, the probability of error is reduced to 1/65536.
+
+  Since TLS uses GCM with 96 bit nonces and limits the length of any
+  record to far less than 64 GiB, TLS GCM ciphersuites are not
+  affected by this bug.
+
+  Reported by Juraj Somorovsky, described also in "Nonce-Disrespecting
+  Adversaries: Practical Forgery Attacks on GCM in TLS"
+  (https://eprint.iacr.org/2016/475.pdf)
+
+* Previously when generating a new self-signed certificate or PKCS #10
+  request, the subject DN was required to contain both common name
+  (CN) and country (C) fields. These restrictions have been removed.
+  GH #496
+
 * The Transform and Keyed_Transform interfaces has been removed. The
   two concrete implementations of these interfaces were Cipher_Mode
-  and the Compressor_tkk. The Cipher_Mode interface remains unchanged
+  and Compressor_Transform. The Cipher_Mode interface remains unchanged
   as the Transform and Keyed_Transform signatures have moved to it;
   no changes to Cipher_Mode usage should be necessary. Any uses of
   Transform& or Keyed_Transform& to refer to a cipher should be replaced
   by Cipher_Mode&. The compression algorithm interface has changed; the start
   function now takes the per-message compression ratio to use. Previously the
   compression level to use had to be set once, at creation time, and
-  the required `secure_vector` argument to start was required to be empty.
+  the required ``secure_vector`` argument to ``start`` was required to be empty.
+  The new API is documented in `compression.rst` in the manual.
 
 * Add IETF versions of the ChaCha20Poly1305 TLS ciphersuites from
   draft-ietf-tls-chacha20-poly1305-04. The previously implemented
@@ -37,13 +190,21 @@ Version 1.11.30, Not Yet Released
 
 * X509_CRL previously had an option to cause it to ignore unknown
   critical extensions. This has been removed.
- 
-* Added support for ChaCha stream cipher with 12 rounds.
+
+* Added StreamCipher::seek allowing seeking to arbitrary position
+  in the key stream. Currently only implemented for ChaCha. (GH #497)
+
+* Added support for ChaCha stream cipher with 8 or 12 rounds.
 
 * Add ECGDSA signature algorithm (GH #479)
 
+* Add support for label argument to KDFs (GH #495)
+
 * Add NIST SP800-108 and 56C KDFs (GH #481)
 
+* Support for Card Verifiable Certificates and the obsolete EMSA1_BSI
+  signature padding scheme have been removed. (GH #487)
+
 * A bug in the IETF version of ChaCha20Poly1305 (with 96 bit nonces)
   caused incorrect computation when the plaintext or AAD was exactly
   a multiple of 16 bytes.
@@ -56,6 +217,17 @@ Version 1.11.30, Not Yet Released
 
 * Fix bcrypt function under Python 3 (GH #461)
 
+* The ``unix_procs`` entropy source is deprecated and will be removed
+  in a future release. This entropy source attempts to get entropy by
+  running Unix programs like ``arp``, ``netstat``, and ``dmesg`` which
+  produce information which may be difficult for a remote attacker to
+  guess. This exists primarily as a last-ditch for Unix systems
+  without ``/dev/random``. But at this point such systems effectively
+  no longer exist, and the use of ``fork`` and ``exec`` by the library
+  complicates effective application sandboxing.
+
+* Changes to avoid implicit cast warnings in Visual C++ (GH #484)
+
 Version 1.10.13, 2016-04-23
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -179,6 +351,19 @@ Version 1.11.29, 2016-03-20
 
 * Support for locking allocator on Windows using VirtualLock. GH #450
 
+Version 1.18.15, 2016-02-13
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* NOTE WELL: Botan 1.8 is not supported for security issues anymore.
+  Moving to 1.10 or 1.11 is certainly recommended.
+* Fix CVE-2014-9742: Insufficient randomness in Miller-Rabin primality check
+* Fix CVE-2016-2194: Infinite loop in modulur square root algorithm
+* Fix CVE-2015-5726: Crash in BER decoder
+* Fix CVE-2015-5727: Excess memory allocation in BER decoder
+  Note: Unlike the fix in 1.10 which checks that the source actually
+  contains enough data to satisfy the read before allocating the
+  memory, 1.8.15 simply rejects all ASN.1 blocks larger than 1 MiB.
+  This simpler check avoids the problem without breaking ABI.
+
 Version 1.10.12, 2016-02-03
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/doc/security.rst b/doc/security.rst
index 23b46f30d..6223943e0 100644
--- a/doc/security.rst
+++ b/doc/security.rst
@@ -19,6 +19,24 @@ Advisories
 2016
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+* 2016-08-30 (CVE-2016-6878) Undefined behavior in Curve25519
+
+  On systems without a native 128-bit integer type, the Curve25519 code invoked
+  undefined behavior. This was known to produce incorrect results on 32-bit ARM
+  when compiled by Clang.
+
+  Introduced in 1.11.12, fixed in 1.11.31
+
+* 2016-08-30 (CVE-2016-6879) Bad result from X509_Certificate::allowed_usage
+
+  If allowed_usage was called with more than one Key_Usage set in the enum
+  value, the function would return true if *any* of the allowed usages were set,
+  instead of if *all* of the allowed usages are set.  This could be used to
+  bypass an application key usage check. Credit to Daniel Neus of Rohde &
+  Schwarz Cybersecurity for finding this issue.
+
+  Introduced in 1.11.0, fixed in 1.11.31
+
 * 2016-03-17 (CVE-2016-2849): ECDSA side channel
 
   ECDSA (and DSA) signature algorithms perform a modular inverse on the
diff --git a/doc/todo.rst b/doc/todo.rst
index d57c319b7..f4720f3ba 100644
--- a/doc/todo.rst
+++ b/doc/todo.rst
@@ -7,6 +7,14 @@ ticket on GitHub to make sure you're on the right track.
 
 Request a new feature by opening a pull request to update this file.
 
+Documentation
+----------------------------------------
+
+* TPM (no docs)
+* PKCS #11 (no docs)
+* X.509 certs, path validation
+* Specific docs covering one major topic (RSA, ECDSA, AES/GCM, ...)
+
 CLI
 ----------------------------------------
 
@@ -14,7 +22,6 @@ CLI
   for an example
 * `encrypt` / `decrypt` tools providing password and/or public key
   based file encryption
-* `bcrypt` cmdlet
 * Make help output more helpful
 * More microbenchmarks in `speed`: modular exponentiation, ECC point
   multiplication, other BigInt operations
@@ -27,6 +34,7 @@ TLS
 * Make TLS v1.0 and v1.1 optional at build time
 * Make finite field DH optional at build time
 * Curve25519 key exchange
+* NEWHOPE (CECPQ1) key exchange (GH #613)
 * TLS OCSP stapling (RFC 6066)
 * Encrypt-then-MAC extension (RFC 7366)
 * Authentication using TOFU (sqlite3 storage)
@@ -40,6 +48,7 @@ TLS
 PKIX
 ----------------------------------------
 
+* Test suite for validation of 'real world' cert chains (GH #611)
 * Support multiple DNS names in certificates
 * X.509 policy constraints
 * OCSP responder logic