More notes

[ci skip]

1 files changed, 103 insertions, 62 deletions

diff --git a/doc/hacking.rst b/doc/hacking.rst
index 4d3424626..5f7d34e76 100644
--- a/doc/hacking.rst
+++ b/doc/hacking.rst
@@ -5,13 +5,11 @@ Source Code Layout
 Under `src` there are directories
 
 * `lib` is the library itself, more on that below
-* `cli` (command line interface) is the implementation of the command line application.
-  It is structed as a multicall binary so each program is relatively
-  independent.
+* `cli` is the command line application `botan`
 * `tests` contain what you would expect. Input files go under `tests/data`.
 * `build-data` contains files read by the configure script. For
   example `build-data/cc/gcc.txt` describes various gcc options.
-* `scripts` contains various scripts: install, distribution, various
+* `scripts` contains misc scripts: install, distribution, various
   codegen things. Scripts controlling CI go under `scripts/ci`.
 * `python/botan.py` is the Python ctypes wrapper
 
@@ -22,7 +20,7 @@ Library Layout
 * `utils` contains various utility functions and types
 * `codec` has hex, base64
 * `block` contains the block cipher implementations
-* `modes` contains block cipher modes
+* `modes` contains block cipher modes (CBC, GCM, etc)
 * `stream` contains the stream ciphers
 * `hash` contains the hash function implementations
 * `passhash` contains password hashing algorithms for authentication
@@ -42,18 +40,18 @@ Library Layout
 * `cert` has `x509` (X.509 PKI OCSP is also here) and `cvc` (Card Verifiable Ceritifcates,
   for ePassports)
 * `tls` contains the TLS implementation
-* `filters` has a filter/pipe API for data transforms
-* `misc` contains odds and ends: format preserving encryption, SRP, threshold
-  secret sharing, all or nothing transform, and others
+* `filters` is a filter/pipe API for data transforms
 * `compression` has the compression wrappers (zlib, bzip2, lzma)
 * `ffi` is the C99 API
-* `vendor` contains bindings to external libraries like OpenSSL and Sqlite3
+* `prov` contains bindings to external libraries like OpenSSL
+* `misc` contains odds and ends: format preserving encryption, SRP, threshold
+  secret sharing, all or nothing transform, and others
 
 Copyright Notice
 ========================================
 
 At the top of any new file add a comment with a copyright and
-a reference to the license, for examplee::
+a reference to the license, for example::
 
   /*
   * (C) 2015,2016 Copyright Holder
@@ -62,7 +60,7 @@ a reference to the license, for examplee::
 
 If you are making a substantial or non-trivial change to an existing
 file, add or update your own copyright statement at the top of the
-file.  If you are making a change in a new year not covered by your
+file. If you are making a change in a new year not covered by your
 existing statement, add the year. Even if the years you are making the
 change are consecutive, avoid year ranges: specify each year separated
 by a comma.
@@ -82,16 +80,18 @@ places but should be used judiciously.
 
 When designing a new API (for use either by library users or just
 internally) try writing out the calling code first. That is, write out
-some code calling your idealized API, then just implement that. This
-can often help avoid cut-and-paste by creating the correct abstractions
-needed to solve the problem at hand.
+some code calling your idealized API, then just implement that API.
+This can often help avoid cut-and-paste by creating the correct
+abstractions needed to solve the problem at hand.
 
 The C++11 `auto` keyword is very convenient but only use it when the
 type truly is obvious (considering also the potential for unexpected
 integer conversions and the like, such as an apparent uint8_t being
 promoted to an int).
 
-Use `override` annotations whenever possible.
+If a variable is defined and not modified, declare it `const`
+
+Use `override` annotations whenever overriding a virtual function.
 
 A formatting setup for emacs is included in `scripts/indent.el` but
 the basic formatting style should be obvious. No tabs, and remove
@@ -104,14 +104,20 @@ Prefer using braces on both sides of if/else blocks, even if only
 using a single statement. Again the current code doesn't always do
 this.
 
+Avoid `using namespace` declarations, even inside of single functions.
+One allowed exception is `using namespace std::placeholders` in
+functions which use `std::bind`.
+
+Use `::` to explicitly refer to the global namespace (eg, when calling
+an OS or library function like `::select` or `::sqlite3_open`).
+
 Sending patches
 ========================================
 
 All contributions should be submitted as pull requests via GitHub
 (https://github.com/randombit/botan). If you are planning a large
 change email the mailing list or open a discussion ticket on github
-before starting out to make sure you are on the right path to
-something which we'll be able to accept.
+before starting out to make sure you are on the right path.
 
 Depending on what your change is, your PR should probably also include
 an update to `doc/news.rst` with a note explaining the change. If your
@@ -133,53 +139,88 @@ than the regularly tested GCC, Clang, and Visual Studio compilers.
 External Dependencies
 ========================================
 
-The library should always be as functional as possible when compiled
-with just C++11. However, feel free to use the C++11 language. Little
-mercy is given to sub-par C++11 compilers that don't actually implement
-the language (some temporary concessions are made for MSVC 2013).
-
-Use of compiler extensions is fine whenever appropriate; this is
-typically restricted to a single file or an internal header. Compiler
-extensions used currently include native uint128_t, SIMD intrinsics,
-inline asm syntax and so on, so there are some existing examples of
-appropriate use.
-
-If you're adding a small OS dependency in some larger piece of code,
-try to contain the actual non-portable operations to utils/os_utils.*
-and then call them from there.
-
-Any external library dependency - even optional ones - is met with as
-one PR submitter put it "great skepticism".
-
-At every API boundary there is potential for confusion that does not
-exist when the call stack is all contained within the boundary.  So
-the additional API really needs to pull its weight. For example a
-simple text parser or such which can be trivially implemented is not
-really for consideration. As a rough idea, I (Jack) equate the cost of
-an external dependency as equal to at least 1000 additional lines of
-code in the library. That is, if the library really does need this
-functionality, and it can be done in the library for less than that,
+Compiler Dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The library should always be as functional as possible when compiled with just
+C++11. However, feel free to use the C++11 language. Little mercy is given to
+sub-par C++11 compilers that don't actually implement the language (some
+temporary concessions are made for MSVC 2013).
+
+Use of compiler extensions is fine whenever appropriate; this is typically
+restricted to a single file or an internal header. Compiler extensions used
+currently include native uint128_t, SIMD intrinsics, inline asm syntax and so
+on, so there are some existing examples of appropriate use.
+
+Generally intrinsics or inline asm is preferred over bare assembly to avoid
+calling convention issues among different platforms; the improvement in
+maintainability is seen as worth any potentially performance tradeoff. One risk
+with intrinsics is that the compiler might rewrite your clever const-time SIMD
+into something with a conditional jump, but code intended to be const-time
+should in any case be annotated so it can be checked at runtime with tools.
+
+Operating System Dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you're adding a small OS dependency in some larger piece of code, try to
+contain the actual non-portable operations to utils/os_utils.* and then call
+them from there.
+
+Old and obsolete systems are supported where convenient but generally speaking
+SunOS 5, IRIX 9, Windows 2000 and company are not secure platforms to build
+anything on so no special contortions are necessary. Patches that complicate the
+code in order to support any OS not supported by its vendor will likely be
+rejected. In writing OS specific code, feel free to assume roughly POSIX 2008,
+or for Windows Vista/2008 Server (the oldest versions still supported by
+Microsoft).
+
+Library Dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Any external library dependency - even optional ones - is met with as one PR
+submitter put it "great skepticism".
+
+At every API boundary there is potential for confusion that does not exist when
+the call stack is all contained within the boundary.  So the additional API
+really needs to pull its weight. For example a simple text parser or such which
+can be trivially implemented is not really for consideration. As a rough idea of
+the bar, equate the viewed cost of an external dependency as at least 1000
+additional lines of code in the library. That is, if the library really does
+need this functionality, and it can be done in the library for less than that,
 then it makes sense to just write the code. Yup.
 
-Given the entire library is (accoriding to SLOCcount) 62K lines of
-code, that may give some estimate of the bar - you can do pretty much
-anything in 1000 lines of well written C++11 (the implementations of
-*all* of the message authentication codes is much less than 1K SLOC).
-
-Current (all optional) external dependencies of the library are
-OpenSSL (for accessing their fast RSA and ECDSA impls, not the
-handshake code!), zlib, bzip2, lzma, sqlite3, plus various operating
-system utilities like basic filesystem operations. These are hugely
-useful libraries that provide serious value, and are worth the trouble
-of maintaining an integration with. And importantly their API contract
-doesn't change often: code calling zlib doesn't bitrot much.
-
-Examples of external dependencies that would be appropriate include
-integration with system crypto (PKCS #11, TPM, CommonCrypto, CryptoAPI
-algorithms), potentially a parallelism framework such as Cilk (as part
-of a larger design for parallel message processing, say), or
-hypothentically use of a safe ASN.1 parser (that is, one written in
-Rust or OCaml providing a C API).
+Given the entire library is (accoriding to SLOCcount) 62K lines of code, that
+may give some estimate of the bar - you can do pretty much anything in 1000
+lines of well written C++11 (the implementations of *all* of the message
+authentication codes is much less than 1K SLOC).
+
+Current (all optional) external dependencies of the library are OpenSSL (for
+accessing their fast RSA and ECDSA impls, not the handshake code!), zlib, bzip2,
+lzma, sqlite3, plus various operating system utilities like basic filesystem
+operations. These are hugely useful libraries that provide serious value, and
+are worth the trouble of maintaining an integration with. And importantly their
+API contract doesn't change often: code calling zlib doesn't bitrot much.
+
+Examples of external dependencies that would be appropriate include integration
+with system crypto (PKCS #11, TPM, CommonCrypto, CryptoAPI algorithms),
+potentially a parallelism framework such as Cilk (as part of a larger design for
+parallel message processing, say), or hypothentically use of a safe ASN.1 parser
+(that is, one written in Rust or OCaml providing a C API).
+
+Test Tools
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Integration to better leverage specialized test or verification tools such as
+valgrind, ASan/UBSan, AFL, LLVM libFuzzer, KLEE, Coq, etc is fine. Typically
+these are not enabled or used during normal builds but are specially set up by
+developers or auditors.
+
+Python
+========================================
+
+The house language for scripts is Python. The aim is to support 2.7 and latest
+3.x with the minimum possible number of explicit version checks (ideally zero).
+Support for CPython 2.6, PyPy, etc is a bonus but not required.
 
 Build Tools and Hints
 ========================================