www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - SQLite testing procedures

reply Robert Clipsham <robert octarineparrot.com> writes:
This is a fairly old document now, but it has some interesting ideas 
that could have a place in D (either the language, tools, or phobos):

http://www.sqlite.org/testing.html

Some notable things other than the range of testing methods used:
  * 100% branch test coverage (7.1)
     - It would be nice if dmd's -cov offered a way to give the coverage 
for statements and branches rather than just per line
  * Coverage testing of defensive code (7.2)
    - Some branches are always (or never) executed, which can mean 100% 
branch coverage is not possible unless you remove the "defensive" code.
    - SQLite solves this with the ALWAYS() and NEVER() macros
    - This would only become relevant if -cov was improved
  * Forcing coverage of boundary values and boolean vector tests (7.3)
    - SQLite has a testcase() macro which adds additional branches 
during coverage analysis to test boundary conditions and similar.
    - D currently has no way of doing this, it would be a simple (and 
useful) addition to phobos though.
  * SQLite makes extensive use of assert().
  * The SQLite developers have found static analysis to be completely 
useless compared to having extensive (100%) coverage tests, as well as 
help from tools such as valgrind (particularly praised in the above). 
They've even found themselves introducing bugs while trying to silence 
compiler warnings!

-- 
Robert
http://octarineparrot.com/
Aug 05 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday 05 August 2011 21:13:17 Robert Clipsham wrote:
 This is a fairly old document now, but it has some interesting ideas
 that could have a place in D (either the language, tools, or phobos):
 
 http://www.sqlite.org/testing.html
 
 Some notable things other than the range of testing methods used:
   * 100% branch test coverage (7.1)
      - It would be nice if dmd's -cov offered a way to give the coverage
 for statements and branches rather than just per line

But each branch of code is on a particular branch, so if every branch is hit, every line is hit. Do you mean cases such as where you have 5 if statements in a row, and every one of them gets hit at one time or another, but in some cases the execution path goes through 3 of them or 2 of them or whatever, but not every single combination is tested? That would be difficult to do given that there's a good chance that certain combinations of branches should never actually occur. It would also complicate the implementation of -cov considerably I would think.
   * Coverage testing of defensive code (7.2)
     - Some branches are always (or never) executed, which can mean 100%
 branch coverage is not possible unless you remove the "defensive" code.
     - SQLite solves this with the ALWAYS() and NEVER() macros
     - This would only become relevant if -cov was improved

It already happens. If you use assert(0) in your code anywhere (prime examples of that being in the default case of a switch statement which should never happen or in a catch block which should never be hit but is required to make the function nothrow because a function in the try block theoretically _could_ throw but never actually will with the arguments that it's given in that case). std.datetime has very high code coverage (100% aside from the assert(0) lines, I believe), but the assert(0) lines are naturally never hit (it would be a bug if they were), so -cov does not actually give it 100% (it gets something more like 98%). - Jonathan M Davis
Aug 05 2011
parent Robert Clipsham <robert octarineparrot.com> writes:
On 05/08/2011 21:22, Jonathan M Davis wrote:
 On Friday 05 August 2011 21:13:17 Robert Clipsham wrote:
 This is a fairly old document now, but it has some interesting ideas
 that could have a place in D (either the language, tools, or phobos):

 http://www.sqlite.org/testing.html

 Some notable things other than the range of testing methods used:
    * 100% branch test coverage (7.1)
       - It would be nice if dmd's -cov offered a way to give the coverage
 for statements and branches rather than just per line

But each branch of code is on a particular branch, so if every branch is hit, every line is hit. Do you mean cases such as where you have 5 if statements in a row, and every one of them gets hit at one time or another, but in some cases the execution path goes through 3 of them or 2 of them or whatever, but not every single combination is tested? That would be difficult to do given that there's a good chance that certain combinations of branches should never actually occur. It would also complicate the implementation of -cov considerably I would think.

// This gets 100% coverage currently if a == 1. No testing of the // second branch is necessary. Or vice-versa. if (a == 1 || b > 2) { } With branch based coverage, both branches must get tested for 100% coverage - SQLite's testing is incredibly thorough, and it shows in their rock solid product.
    * Coverage testing of defensive code (7.2)
      - Some branches are always (or never) executed, which can mean 100%
 branch coverage is not possible unless you remove the "defensive" code.
      - SQLite solves this with the ALWAYS() and NEVER() macros
      - This would only become relevant if -cov was improved

It already happens. If you use assert(0) in your code anywhere (prime examples of that being in the default case of a switch statement which should never happen or in a catch block which should never be hit but is required to make the function nothrow because a function in the try block theoretically _could_ throw but never actually will with the arguments that it's given in that case). std.datetime has very high code coverage (100% aside from the assert(0) lines, I believe), but the assert(0) lines are naturally never hit (it would be a bug if they were), so -cov does not actually give it 100% (it gets something more like 98%).

It might be worth reading the article about this, I can't summarize it sufficiently.
 - Jonathan M Davis

-- Robert http://octarineparrot.com/
Aug 05 2011
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 05/08/2011 21:13, Robert Clipsham wrote:
 * Forcing coverage of boundary values and boolean vector tests (7.3)
 - SQLite has a testcase() macro which adds additional branches during
 coverage analysis to test boundary conditions and similar.
 - D currently has no way of doing this, it would be a simple (and
 useful) addition to phobos though.

I came up with a quick template to do this - should I make a pull request, and if so, where to? ---- void coverageTest(size_t line) { // Make sure this doesn't get optimized away static size_t count = 0; count += line; } string testcase(string test, size_t line=__LINE__)() { version(D_Coverage) { // This has to be on one line so coverage // line numbers don't get messed up. return `if (`~test.stringof~`) { coverageTest(`~line.stringof~`); }`; } else { return ``; } } ---- Usage: mixin(testcase!(`a < b`)); -- Robert http://octarineparrot.com/
Aug 05 2011
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
Robert Clipsham Wrote:

 This is a fairly old document now, but it has some interesting ideas 
 that could have a place in D (either the language, tools, or phobos):

There is a part about integer overflow checks:
8.6 Signed-Integer Overflow Checks

The various C language standards say that the signed-integer overflow behavior is undefined. In other words, when you add a value to a signed integer such that the result is too large to fit in that integer, the value does not necessarily wrap around to a negative number, as most programmers expect. It might do that. But it might do something completely different. See, for example, here and here. Even the same compiler might do something different with signed integer overflow in different places in the code or at different optimizations settings. SQLite never overflows a signed integer. To verify this, the test suites are run at least once when compiled with the -ftrapv option to GCC. The -ftrapv option causes GCC to generate code that will panic() on a signed integer overflow. In addition, there are many test cases the strive to provoke integer overflows using boundary value calculations such as "SELECT -1*(-9223372036854775808);".< Bye, bearophile
Aug 06 2011