mORMot Open Source

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

How does our unit testing work?

What about automated testing?

You know that testing is (almost) everything if you want to avoid regression problems in your application.

How can you be confident that any change made to your software code won't create any error in other part of the software?

So automated unit testing is the good candidate for implementing this.

And even better, testing-driven coding is great:
0. write a void implementation of a feature, that is code the interface with no implementation;
1. write a test code;
2. launch the test - it must fail;
3. implement the feature;
4. launch the test - it must pass;
5. add some features, and repeat all previous tests every time you add a new feature.

It could sounds like a waste of time, but such coding improve your code quality a lot, and, at least, it help you write and optimize every implementation feature.

But don't forget that unit testing is not enough: you have to do tests with your real application, and perform tasks like any user, in order to validate it works as expected. That's why we added the writing and cross-referencing of test protocols in our   SynProject documentation tool.

So how is testing implemented in our framework?

We may have used DUnit - http://sourceforge.net/projects/dunit

But I didn't like the fact that it relies on IDE and create separated units for testing. I find it useful to make tests in pure code, in the same unit which implement them. Smartlink of the Delphi compiler won't put the testing code in your final application, so I don't see a lot of drawbacks. And I don't like visual interfaces with red or green lights... I prefer text files and command line. And DUnit code is bigger than mine, and I don't need so many options. That's a matter of taste - you can not agree, that's fine.

So what about using RTTI for adding tests to your program?

The SynCommons unit implements two classes:

type
  /// a class used to run a suit of test cases
  TSynTests = class(TSynTest)
  private
    function GetTestCase(Index: integer): TSynTestCase;
    function GetTestCaseCount: Integer;
    function GetFailedCaseIdent(Index: integer): string;
    function GetFailedCount: integer;
    function GetFailedMessage(Index: integer): string;
    function GetFailedCase(Index: integer): TSynTestCase;
  protected
    /// a list containing all failed tests after a call to the Run method
    // - if integer(Objects[]) is equal or higher than InternalTestsCount,
    // the Objects[] points to the TSynTestCase, and the Strings[] to the
    // associated failure message
    // - if integer(Objects[]) is lower than InternalTestsCount, then
    // it is an index to the corresponding published method, and the Strings[]
    // contains the associated failure message
    fFailed: TStringList;
    fTestCase: TObjectList;
    fAssertions: integer;
    fAssertionsFailed: integer;
    fCurrentMethod, fCurrentMethodFirstTestCaseIndex: integer;
    fSaveToFile: Text;
    /// this method is called during the run, for every testcase
    // - this implementation just report some minimal data to the console
    // by default, but may be overriden to update a real UI or reporting system
    // - the TestMethodIndex is first -1 before any TestMethod[] method call,
    // then called once after every TestMethod[] run
    procedure DuringRun(TestCaseIndex, TestMethodIndex: integer); virtual;
  public
    /// if set to a text file address, some debug messages will be reported to
    // this text file
    // - for example, use the following line to report to the console:
    // !  ToConsole := @Output;
    // - you can also use the SaveToFile() method to create an external file
    ToConsole: ^Text;
    /// you can put here some text to be displayed at the end of the messages
    // - some internal versions, e.g.
    // - every line of text must explicitly BEGIN with #13#10
    CustomVersions: string;
{$ifdef MSWINDOWS}
    /// contains the run elapsed time
    RunTimer: TPrecisionTimer;
{$endif}
    /// create the test instance
    // - if an identifier is not supplied, the class name is used, after
    // T[Syn][Test] left trim and un-camel-case
    // - this constructor will add all published methods to the internal
    // test list, accessible via the Count/TestName/TestMethod properties
    constructor Create(const Ident: string = '');
    /// finalize the class instance
    // - release all registered Test case instance
    destructor Destroy; override;
    /// save the debug messages into an external file
    // - if no file name is specified, the current Ident is used
    procedure SaveToFile(const DestPath: TFileName; const FileName: TFileName='');
    /// register a specified Test case instance
    // - all these instances will be freed by the TSynTests.Destroy
    // - the published methods of the children must call this method in order
    // to add test cases
    // - example of use (code from a TSynTests published method):
    // !  AddCase(TOneTestCase.Create(self));
    procedure AddCase(TestCase: TSynTestCase); overload;
      {$ifdef HASINLINE}inline;{$endif}
    /// register a specified Test case instance
    // - an instance of the supplied class is created, and will be freed by
    // TSynTests.Destroy
    // - the published methods of the children must call this method in order
    // to add test cases
    // - example of use (code from a TSynTests published method):
    // !  AddCase([TOneTestCase]);
    procedure AddCase(const TestCase: array of TSynTestCaseClass); overload;
    /// call of this method will run all associated tests cases
    // - function will return TRUE if all test passed
    // - all failed test cases will be added to the Failed[] list
    // - the TestCase[] list is created first, by running all published methods,
    // which must call the AddCase() method above to register test cases
    // - the Failed[] list is cleared at the beginning of the run
    // - Assertions and AssertionsFailed properties are reset and computed during
    // the run
    function Run: Boolean; virtual;
    /// the number of items in the TestCase[] array
    property TestCaseCount: Integer read GetTestCaseCount;
    /// an array containing all registered Test case instances
    // - Test cases are registered by the AddCase() method above, mainly
    // by published methods of the children
    // - Test cases instances are freed by TSynTests.Destroy
    property TestCase[Index: integer]: TSynTestCase read GetTestCase;
    /// number of failed tests after the last call to the Run method
    property FailedCount: integer read GetFailedCount;
    /// retrieve the TSynTestCase instance associated with this failure
    // - returns nil if this failure was not trigerred by a TSynTestCase,
    // but directly by a method
    property FailedCase[Index: integer]: TSynTestCase read GetFailedCase;
    /// retrieve the ident of the case test associated with this failure
    property FailedCaseIdent[Index: integer]: string read GetFailedCaseIdent;
    /// retrieve the error message associated with this failure
    property FailedMessage[Index: integer]: string read GetFailedMessage;
    /// the number of assertions (i.e. Check() method call) in all tests
    // - this property is set by the Run method above
    property Assertions: integer read fAssertions;
    /// the number of assertions (i.e. Check() method call) which failed in all tests
    // - this property is set by the Run method above
    property AssertionsFailed: integer read fAssertionsFailed;
  published
    { all published methods of the children will be run as test cases registering
      - these methods must be declared as procedure with no parameter
      - every method should create a customized TSynTestCase instance,
        which will be registered with the AddCase() method, then automaticaly
        destroyed during the TSynTests destroy  }
  end;

which is used to register/list the tests; and

type
  /// a class implementing a test case
  // - should handle a test unit, i.e. one or more tests
  // - individual tests are written in the published methods of this class
  TSynTestCase = class(TSynTest)
  protected
    fOwner: TSynTests;
    fAssertions: integer;
    fAssertionsFailed: integer;
    fAssertionsBeforeRun: integer;
    fAssertionsFailedBeforeRun: integer;
    fMethodIndex: integer;
    fTestCaseIndex: integer;
    /// any text assigned to this field will be displayed on console
    fRunConsole: string;
  public
    /// create the test case instance
    // - must supply a test suit owner
    // - if an identifier is not supplied, the class name is used, after
    // T[Syn][Test] left trim and un-camel-case
    constructor Create(Owner: TSynTests; const Ident: string = ''); virtual;
    /// used by the published methods to run a test assertion
    // - condition must equals TRUE to pass the test
    // - function return TRUE if the condition failed, in order to allow the
    // caller to stop testing with such code:
    // ! if Check(A=10) then exit;
    function Check(condition: Boolean; const msg: string = ''): Boolean;
      {$ifdef HASINLINE}inline;{$endif}
    /// used by the published methods to run a test assertion
    // - condition must equals FALSE to pass the test
    // - function return TRUE if the condition failed, in order to allow the
    // caller to stop testing with such code:
    // ! if CheckNot(A<>10) then exit;
    function CheckNot(condition: Boolean; const msg: string = ''): Boolean;
      {$ifdef HASINLINE}inline;{$endif}
    /// create a temporary string random content
    // - it somewhat faster if CharCount is a multiple of 5
    function RandomString(CharCount: Integer): RawByteString;
    /// create a temporary string random content
    // - it somewhat faster if CharCount is a multiple of 5
    function RandomUTF8(CharCount: Integer): RawUTF8;
    /// this method is trigerred internaly - e.g. by Check() - when a test failed
    procedure TestFailed(const msg: string);
    /// the test suit which owns this test case
    property Owner: TSynTests read fOwner;
    /// the test name
    // - either the Ident parameter supplied to the Create() method, either
    // an uncameled text from the class name
    property Ident: string read GetIdent;
    /// the number of assertions (i.e. Check() method call) for this test case
    property Assertions: integer read fAssertions;
    /// the number of assertions (i.e. Check() method call) for this test case
    property AssertionsFailed: integer read fAssertionsFailed;
    /// the index of the associated Owner.TestMethod[] which created this test
    property MethodIndex: integer read fMethodIndex;
    /// the index of the test case, starting at 0 for the associated MethodIndex
    property TestCaseIndex: integer read fTestCaseIndex;
  published
    { all published methods of the children will be run as individual tests
      - these methods must be declared as procedure with no parameter }
  end;

Sample code

Here are the functions we want to test:

function Add(A,B: double): Double; overload;
begin
  result := A+B;
end;

function Add(A,B: integer): integer; overload;
begin
  result := A+B;
end;

function Multiply(A,B: double): Double; overload;
begin
  result := A*B;
end;

function Multiply(A,B: integer): integer; overload;
begin
  result := A*B;
end;

So we create three classes one for the whole test suit, one for testing addition, one for testing multiplication:

type
  TTestNumbersAdding = class(TSynTestCase)
  published
    procedure TestIntegerAdd;
    procedure TestDoubleAdd;
  end;

  TTestNumbersMultiplying = class(TSynTestCase)
  published
    procedure TestIntegerMultiply;
    procedure TestDoubleMultiply;
  end;

  TTestSuit = class(TSynTests)
  published
    procedure MyTestSuit;
  end;

The trick is to create published methods.

Here is how one of these test methods are implemented (I let you guess the others):

procedure TTestNumbersAdding.TestDoubleAdd;
var A,B: double;
    i: integer;
begin
  for i := 1 to 1000 do
  begin
    A := Random;
    B := Random;
    Check(SameValue(A+B,Adding(A,B)));
  end;
end;

The SameValue() is necessary because of floating-point precision problem, we can't trust plain = operator.

And here is the test case implementation:

procedure TTestSuit.MyTestSuit;
begin
  AddCase([TTestNumbersAdding,TTestNumbersMultiplying]);
end;

And the main program:

  with TTestSuit.Create do
  try
    ToConsole := @Output; // so we will see something on screen
    Run;
    readln;
  finally
    Free;
  end;

Just run this program, and you'll get:

Suit
----


1. My test suit

 1.1. Numbers adding:
  - Test integer add: 1000 assertions passed
  - Test double add: 1000 assertions passed
  Total failed: 0 / 2000  - Numbers adding PASSED

 1.2. Numbers multiplying:
  - Test integer multiply: 1000 assertions passed
  - Test double multiply: 1000 assertions passed
  Total failed: 0 / 2000  - Numbers multiplying PASSED


Generated with: Delphi 7 compiler

Time elapsed for all tests: 1.96ms
Tests performed at 23/07/2010 15:24:30

Total assertions failed for all test suits:  0 / 4000

! All tests passed successfully.

You can see that all text on screen was created by "uncamelcasing" the method names, and that the test suit just follows the classes defined.

I've uploaded this test in the SQLite3\Sample\07 - SynTest folder of our Source Code Repository.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: How does our unit testing work?

Here is a direct access to the test suit sample unit from our Source Code Repository:

http://synopse.info/fossil/artifact?nam … c74e1839dc

You'll see how simple it's to add testing to your apps, with this two testing classes.

The resulting source code seems at least as clear and readable than the same tests written using the DUnit framework. And with some improvements like using CamelCase and RTTI. And I'm not sure it's just a matter of taste, this time!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: How does our unit testing work?

Here is a typical report of the unit tests supplied with the framework.

More than 1,000,000 tests are performed, with good code coverage.

We tried to implement test driven development for all root classes of our ORM Framework.

All low-level (numerical or UTF-8 text conversion) and high-level features (RTTI, ORM, JSON, database, client/server) were tested before their implementation.

We even made some basic regression tests about the encryption or pdf generation part.

Synopse SQLite3 Framework Automated tests
-------------------------------------------


1. Synopse libraries

 1.1. Low level common:
  - System copy record: 22 assertions passed
  - Fast string compare: 7 assertions passed
  - IdemPropName: 8 assertions passed
  - Url encoding: 104 assertions passed
  - Soundex: 29 assertions passed
  - Numerical conversions: 330,015 assertions passed
  - Curr64: 20,012 assertions passed
  - CamelCase: 5 assertions passed
  - Bits: 4,614 assertions passed
  - Ini files: 7,000 assertions passed
  - UTF8: 12,020 assertions passed
  - TSynTable: 873 assertions passed
  Total failed: 0 / 374,709  - Low level common PASSED

 1.2. Low level types:
  - Iso8601 date and time: 24,000 assertions passed
  - Url decoding: 1,200 assertions passed
  - RTTI: 22 assertions passed
  - Json encode decode: 606 assertions passed
  Total failed: 0 / 25,828  - Low level types PASSED

 1.3. Big table:
  - TSynBigTable: 19,232 assertions passed
  - TSynBigTableString: 16,261 assertions passed
  - TSynBigTableMetaData: 255,754 assertions passed
  - TSynBigTableRecord: 195,754 assertions passed
  Total failed: 0 / 487,001  - Big table PASSED

 1.4. Cryptographic routines:
  - Adler32: 1 assertion passed
  - MD5: 1 assertion passed
  - SHA1: 5 assertions passed
  - SHA256: 5 assertions passed
  - AES256: 612 assertions passed
  - Base64: 2,000 assertions passed
  Total failed: 0 / 2,624  - Cryptographic routines PASSED

 1.5. Compression:
  - In memory compression: 12 assertions passed
  - Gzip format: 13 assertions passed
  - Zip format: 10 assertions passed
  Total failed: 0 / 35  - Compression PASSED

 1.6. Synopse PDF:
  - TPdfDocument: 4 assertions passed
  - TPdfDocumentGDI: 3 assertions passed
  Total failed: 0 / 7  - Synopse PDF PASSED


2. SQLite3

 2.1. Basic classes:
  - TSQLCache: 612 assertions passed
  - TSQLRecord: 31 assertions passed
  - TSQLRecordSigned: 200 assertions passed
  - TSQLModel: 3 assertions passed
  Total failed: 0 / 846  - Basic classes PASSED

 2.2. File based:
  - Direct access: 10,124 assertions passed
  - TSQLTableJSON: 20,030 assertions passed
  - TSQLRestClientDB: 29,988 assertions passed
  Total failed: 0 / 60,142  - File based PASSED

 2.3. File based WAL:
  - Direct access: 10,124 assertions passed
  - TSQLTableJSON: 20,030 assertions passed
  - TSQLRestClientDB: 29,988 assertions passed
  Total failed: 0 / 60,142  - File based WAL PASSED

 2.4. Memory based:
  - Direct access: 10,123 assertions passed
  - TSQLTableJSON: 20,030 assertions passed
  - TSQLRestClientDB: 104,317 assertions passed
  Total failed: 0 / 134,470  - Memory based PASSED

 2.5. Client server access:
  - TSQLite3HttpServer: 3 assertions passed
  - TSQLite3HttpClient: 3 assertions passed
  - Http client keep alive: 3,001 assertions passed
     first in 10.61ms, done in 297.30ms i.e. 3363/s, aver. 297us, 15.7 MB/s
  - Http client multi connect: 3,001 assertions passed
     first in 303us, done in 547.09ms i.e. 1827/s, aver. 547us, 8.5 MB/s
  - Named pipe access: 3,003 assertions passed
     first in 79.08ms, done in 162.20ms i.e. 6165/s, aver. 162us, 28.8 MB/s
  - Local window messages: 3,002 assertions passed
     first in 100us, done in 79.61ms i.e. 12561/s, aver. 79us, 58.7 MB/s
  - Direct in process access: 3,001 assertions passed
     first in 56us, done in 56.26ms i.e. 17772/s, aver. 56us, 83.1 MB/s
  Total failed: 0 / 15,014  - Client server access PASSED


Synopse framework used: 1.12
SQlite3 engine used: 3.7.4
Generated with: Delphi 6 compiler

Time elapsed for all tests: 20.89s
Tests performed at 1/20/2011 11:28:20 AM

Total assertions failed for all test suits:  0 / 1,160,818

! All tests passed successfully.
Done - Press ENTER to Exit

About speed of this log file, it was run on a pretty old mono core P4 computer... so expect much better speed results on modern hardware!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: How does our unit testing work?

Today, we reached more than 2,000,000 regression tests for our framework...

edwinsn

Member

Registered: 2010-07-02

Posts: 1,218

Re: How does our unit testing work?

Thanks for your effort, Arnaud!

---

Delphi XE4 Pro on Windows 7 64bit.
Lazarus trunk built with fpcupdelux on Windows with cross-compile for Linux 64bit.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: How does our unit testing work?

The regression tests included with mORMot are close to 10,000,000 checks by now in TestSQL3.dpr and SQLite3SelfTests.pas

We try to make test-driven development in our implementation.
Our little testing classes including within the framework is pretty workable, and has a very small overhead about execution time.

Global testing status is shown on a console text (see above) by default.
We use to include this text within the testing auto-generated documentation (via SynProject) to the Test document.
This is some kind of easy-to-read proof that all automated testing passed successfully before a release.

But you can also log the testing details, including all per-method timing.
See for instance in SQLite3SelfTests.pas:

procedure SQLite3ConsoleTests;
begin
  AllocConsole;
  // will create around 280 MB of log file, if executed
  if false then
  with TSQLLog.Family do begin
    Level := LOG_VERBOSE;
    HighResolutionTimeStamp := true;
    TSynLogTestLog := TSQLLog;
  (....)

Just replace the 'if false' by 'if true', and you will have full logging of all tests, including run SQL statement, and full stack trace in case of a trace failure.
DUnit does not have these nice features. And I do not miss its tree-based GUI.

ComingNine

Member

Registered: 2010-07-29

Posts: 294

Re: How does our unit testing work?

On Page 393 of the doc v1.18, there is a "ToConsole := @Output". In your first post above, there is a "ToConsole: ^Text;".

However, this "ToConsole" seems to be removed in the NightlyBuild. Could you help to comment the reason to remove this and what is the replacement ?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: How does our unit testing work?

This ToConsole reference is deprecated.
We forgot to remove the comment in the source code, so it appears in the documentation at a wrong place.
I just fixed this - see http://synopse.info/fossil/info/11f646182a
Thanks for the remark - it was indeed confusing!

To log to console, you have now the EchoToConsole property.

ComingNine

Member

Registered: 2010-07-29

Posts: 294

Re: How does our unit testing work?

Thank you for your helpful comments !