mORMot Open Source

Leslie7

Member

Registered: 2015-06-25

Posts: 248

Testing the DB engines

ab,

for a long time I had this  view that the best way to catch problems with DB engines is to test them against their previous versions and against other engines in an automated fashion.

(Because of differences in the SQL dialects this  is not an obvious task, but frameworks like AnyDac or mORMot are already dealing with these differences. So even if it is not always the case for now I am going to assume that this will not be a big problem. )

Data access frameworks usually  have a layer communicating with the engines.  Testing could be made very easy, if this layer could send every request to several engines. And the result sets are automatically compared to each other. (The comparison can be done in the background, it would not keep the user/tester waiting. See the details later.) This way any change in the behavior of a new version can be detected automatically. The same goes if an engine handles something differently from other engines.

These can be caught automatically:

- different rows or field values in result sets
- different ordering of the result set
- longer response times than expected   
- issues with transaction handling (these can be especially difficult to spot normally)

During the whole test phase up to beta or public every tester could  test multiply engines without any extra job on anyone's part.

----------

In order this to work it has to be ensured that all databases are at the same state before a new SQL command is started. This can be done two ways:

1. Wait until all the engines finished the previous SQL command. This would slow down the test system.
2. Something similar to event sourcing. DB events has a serialized  unique ID. The execution of the events is queued and logged for every tested engine.  Practically this ID tells  us where every engine is at on our virtual timeline.

The logs produced like this can be very informative about every specific real life application.  It can show where the bottlenecks are with different systems and give a clear indication which db engine/hardware/server deployment combo is better for the specific use case. Migrating to an other engine can be made safer as well, since all db activity is automatically compered and tested for both systems.

Last edited by Leslie7 (2020-08-18 21:20:53)

Leslie7

Member

Registered: 2015-06-25

Posts: 248

Re: Testing the DB engines

An other important way to look at this is that instead of making a decision about which setup to use we can continuously test and develop for multiply engines and setups with the ability to change the live setup any time with no extra effort.