Serious issues with batches

alpinistbg · 2018-11-24 13:23:52

Hi,

Is there anybody that uses mORMot batches in a production environment and experienced occasionally Client.BatchSend() failures when the load increases? My case is exactly that one.

See: https://synopse.info/forum/viewtopic.php?id=4782

If somebody has a plausible solution I'll very much appreciate if he can share it there, because for me it is kind of a showstopper. Recently I've found that on an installation with only 4 client computers and one mORMot server, in some occasions, the clients fail to execute their batches and the database becomes inconsistent.

Further, I've found that it is not possible to re-send a batch, even the failure is detected. You have just one chance to send it, after that the batch contents were deleted. In the case of failure, there is not much information for the cause (only status code of 500), there is information in the server log but it is misleading.

The problem (for the failure) lies actually in the way the server acquires the DB transaction and the small constant timeout given. And it is aggravated by the fact that not much information is given back to the caller.

Аlthough mORMot pretends to be "made for speed and versatility", my latest experience shows that it can't handle properly even deployments as small as of four clients.

Once more, I'll very much appreciate if anybody can share some similar experience.

Best regards,

ab · 2018-11-24 13:43:51

We use batches database access in production with thousands of clients, and GBs of databases.
In practice, we use SQlite3 as local storage, so that we follow the "Microservices" pattern.

But the batches are run on the server side, never from the client side: we always encapsulate the database access into a persistence service.
In short, we use remote SOA, never remote ORM - also for security purposes.
The client sends application-specific data (as DTOs), then the server make the proper persistence using the ORM, using batch if needed.

If you don't follow this SOA-only design (which is the way to properly implement any client-server process), you may indeed be subject to various problems.

alpinistbg · 2018-11-24 16:25:50

ab wrote:

If you don't follow this SOA-only design (which is the way to properly implement any client-server process), you may indeed be subject to various problems.

Wow! It sounds like a beginning of a tough discussion.

First of all, why I'll be a subject to various problems when I don't follow a particular pattern? Second - I don't think that alleviates the issue with the transactions. Same thing may happen in the server also, taking in account there are several threads running.

I just have an application which uses mORMot as a DB layer. In some occasions I want to use it standalone, in other - I want to use it from several computers with a separate server. It is not a complicated one and I don't want to write a middle/app layer for it. Nor it will have to have more than 10 clients or be published on Internet to get security issues. Still it has to be consistent, hasn't it? How to achieve that without transactions (a.k.a. batches in your terminology)?

Kind regards,

danielkuettner · 2018-11-24 21:20:27

Yes I'm using it in production without problems.

ab · 2018-11-25 17:29:28

Check out how MicroServices work: each one use their own database, and communicate either via events.

In mORMot batches, transactions are used mainly for performance, not for atomicity.
If your data model expects transactions for proper state, it means that you have foreign references between tables.
We use the "document sharding" or "aggregate" model instead (see the doc about it) instead of a relational model. This aggregate is the bound of the transactions.

alpinistbg · 2018-11-25 19:20:53

ab wrote:

In mORMot batches, transactions are used mainly for performance, not for atomicity.

I see. Since the SQLite transaction overhead is added to every single statement (which is in its own implicit transaction), you use the side effect of the explicit transactions to skip low level writes in between records. Why then is all the fuss about the "12.3.3. Unit Of Work pattern" in your documentation?

ab wrote:

If your data model expects transactions for proper state, it means that you have foreign references between tables.

Correct. What's wrong about it?

ab wrote:

We use the "document sharding" or "aggregate" model instead (see the doc about it) instead of a relational model. This aggregate is the bound of the transactions.

Why should I change the model of my application just to walk around an incapability?

Anyway, why we must discuss software architectural matter, when those "batches" exists into the library and obviously they have a concurrency issue?

Kind regards,

ab · 2018-11-25 19:35:00

Sorry if you find the documentation fussy, but it is to be read as a whole (including the architectural part), not as a magic recipe.
In DDD - the context in which the documentation was written - the "Unit of Work" pattern is expected/required to be run inside a "Persistence Service", which is on the server side, together with the business logic (the "Domain Services").
If you have a persistence service, and the writing fails, you have all data in your business layer to handle the problem by recreating a new batch later on, or temporary store the information locally.

Even a professional RDBS like Oracle or MSSQL may/will fail. What if the storage server is down, or the network unavailable?
It is up to the client code to handle such cases.
If you expect the framework to resolve all potential design problems, you are indeed overestimating its capabilities.

Of course, we are welcome for any patch improving the existing framework code.

alpinistbg · 2018-11-25 20:21:28

Even a professional RDBS like Oracle or MSSQL may/will fail. What if the storage server is down, or the network unavailable?

Right. MSSQL can fail in some spectacular ways. But in most cases the reason is returned back to the caller.

I order to handle such a fail, the caller must to *know* the reason. The batch may fail because of duplicate PK, unique index, I/O error, etc. Different failures have different resolutions. In case of mORMot I don't know the exact reason because it is not marshaled back correctly. (I've got a text like "batch broken because of ...")

Even If I know that acquiring the transaction failed, I can't replay the batch, because its content is already cleared. I think it is quite trivial to preserve the contents, even the batch is closed for further modifications, and just to add a "BatchResent" method for example. It is matter of few lines of code, IMHO.

Further, I think the constant of 2000 ms for begin transaction timeout is very optimistic and it should be much more than that. Also, some (exponential) backoff must be provided into the retrying loop. We're dealing with RDBMS with limited transaction capabilities and locking resolution (not versioning).

Of course, we are welcome for any patch improving the existing framework code.

How can I make or suggest a patch?

Edit:
Please, explain me the right way to contribute to the project. Should I post patches here with a proper explanation, or, to make a branch in a repository to make changes there? Have you adopted some procedures for collaboration, etc.

Thank you!

Regards,

Last edited by alpinistbg (2018-11-26 09:31:28)

edwinsn · 2018-11-26 12:45:06

@alpinistbg, I replied you in the post you mentioned above.

alpinistbg wrote:

Right. MSSQL can fail in some spectacular ways. But in most cases the reason is returned back to the caller.
I order to handle such a fail, the caller must to *know* the reason. The batch may fail because of duplicate PK, unique index, I/O error, etc. Different failures have different resolutions. In case of mORMot I don't know the exact reason because it is not marshaled back correctly. (I've got a text like "batch broken because of ...")

I totally agree! The actual database error messages must be transferred back to the the caller (the mORMot clients)

Even If I know that acquiring the transaction failed, I can't replay the batch, because its content is already cleared. I think it is quite trivial to preserve the contents, even the batch is closed for further modifications, and just to add a "BatchResent" method for example. It is matter of few lines of code, IMHO.

This can be achieved by using a standalone TSQLRestBatch, but I'm not sure if TSQLRestBatch is serializeable/unserializeable.

Further, I think the constant of 2000 ms for begin transaction timeout is very optimistic and it should be much more than that. Also, some (exponential) backoff must be provided into the retrying loop. We're dealing with RDBMS with limited transaction capabilities and locking resolution (not versioning).

Agreed, the 2-second timeout should be customizable.

Please, explain me the right way to contribute to the project. Should I post patches here with a proper explanation, or, to make a branch in a repository to make changes there? Have you adopted some procedures for collaboration, etc.

I believe you can contribute via a github pull request (https://github.com/synopse/mORMot/pulls … s%3Aclosed), or do something like the OP here: https://synopse.info/forum/viewtopic.php?id=4795

ab · 2018-11-26 18:14:54

GitHub pull requests are preferred, since we can review and annotate them easily!

Thanks all!

mORMot Open Source

#1 2018-11-24 13:23:52

Serious issues with batches

#2 2018-11-24 13:43:51

Re: Serious issues with batches

#3 2018-11-24 16:25:50

Re: Serious issues with batches

#4 2018-11-24 21:20:27

Re: Serious issues with batches

#5 2018-11-25 17:29:28

Re: Serious issues with batches

#6 2018-11-25 19:20:53

Re: Serious issues with batches

#7 2018-11-25 19:35:00

Re: Serious issues with batches

#8 2018-11-25 20:21:28

Re: Serious issues with batches

#9 2018-11-26 12:45:06

Re: Serious issues with batches

#10 2018-11-26 18:14:54

Re: Serious issues with batches

Board footer