Batch Update/Insert

Kobe · 2013-04-08 12:23:45

Hi,

Amazing framework, nice job.

I was looking at batch operations with the class TSQLHttpclient and I would like to know, what is the number of records that is recomended to use in a batch transaction?

More than 3? What would be more optimized?

ab · 2013-04-08 12:29:41

Even with 1 record per batch, I suspect you won't find any speed difference.
The server-side JSON parsing is blazzing fast, and won't be slower than direct execution via a CRUD method.
The real bottleneck will be the database engine, not the ORM/JSON process.

What may be optimized is the maximum number of records per batch.
Depending on the database engine, too many rows within a transaction may be slower: before the transaction is committed, all data is stored in a temporary WAL file, which takes time to maintain.
Here, a maximum of some 1000s of rows per transaction is a good idea.
In fact, this is not a limitation of the batch itself, but the transaction handling.
With Oracle or SQLite3, this maximum number of rows per transaction is a known speed parameter.

Kobe · 2013-04-09 13:11:17

ok,

Does the event OnUpdateEvent is called when I use the Batch Update/Insert?
If yes, Is it call for each row?

Last edited by Kobe (2013-04-09 13:11:48)

ab · 2013-04-09 15:50:40

Yes, it is, AFAIK.

Kobe · 2014-09-04 17:22:02

Hello, after I updated my source with the lastest version (http://synopse.info/fossil/info/0ec7937 … 2d6927dde3) the OnUpdateEvent is not being called anymore after Batch Update/Insert !

My software really rely on this feature. Is there some new configuration, that I have to do for it to work as it used to work before?

Last edited by Kobe (2014-09-04 17:32:00)

Kobe · 2014-09-04 19:23:12

I don't know if it's right, but looks like it stopped after this change -> http://synopse.info/fossil/info/c999b07 … f8996d3552

ab · 2014-09-04 19:54:19

Could you please create a ticket?

Kobe · 2014-09-04 20:18:43

Ticket created -> http://synopse.info/fossil/info/7a2d9a2 … b2e3720f23

ab · 2014-09-06 14:46:33

No, I think it has nothing to do with the "fixed BATCH process to generate valid JSON conten" commit - http://synopse.info/fossil/info/c999b07e314012b

What is your database configuration? Are you using the internal SQLite3 engine, or an external DB?
Do the regular Update()/Add() methods trigger the notifications? Is it a BATCH problem only?

From our side, we only found out a problem with BATCH insert and SQlite3.
Should be fixed by http://synopse.info/fossil/info/d04e19bec3
No problem detected with BATCH update and SQLite3, or other BATCH modes.

We need additional info here, if this fix is not enough.

Kobe · 2014-09-06 15:56:05

I use internal SQLite3 engine. Now it's fixed, thank you.

ab · 2014-09-06 15:58:48

Happy we fixed it.

Thanks for the feedback!

mORMot Open Source

#1 2013-04-08 12:23:45

Batch Update/Insert

#2 2013-04-08 12:29:41

Re: Batch Update/Insert

#3 2013-04-09 13:11:17

Re: Batch Update/Insert

#4 2013-04-09 15:50:40

Re: Batch Update/Insert

#5 2014-09-04 17:22:02

Re: Batch Update/Insert

#6 2014-09-04 19:23:12

Re: Batch Update/Insert

#7 2014-09-04 19:54:19

Re: Batch Update/Insert

#8 2014-09-04 20:18:43

Re: Batch Update/Insert

#9 2014-09-06 14:46:33

Re: Batch Update/Insert

#10 2014-09-06 15:56:05

Re: Batch Update/Insert

#11 2014-09-06 15:58:48

Re: Batch Update/Insert

Board footer