TRecordVersion doesn't work as expected

avavdoshin · 2023-05-30 06:36:55

Hello, all!

Does anybody else using TRecordVersion in mormot2?

I'm stuck with TRecordVersion handling and batch updates.
SQlite db via sqlite3static.
Batch add works perfect, TRecordVersion increments as expected.
But for batch updates via TRestBatch only few of TOrms update their TRecordVersion field, for others TRecordVersion field stays remained. I can see that TRecordVersion doesn't change because i store date/time timestamp in my TOrm class and change it for every update operation.

Maybe i'm doing something wrong way? First i load my TOrm from DB via CreateAndFillPrepared, then i modify some fields and add modified TOrm to batch via TRestBatch.add(TOrm, true) (repeat this for all my TOrms i need to change). Then I use batchSend for sending changes to db.

tbo · 2023-05-30 11:38:30

avavdoshin wrote:

Does anybody else using TRecordVersion in mormot2?

I use TRecordVersion. Since I haven't noticed anything negative yet, I haven't looked for a bug either. Can you give us any more information, such as:

Delphi 11.3, mORMot2 GitHub commit 5186

Or a whole example, as Martin Doyle has done here. Then it is easier to search for the error.

With best regards
Thomas

avavdoshin · 2023-05-30 12:38:00

tbo wrote:

I use TRecordVersion. Since I haven't noticed anything negative yet, I haven't looked for a bug either. Can you give us any more information, such as:
Delphi 11.3, mORMot2 GitHub commit 5186
Or a whole example, as Martin Doyle has done here. Then it is easier to search for the error.
Thomas

hello, Thomas, thank you for answer.
FPC 3.3.1, x86_64, windows 10 (codetyphon 8.0). Mormot2 github version 2.1.5479

Whole example too complicated, here is simplified part of code:

type
TFSDBGlobalStates = class(TOrm)
private
{ some attributes }
FChangeTimeStamp: TUnixMSTime;
FVersion: TRecordVersion;
public
class procedure InitializeTable(const Server: IRestOrmServer; const FieldNames: RawUtf8; Options: TOrmInitializeTableOptions); override;
published
{ some properties }
property ChangeTimeStamp: TUnixMSTime read FChangeTimeStamp write FChangeTimeStamp;
property Version: TRecordVersion read FVersion write FVersion;
end;

function findORM(asrcORM: TOrm; ormClass: TOrmClass):TOrm;
begin
{some code}
if ormClass = TFSDBGlobalStates then
result := ormClass.CreateAndFillPrepare(LocalDBOrm.Orm, 'ObjectGUIDid = ? and SourceGUIDid = ? and StateCodeID = ?', [FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).ObjectGUIDid, false), FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).SourceGUIDid, false), FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).StateCodeID, false)])
{some code}
else
result := nil;
if assigned(result) then
try
result.FillOne;
except
FreeAndNilSafe(result);
end;
end;

procedure CopyOrmClass(aSrcOrm: TOrm; var aDstOrm: TOrm);
begin
{ some code }
if aSrcOrm.ClassName = 'TFSDBLocalStates' then
begin
{some code}
(aDstOrm as TFSDBGlobalStates).ChangeTimeStamp := (aSrcOrm as TFSDBLocalStates).ChangeTimeStamp;
end;
{some code}
end;

Procedure MainProc;
var
srcORM, dstORM: TORM;
SendResult: integer;
begin
tmpAddBatch := TRestBatch.Create(localDBOrm.Orm, TFSDBGlobalStates, 0);
tmpUpdBatch := TRestBatch.Create(localDBOrm.Orm, TFSDBGlobalStates, 0);
{ some code where i get srcOrm }
try
while srcORM.FillOne do
begin
dstORM := FindORM(srcORM, TFSDBGlobalStates);
try
if not assigned(dstORM) then
dstOrm := TFSDBGlobalStates.Create;
CopyOrmClass(srcOrm, dstORM);
if dstOrm.ID > 0 then
tmpUpdBatch.Update(dstOrm)
else
tmpAddBatch.Add(dstORM, true);
finally
FreeAndNilSafe(dstOrm);
end;
end;
finally
FreeAndNilSafe(srcOrm);
end;
{some code}
if (tmpAddBatch.Count>0) or (tmpUpdBatch.count>0) then
begin
{some code}
if tmpAddBatch.Count>0 then
sendResult := LocalDBOrm.Orm.BatchSend(tmpAddBatch)
else
sendResult := HTTP_SUCCESS;
if sendResult=HTTP_SUCCESS then
if tmpUpdBatch.Count >0 then
sendResult := LocalDBOrm.Orm.BatchSend(tmpUpdBatch)
else
sendResult := HTTP_SUCCESS;
{check for unsuccessfull send removed for this example}
if sendresult=HTTP_SUCCESS then
LocalDBOrm.Orm.commit;
end;
end;

profh · 2023-05-31 01:51:58

if dstOrm.ID > 0 then
tmpUpdBatch.Update(dstOrm)
else
tmpAddBatch.Add(dstORM, true);

try it as follow:

          if dstOrm.ID > 0 then
            tmpAddBatch.Delete(dstOrm.IDValue);
          
          tmpAddBatch.Add(dstORM, true);

Last edited by profh (2023-05-31 02:27:06)

avavdoshin · 2023-05-31 04:51:11

profh wrote:

          if dstOrm.ID > 0 then
            tmpAddBatch.Delete(dstOrm.IDValue);
          
          tmpAddBatch.Add(dstORM, true);

Thanks for idea, but i have 2 problems with this solution:

1. For some classes i need to keep ID for dstORM if it already exists for reference integrity
2. I synchronize my master db with slave db periodically, so i have this problem: https://github.com/synopse/mORMot2/issues/147
(synchronize doesn't work if i create record on master db, delete it and then synchronize with slave db - because it tries to delete record which doesn't exist in slave db yet)

mdoyle · 2023-06-01 20:18:25

Not sure what you try to accomplish with the Version field. I use it for master slave replication without any issues. Never checked the numbers but the result of the replication.
It is difficult helping you with this fragment of code. Please provide a working example. But please follow the forum rules and do not copy long code directly into the forum.
I do not understand your batch approach. Please check the documentation 12.3. The batch process is pretty straightforward and works for me.

avavdoshin · 2023-06-02 12:00:49

mdoyle wrote:

Not sure what you try to accomplish with the Version field. I use it for master slave replication without any issues. Never checked the numbers but the result of the replication.
It is difficult helping you with this fragment of code. Please provide a working example. But please follow the forum rules and do not copy long code directly into the forum.
I do not understand your batch approach. Please check the documentation 12.3. The batch process is pretty straightforward and works for me.

Hello and thanks for your answer.

First of all, sorry for my very lame English.

I use Version field for periodical replication from my master database to slave database(s) (from master read-write server to read-only server).
I use batch because data in srcorm are came from another server(s), then i process them and store correspondent data in master orm. There is batch of data in srcorm, some data may result in an dstorm entry being added, some may result in an dstorm
entry being updated, but i need to process all of them in one transaction for many reasons.

Here is a working example with results in csv: https://gist.github.com/avavdoshin/fd1d … f8f0638691
Record versions in this example doesn't changes for updates, so this changes will not be replicated to slave database.

tbo · 2023-06-02 13:38:16

avavdoshin wrote:

Here is a working example with results in csv: https://gist.github.com/avavdoshin/fd1d … f8f0638691
Record versions in this example doesn't changes for updates, so this changes will not be replicated to slave database.

I think you are using Batch incorrectly in your example. In mORMot "ex" and "test" directory, or in this forum post, you can find examples of how to use TRestBatch. Just search for "TRestBatch.Create", "BatchStart" or "BatchSend". Use TRestBatch in your example exactly as they are used in mORMot. Then compare the results and report it here. Or someone else from the forum can help. Maybe I have some time on the weekend, then I can also look again. Otherwise you have to wait until Arnaud is back.

With best regards
Thomas

mdoyle · 2023-06-03 14:54:47

Thank you for sending your example. That makes your question more clear.
I did some tests. It looks like that BatchUpdate does not update the Version field. This is a highly optimized function and focuses on the „simple fields“ and time fields only.
You could use Update instead. I know that Update is slower than BatchUpdate. But at least the Version field is updated correctly.

Last edited by mdoyle (2023-06-04 12:06:08)

mdoyle · 2023-06-04 12:02:45

I created some tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.
Hope that helps finding the issue and a fix.

tbo · 2023-06-04 14:14:47

mdoyle wrote:

I created some tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.

I think your upload is not complete. The file Test1.json is missing.

With best regards
Thomas

mdoyle · 2023-06-05 16:18:43

Sorry, you are right. It is the one which is created in test.orm.sqlite3.pas.
I updated the gist. Please check again https://gist.github.com/martin-doyle/00 … ceac4698fc.

tbo · 2023-06-05 18:31:59

mdoyle wrote:

I updated the gist. Please check again https://gist.github.com/martin-doyle/00 … ceac4698fc.

Delphi 11.3, mORMot 2.1.5499
I have only tested case TTestReplicationSimple "AddCase([TTestReplicationSimple])" for the first time. I get this error reported:

EInOutError {Message:"E/A-Fehler 6"} [Main] at 6c74b3

When I run all three test cases, I get this error reported:

...
fail  #4064 SameRecord ID Master: 107 Slave: 107 
fail  SynReplicationTests.TTestReplicationBatch(02fea050) Replication batch - Test batch update [SameRecord ID Master: 107 Slave: 107] 
EXC   EInOutError {Message:"E/A-Fehler 6"} [Main] at 6b84b3

I think every test case should pass without exception. Sorry, have no time to take a closer look in the next few days.

With best regards
Thomas

mdoyle · 2023-06-05 19:46:24

Sorry again. I apologize for the inconvenience.
I tested on Linux with FPC. For Windows {$apptype console} is missing. Updated the files again. Hope it works now. Thank you for taking care.

 Replication test suite
  ------------------------


1. Test suite

 1.1. Replication simple:
  - Test add: 45 assertions passed  46.93ms
  - Test update: 97 assertions passed  14.95ms
  - Test delete: 72 assertions passed  13.42ms
  Total failed: 0 / 214  - Replication simple PASSED  80.92ms

 1.2. Replication batch:
  - Test batch add: 4,011 assertions passed  42.88ms
!  - Test batch update: 10 / 7,038 FAILED  55.69ms
  - Test batch delete: 6,968 assertions passed  56.58ms
  Total failed: 10 / 18,017  - Replication batch FAILED  159.90ms

 1.3. Replication special:
  - Test add ten and delete first record: 77 assertions passed  12.38ms
!  - Test add ten and delete last record: 3 / 77 FAILED  11.63ms
!  - Test add one and delete record: 1 / 13 FAILED  11.41ms
  Total failed: 4 / 167  - Replication special FAILED  39.53ms


Generated with: Delphi 11.3 Alexandria 32 bit Win compiler

Time elapsed for all tests: 282.22ms
Performed 2023-06-05 21:33:25 by mdoyle on WIN11

Total assertions failed for all test suits:  14 / 18,398
! Some tests FAILED: please correct the code.

Last edited by mdoyle (2023-06-05 19:47:55)

mdoyle · 2023-06-19 17:48:36

It looks like BatchAdd and BatchDelete update the Version field but BatchUpdate does not..

Please find the tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.

avavdoshin · 2023-07-12 04:35:41

mdoyle wrote:

It looks like BatchAdd and BatchDelete update the Version field but BatchUpdate does not.

Arnaud, can you please try to fix this issue?

ab · 2023-08-23 10:29:28

Sorry for the delay.

Please try with https://github.com/synopse/mORMot2/commit/e68d137853d

Now BatchUpdate() works with TRecordVersion, at least in the newly introduced regression tests.

avavdoshin · 2023-08-23 10:32:05

ab wrote:

Now BatchUpdate() works with TRecordVersion, at least in the newly introduced regression tests.

Thank you so much!
I'll try ASAP

mORMot Open Source

#1 2023-05-30 06:36:55

TRecordVersion doesn't work as expected

#2 2023-05-30 11:38:30

Re: TRecordVersion doesn't work as expected

#3 2023-05-30 12:38:00

Re: TRecordVersion doesn't work as expected

#4 2023-05-31 01:51:58

Re: TRecordVersion doesn't work as expected

#5 2023-05-31 04:51:11

Re: TRecordVersion doesn't work as expected

#6 2023-06-01 20:18:25

Re: TRecordVersion doesn't work as expected

#7 2023-06-02 12:00:49

Re: TRecordVersion doesn't work as expected

#8 2023-06-02 13:38:16

Re: TRecordVersion doesn't work as expected

#9 2023-06-03 14:54:47

Re: TRecordVersion doesn't work as expected

#10 2023-06-04 12:02:45

Re: TRecordVersion doesn't work as expected

#11 2023-06-04 14:14:47

Re: TRecordVersion doesn't work as expected

#12 2023-06-05 16:18:43

Re: TRecordVersion doesn't work as expected

#13 2023-06-05 18:31:59

Re: TRecordVersion doesn't work as expected

#14 2023-06-05 19:46:24

Re: TRecordVersion doesn't work as expected

#15 2023-06-19 17:48:36

Re: TRecordVersion doesn't work as expected

#16 2023-07-12 04:35:41

Re: TRecordVersion doesn't work as expected

#17 2023-08-23 10:29:28

Re: TRecordVersion doesn't work as expected

#18 2023-08-23 10:32:05

Re: TRecordVersion doesn't work as expected

Board footer