mORMot Open Source

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

TRecordVersion doesn't work as expected

Hello, all!

Does anybody else using TRecordVersion in mormot2?

I'm stuck with TRecordVersion handling and batch updates.
SQlite db via sqlite3static.
Batch add works perfect, TRecordVersion increments as expected.
But for batch updates via TRestBatch only few of TOrms update their TRecordVersion field, for others TRecordVersion field stays remained. I can see that TRecordVersion doesn't change because i store date/time timestamp in my TOrm class and change it for every update operation.

Maybe i'm doing something wrong way? First i load my TOrm from DB via CreateAndFillPrepared, then i modify some fields and add modified TOrm to batch via TRestBatch.add(TOrm, true) (repeat this for all my TOrms i need to change). Then I use batchSend for sending changes to db.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: TRecordVersion doesn't work as expected

Does anybody else using TRecordVersion in mormot2?

I use TRecordVersion. Since I haven't noticed anything negative yet, I haven't looked for a bug either. Can you give us any more information, such as:

Delphi 11.3, mORMot2 GitHub commit 5186

Or a whole example, as Martin Doyle has done here. Then it is easier to search for the error.

With best regards
Thomas

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

Re: TRecordVersion doesn't work as expected

I use TRecordVersion. Since I haven't noticed anything negative yet, I haven't looked for a bug either. Can you give us any more information, such as:

Delphi 11.3, mORMot2 GitHub commit 5186

Or a whole example, as Martin Doyle has done here. Then it is easier to search for the error.

Thomas

hello, Thomas, thank you for answer.
FPC 3.3.1, x86_64, windows 10 (codetyphon 8.0). Mormot2 github version 2.1.5479

Whole example too complicated, here is simplified part of code:

type
  TFSDBGlobalStates = class(TOrm)
  private
    { some attributes }
    FChangeTimeStamp: TUnixMSTime;
    FVersion: TRecordVersion;
  public
    class procedure InitializeTable(const Server: IRestOrmServer; const FieldNames: RawUtf8; Options: TOrmInitializeTableOptions); override;
  published
    { some properties }
    property ChangeTimeStamp: TUnixMSTime read FChangeTimeStamp write FChangeTimeStamp;
    property Version: TRecordVersion read FVersion write FVersion;
  end;

function findORM(asrcORM: TOrm; ormClass: TOrmClass):TOrm;
begin
  {some code}
  if ormClass = TFSDBGlobalStates then
    result := ormClass.CreateAndFillPrepare(LocalDBOrm.Orm, 'ObjectGUIDid = ? and SourceGUIDid = ? and StateCodeID = ?', [FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).ObjectGUIDid, false), FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).SourceGUIDid, false), FindGlobalDictID(SrvId, (aSrcOrm as TFSDBLocalStates).StateCodeID, false)])
  {some code}
  else
    result := nil;
  if assigned(result) then
    try
      result.FillOne;
    except
      FreeAndNilSafe(result);
    end;
end;

procedure CopyOrmClass(aSrcOrm: TOrm; var aDstOrm: TOrm);
begin
  { some code }
  if aSrcOrm.ClassName = 'TFSDBLocalStates' then
    begin
      {some code}
      (aDstOrm as TFSDBGlobalStates).ChangeTimeStamp := (aSrcOrm as TFSDBLocalStates).ChangeTimeStamp;
    end;
  {some code}   
end;

Procedure MainProc;
var
  srcORM, dstORM: TORM;
  SendResult: integer; 
begin
  tmpAddBatch := TRestBatch.Create(localDBOrm.Orm, TFSDBGlobalStates, 0);
  tmpUpdBatch := TRestBatch.Create(localDBOrm.Orm, TFSDBGlobalStates, 0);
  { some code where i get srcOrm  }
  try
    while srcORM.FillOne do
      begin
        dstORM := FindORM(srcORM, TFSDBGlobalStates);
        try
          if not assigned(dstORM) then
            dstOrm := TFSDBGlobalStates.Create;
          CopyOrmClass(srcOrm, dstORM);
          if dstOrm.ID > 0 then
            tmpUpdBatch.Update(dstOrm)
          else
            tmpAddBatch.Add(dstORM, true);
        finally
          FreeAndNilSafe(dstOrm);
        end;
      end;   
  finally
    FreeAndNilSafe(srcOrm);
  end;
  {some code}
  if (tmpAddBatch.Count>0) or (tmpUpdBatch.count>0) then
    begin
      {some code}
      if tmpAddBatch.Count>0 then
        sendResult := LocalDBOrm.Orm.BatchSend(tmpAddBatch)
      else
        sendResult := HTTP_SUCCESS;
      if sendResult=HTTP_SUCCESS then
        if tmpUpdBatch.Count >0 then
          sendResult := LocalDBOrm.Orm.BatchSend(tmpUpdBatch)
        else
          sendResult := HTTP_SUCCESS;
      {check for unsuccessfull send removed for this example}
      if sendresult=HTTP_SUCCESS then
        LocalDBOrm.Orm.commit;
     end;   
end;

profh

Member

Registered: 2010-07-02

Posts: 161

Re: TRecordVersion doesn't work as expected

if dstOrm.ID > 0 then
            tmpUpdBatch.Update(dstOrm)
          else
            tmpAddBatch.Add(dstORM, true);

try it as follow:

          if dstOrm.ID > 0 then
            tmpAddBatch.Delete(dstOrm.IDValue);
          
          tmpAddBatch.Add(dstORM, true);

Last edited by profh (2023-05-31 02:27:06)

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

Re: TRecordVersion doesn't work as expected

          if dstOrm.ID > 0 then
            tmpAddBatch.Delete(dstOrm.IDValue);
          
          tmpAddBatch.Add(dstORM, true);

Thanks for idea, but i have 2 problems with this solution:

1. For some classes i need to keep ID for dstORM if it already exists for reference integrity
2. I synchronize my master db with slave db periodically, so i have this problem: https://github.com/synopse/mORMot2/issues/147
  (synchronize doesn't work if i create record on master db, delete it and then synchronize with slave db - because it tries to delete record which doesn't exist in slave db yet)

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

Not sure what you try to accomplish with the Version field. I use it for master slave replication without any issues. Never checked the numbers but the result of the replication.
It is difficult helping you with this fragment of code. Please provide a working example. But please follow the forum rules and do not copy long code directly into the forum.
I do not understand your batch approach. Please check the documentation 12.3. The batch process is pretty straightforward and works for me.

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

Re: TRecordVersion doesn't work as expected

Not sure what you try to accomplish with the Version field. I use it for master slave replication without any issues. Never checked the numbers but the result of the replication.
It is difficult helping you with this fragment of code. Please provide a working example. But please follow the forum rules and do not copy long code directly into the forum.
I do not understand your batch approach. Please check the documentation 12.3. The batch process is pretty straightforward and works for me.

Hello and thanks for your answer.

First of all, sorry for my very lame English.

I use Version field for periodical replication from my master database to slave database(s) (from master read-write server to read-only server).
I use batch because data in srcorm are came from another server(s), then i process them and store correspondent data in master orm. There is batch of data in srcorm, some data may result in an dstorm entry being added, some may result in an dstorm
entry being updated, but i need to process all of them in one transaction for many reasons.

Here is a working example with results in csv: https://gist.github.com/avavdoshin/fd1d … f8f0638691
Record versions in this example doesn't changes for updates, so this changes will not be replicated to slave database.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: TRecordVersion doesn't work as expected

Here is a working example with results in csv: https://gist.github.com/avavdoshin/fd1d … f8f0638691
Record versions in this example doesn't changes for updates, so this changes will not be replicated to slave database.

I think you are using Batch incorrectly in your example. In mORMot "ex" and "test" directory, or in this forum post, you can find examples of how to use TRestBatch. Just search for "TRestBatch.Create", "BatchStart" or "BatchSend". Use TRestBatch in your example exactly as they are used in mORMot. Then compare the results and report it here. Or someone else from the forum can help. Maybe I have some time on the weekend, then I can also look again. Otherwise you have to wait until Arnaud is back.

With best regards
Thomas

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

Thank you for sending your example. That makes your question more clear.
I did some tests. It looks like that BatchUpdate does not update the Version field. This is a highly optimized function and focuses on the „simple fields“ and time fields only.
You could use Update instead. I know that Update is slower than BatchUpdate. But at least the Version field is updated correctly.

Last edited by mdoyle (2023-06-04 12:06:08)

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

I created some tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.
Hope that helps finding the issue and a fix.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: TRecordVersion doesn't work as expected

I created some tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.

I think your upload is not complete. The file Test1.json is missing.

With best regards
Thomas

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

Sorry, you are right. It is the one which is created in test.orm.sqlite3.pas.
I updated the gist. Please check again https://gist.github.com/martin-doyle/00 … ceac4698fc.

tbo

Member

Registered: 2015-04-20

Posts: 353

Re: TRecordVersion doesn't work as expected

I updated the gist. Please check again https://gist.github.com/martin-doyle/00 … ceac4698fc.

Delphi 11.3, mORMot 2.1.5499
I have only tested case TTestReplicationSimple "AddCase([TTestReplicationSimple])" for the first time. I get this error reported:

EInOutError {Message:"E/A-Fehler 6"} [Main] at 6c74b3

When I run all three test cases, I get this error reported:

...
fail  #4064 SameRecord ID Master: 107 Slave: 107 
fail  SynReplicationTests.TTestReplicationBatch(02fea050) Replication batch - Test batch update [SameRecord ID Master: 107 Slave: 107] 
EXC   EInOutError {Message:"E/A-Fehler 6"} [Main] at 6b84b3

I think every test case should pass without exception. Sorry, have no time to take a closer look in the next few days.

With best regards
Thomas

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

Sorry again. I apologize for the inconvenience.
I tested on Linux with FPC. For Windows {$apptype console} is missing. Updated the files again. Hope it works now. Thank you for taking care.

 Replication test suite
  ------------------------


1. Test suite

 1.1. Replication simple:
  - Test add: 45 assertions passed  46.93ms
  - Test update: 97 assertions passed  14.95ms
  - Test delete: 72 assertions passed  13.42ms
  Total failed: 0 / 214  - Replication simple PASSED  80.92ms

 1.2. Replication batch:
  - Test batch add: 4,011 assertions passed  42.88ms
!  - Test batch update: 10 / 7,038 FAILED  55.69ms
  - Test batch delete: 6,968 assertions passed  56.58ms
  Total failed: 10 / 18,017  - Replication batch FAILED  159.90ms

 1.3. Replication special:
  - Test add ten and delete first record: 77 assertions passed  12.38ms
!  - Test add ten and delete last record: 3 / 77 FAILED  11.63ms
!  - Test add one and delete record: 1 / 13 FAILED  11.41ms
  Total failed: 4 / 167  - Replication special FAILED  39.53ms


Generated with: Delphi 11.3 Alexandria 32 bit Win compiler

Time elapsed for all tests: 282.22ms
Performed 2023-06-05 21:33:25 by mdoyle on WIN11

Total assertions failed for all test suits:  14 / 18,398
! Some tests FAILED: please correct the code.

Last edited by mdoyle (2023-06-05 19:47:55)

mdoyle

Member

Registered: 2020-11-29

Posts: 25

Re: TRecordVersion doesn't work as expected

It looks like BatchAdd and BatchDelete update the Version field but BatchUpdate does not..

Please find the tests in https://gist.github.com/martin-doyle/00 … ceac4698fc.

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

Re: TRecordVersion doesn't work as expected

It looks like BatchAdd and BatchDelete update the Version field but BatchUpdate does not.

Arnaud, can you please try to fix this issue?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,662

Website

Re: TRecordVersion doesn't work as expected

Sorry for the delay.

Please try with https://github.com/synopse/mORMot2/commit/e68d137853d

Now BatchUpdate() works with TRecordVersion, at least in the newly introduced regression tests.

avavdoshin

Member

Registered: 2022-11-16

Posts: 25

Re: TRecordVersion doesn't work as expected

Now BatchUpdate() works with TRecordVersion, at least in the newly introduced regression tests.

Thank you so much!
I'll try ASAP