#1 Re: mORMot 2 » MGet fails silently if Content-Length unknown » 2024-02-08 18:56:23

vs

Incredible. Thank you!
I'll try to do some tests.

#2 Re: mORMot 2 » MGet fails silently if Content-Length unknown » 2024-02-07 22:50:12

vs

Great as always, ab!
Thank you!

Generally speaking also plus FTP, FTPS, and SFTP wink

ftp://[user[:password]@]server[:port]/path/to/remote/resource.mpeg
sftp://[user[:password]@]server[:port]/path/to/remote/resource.mpeg

From
https://ffmpeg.org/ffmpeg-protocols.html#ftp

#3 Re: mORMot 2 » MGet fails silently if Content-Length unknown » 2024-02-07 18:45:38

vs

I'm afraid to be wrong, but maybe it makes sense to add the ability to download files over SMB to MGet.
Or move this feature into a separate utility, something like

smbget - wget-like utility for download files over SMB

https://manpages.ubuntu.com/manpages/tr … get.1.html

#4 Re: mORMot 2 » RunRedirect and TOnRedirect small confusion of descriptions » 2024-02-07 17:56:10

vs

Humm, you are rigth. Testing with "ping" works fine.
But with small test program like

 
begin
  for i := 10 to 80 do
    begin
      write(IntToStr(i) + ',');
      sleep(200);
    end;
end. 

I get this log

Start at: 07.02.2024 19:54:58
OnRedirect at: 07.02.2024 19:55:06
10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52
OnRedirect at: 07.02.2024 19:55:12
,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,
Exit Code: 0
function TFoo.OnRedirect(const text: RawByteString; pid: cardinal): boolean;
begin
  Result := False;
//  Writeln(pid);
  if text <> '' then
    begin
      Writeln('OnRedirect at: ' + DateTimeToStr(Now));
      Writeln(text);
    end;
end;

#6 Re: mORMot 2 » RunRedirect and TOnRedirect small confusion of descriptions » 2024-02-06 19:55:52

vs

As always, thanks for the lightning-fast response, ab! ))


Add.

Is it possible to add RedirectOutput execution after a certain timeout rather than after the buffer is full?
For long processes with percentage progress, a lot of time passes until the buffer is filled out and the data is transferred to TOnRedirect.
Thanks!

#7 mORMot 2 » RunRedirect and TOnRedirect small confusion of descriptions » 2024-02-06 18:55:22

vs
Replies: 6

Hi!

mormot.core.os

  // - should return true to stop the execution, or false to continue
  .....
  // - the raw process ID (dword on Windows, cint on POSIX) is also supplied
  TOnRedirect = function(const text: RawByteString; pid: cardinal): boolean of object;


  // - will optionally call onoutput() to notify the new output state
  // - can abort if onoutput() callback returns false, or waitfordelayms expires
  ....
  function RunRedirect(const cmd: TFileName; exitcode: PInteger = nil;


Actually, it works fine as described for TOnRedirect. i.e. when the result is false then execution continue.

#8 Re: mORMot 2 » IDocList/IDocDict JSON for Delphi and FPC » 2024-02-02 23:51:05

vs

Thank you Arnaud for your daily work and for sharing your knowledge and experience with us!
Great job!

Regarding your question in the blog article about the magic number 1.0594631 is the twelfth root of 2, if I'm not mistaken.
And your hint about programming music suggests that there are 12 semitones in an octave and each semitone differs from its neighbors by 1.0594631 (the twelfth root of 2).
I guessed? )))))

#9 Re: Free Pascal Compiler » VsCode as editor » 2024-01-21 22:33:26

vs
macfly wrote:

Yes, I can create a tutorial.

With some examples of Tasks to do the builds, compilation, etc. This part was where I invested the most time, but it wasn't complicated.

Maybe I missed something. Is there a link to your tutorial? Thank you!

#10 Re: mORMot 1 » Firebird transactions » 2024-01-21 17:10:37

vs

Do you change line 616 cMaxStm = 50 in mormot.db.sql.ibx.

Yes, 48 and below working fine ))). I added the numbers for Ibx firebird 25 to my previous post.

Previous test on 2.5 pass without problem.

Because, as I wrote earlier, embedded version of fbclient.dll from firebird 5 installation was used instead of connecting to 2.5 firebird server

Edited: Or execute statement size exceeded  64K  limit on Fb2.5, Fb3 and above have limit of 10MB.

Yes, that's right, https://www.firebirdfaq.org/faq197/

Do you try to add define LEGACYFIREBIRDAPIONLY to force using of legacy api with Fb4/5?

Yes, but without success.

! EXC   EIBInterBaseError {Message:"Engine Code: 335544351\r\nunsuccessful metadata update"} [Main] at 0100375a46 client/include/wloadlibrary.inc tfbclientapi.ibdatabaseerror (209)

#11 Re: mORMot 1 » Firebird transactions » 2024-01-20 18:41:03

vs

More accurate test results.

Running tests using Synopse mORMot framework 2.2.6792, compiled with Free Pascal 3.2.2 64 bit, against SQLite 3.44.2, on Windows 11 64bit (10.0.22631), at 2024-01-20 20:02:16.

Insertion speed (rows/second)                                           Read speed (rows/second)
                  Direct      Batch       Trans       Batch Trans   |                     By one      All Virtual All Direct
Zeos firebird 25  2739        76921       5911        58403         |   Zeos firebird 25  7811        407398      427313
Zeos firebird 4   2471        57124       7942        41158         |   Zeos firebird 4   7868        391282      412609
Zeos firebird 5   1960        70443       7295        49310         |   Zeos firebird 5   8212        413479      498529
Ibx firebird 25   756         43030       6628        40509         |   Ibx firebird 25   1356        100883      117096
Ibx firebird 4    1824        33112       7650        42821         |   Ibx firebird 4    1617        138475      104979
Ibx firebird 5    1367        60364       6479        40248         |   Ibx firebird 5    1791        235510      211555

A separate test was run for each driver and each firebird server version.

Ibx firebird 25 did not pass the test due to an error: Solved by changing line 616 cMaxStm = 50 in mormot.db.sql.ibx at cMaxStm = 45

Project PerfTestConsole raised exception class 'EIBInterBaseError' with message:
Engine Code: 335544569
Dynamic SQL Error
-SQL error code = -204
-Implementation limit exceeded
-block size exceeds implementation restriction

 In file 'FBClientAPI.pas' at line 412:

#12 Re: mORMot 1 » Firebird transactions » 2024-01-20 01:43:00

vs
ttomas wrote:

2. Correct, this is the reason I create ibx mormot driver. Main goal is to have 0 active transaction if nobody is connected to app/server.

Great, thanks for sharing your experience!

ttomas wrote:

{$define ZEOSTRANS} is only for testing, not recomended for production.

Ок, thanks. Yes, it is clear!

tomas wrote:

@vs, do you success build with delphi, if yes delphi version please. @ab alearedy commit changes in trunk.

Yes, with my local edits MVCServerFirebirdIbx compiling successful for both (Lazarus and Delphi 12) and working as expected, except FTS.
Tomorrow I'll try to play with ..\ex\extdb-bench\PerfTestConsole.dpr for both (Lazarus and Delphi 12) with the latest ab's commit.

I suspect that the performance figures that I gave a little earlier are not entirely correct.
Most likely the embedded version of fbclient.dll server from firebird 5 installation was used instead of connecting to a particular version of firebird server.
But this is just a guess, I'll try to figure it out tomorrow.

Add.
By the way. Thanks madorin, there is a pretty good library with precise transaction controls for Firebird. Separate for reading and writing transactions, at least.
Maybe this could be a starting point for the next Firebird driver for mORMot 2
https://github.com/madorin/fibplus/
https://github.com/madorin/fibplus/commits/master

#13 Re: mORMot 1 » Firebird transactions » 2024-01-19 21:43:32

vs

ab, thanks for the tip.
Still, there may be a small typo, line 10 in ..\ex\extdb-bench\PerfTestConsole.dpr

  {../../src/$R mormot.win.default.manifest.res} 

ttomas, thank you for the clarification!

1 and 3 - clear.

2. A few words about this. As I wrote earlier in this thread, this long transaction leads to an increase records versioning for frequently updated tables, which for systems running 24/7 after some time, leads to a dramatically decrease performance for both reading and writing. Firebird 4 solved this problem at some point, but still, a long transaction that opens when the server starts, in my opinion, is not a very correct configuration. Therefore fbintf seems more suitable.

#14 Re: mORMot 1 » Firebird transactions » 2024-01-19 12:27:54

vs

Some test results

Running tests using Synopse mORMot framework 2.2.6749, compiled with Free Pascal 3.2.2 64 bit, against SQLite 3.44.2, on Windows 8 64bit (6.2.9200), at 2024-01-19 13:28:17.

Insertion speed								|	Read speed
 			Direct	Batch	Trans	Batch Trans		| 				By one	All Virtual	All Direct
Zeos firebird 25	2265	60781	16408	65852			|	Zeos firebird 25	10884	366770		399840
Zeos firebird 4		2187	55836	6949	63136			|	Zeos firebird 4		6944	373538		412864
Zeos firebird 5		2473	65590	8084	52379			|	Zeos firebird 5		8154	403177		490196
Ibx firebird 25		2366	43955	15503	40030			|	Ibx firebird 25		3850	164076		120290
Ibx firebird 4		1296	37603	3675	49165			|	Ibx firebird 4		1798	106503		99975
Ibx firebird 5		1275	30746	4452	40146			|	Ibx firebird 5		1967	221028		214850

For some reason log shows Windows 8 64bit, but in fact Windows 11 64bit.

#15 Re: mORMot 1 » Firebird transactions » 2024-01-19 02:05:37

vs
ttomas wrote:

Yes need to be changed to string. I will try to create push request.

Thank you so much.

Fixing these two issues
for Lazarus:

MVCServerFirebirdIbx.dpr(54,17) Error: identifier idents no member "CreateDescendingPK"
MVCServerFirebirdIbx.dpr(95,1) Error: Can't open resource file "D:\Projects\Delphi\Components\Synopse\mORMot2-2.0.stable\ex\mvc-blog\mormot.win.default.manifest.res"

and
for Delphi:

[dcc32 Error] mormot.db.sql.ibx.pas(120): E2010 Incompatible types: 'UTF8String' and 'string'
[dcc32 Error] MVCServerFirebirdIbx.dpr(54): E2003 Undeclared identifier: 'CreateDescendingPK'

MVCServerFirebirdIbx compiling successful for both and working as expected, except FTS

#16 Re: mORMot 1 » Firebird transactions » 2024-01-18 14:51:12

vs

fbintf1.4-0 already installed on Delphi.

For Lazarus:
MVCServerFirebirdIbx.dpr(54,17) Error: identifier idents no member "CreateDescendingPK"
MVCServerFirebirdIbx.dpr(95,1) Error: Can't open resource file "D:\Projects\Delphi\Components\Synopse\mORMot2-2.0.stable\ex\mvc-blog\mormot.win.default.manifest.res"

mormot.db.sql.ibx.pas(119):
property FirebirdLibraryPathName: RawUtf8
but
mormot.db.sql.ibx.pas(87):
fFirebirdLibraryPathName: string;

#17 Re: mORMot 1 » Firebird transactions » 2024-01-18 00:22:27

vs

Two small inaccuracies

commit 2.2.6749

[dcc32 Error] mormot.db.sql.ibx.pas(120): E2010 Incompatible types: 'UTF8String' and 'string'
[dcc32 Error] MVCServerFirebirdIbx.dpr(54): E2003 Undeclared identifier: 'CreateDescendingPK'

#18 Re: mORMot 2 » LDAP Active Directory list users » 2024-01-05 17:30:24

vs

For today's commit, "2.2.6617" works fine. Thank you!
But strangely, I followed https://github.com/synopse/mORMot2/commits/master every day and did not see any changes regarding this issue.
Probably I was inattentive and missed it))))
That's why I decided to bother you today.

#19 Re: mORMot 2 » LDAP Active Directory list users » 2024-01-05 14:50:51

vs

Hi Ab.
Was the log I sent useful?

#20 Re: mORMot 2 » LDAP Active Directory list users » 2023-11-20 21:14:11

vs
ab wrote:

Did you try both OU=1\+2 MEDIA and OU=1+2 MEDIA content?

Yes. Same results.

ab wrote:

Can you enable the application console, and define a ASNDEBUG conditional and report the output to the console during the search?

Sorry. I do not have permission to publish the log of communication with the corporate AD server. Can I send the log to your email address?

#21 Re: mORMot 2 » LDAP Active Directory list users » 2023-11-20 20:12:13

vs

It is MS.
I tried it, but unfortunately parameter passes, but TLdapClient.GetUserInfo returns false.
I.e. Search returns true but SearchResult.Count = 0.

#22 Re: mORMot 2 » LDAP Active Directory list users » 2023-11-19 12:59:49

vs

Hi Ab.

The call

TLdapClient.GetGroupInfo('', vBaseDN, vLdapGroup, '', True);

Successfully returns the corresponding group (vLdapGroup) with N members.
One of the member from this group has DN returned by vLdapGroup.member[k] kind of 'CN=Super User,OU=1\+2 MEDIA,OU=Domain Users,DC=1plus2,DC=corp';

To determine sAMAccountName for this user, the next call

TLdapClient.GetUserInfo('', vLdapGroup.member[k], '', vLdapUser);

returns an error
Invalid input name: CN=Super User,OU=1\+2 MEDIA,OU=Domain Users,DC=1plus2,DC=corp

If I remove \ before +, then such a user does not exist.
The actual name of this OU is "1+2 Media" without quotes.

How to correctly call a TLdapClient.GetUserInfo with such a parameter?

In continuation of our conversation, this is not a problem of non ASCII characters, rather this is a problem of special characters for AD.

Thanks a lot.

#23 Re: mORMot 1 » Firebird transactions » 2021-08-08 00:01:02

vs

It looks like firebird 4 solves that issue of commit retaining transaction

#24 Re: mORMot 1 » Firebird transactions » 2021-05-31 17:21:40

vs

I think it is a bad idea to set up an AutoCommit for transaction for a Rest Server that has as backend firebird database.
Below is a snippet from The Firebird Book: A Reference for Database Developers Copyright © 2004 by Helen Borrie and IBPhoenix

COMMIT with the RETAIN Option

The optional RETAIN [SNAPSHOT] extension to the COMMIT statement causes the server to retain a “snapshot” of the physical transaction’s context at the time the statement is executed and start a new transaction as a clone of the committed one. If this so-called soft commit is used on a SNAPSHOT or SNAPSHOT TABLE STABILITY transaction, the cloned transaction preserves the same snapshot of the data as the original transaction had when it started.

Although it does commit the work permanently and thus change the state of the database, COMMIT RETAIN (CommitRetaining) does not release resources. In the lifespan of a logical task that comprises many repetitions of a similar operation, cloning the context reduces some of the overhead that would be incurred by clearing resources each time with COMMIT, only to allocate the identical resources all over again when a new transaction is started. In particular, it preserves the cursors on sets selected and currently “open.”

The same TID remains active in the TSB and never appears there as “committed.” For this reason, it is often referred to as soft commit, in contrast with the “hard” commit performed by an unmodified COMMIT statement. Each soft commit is like a savepoint with no return. Any subsequent ROLLBACK reverses only the changes that have been posted since the last soft commit. The benefit of the soft commit is that it makes life easy for programmers, especially those using components that implement “scrolling dataset” behavior. It was introduced to support the data grid user interface favored by many users of the
Borland Delphi development environment. By retaining the transaction context, the application can display a seamless before-to-after transition that reduces the effort the programmer would otherwise need to invest in starting new transactions, opening new cursors, and resynchronizing them with row sets buffered on the client. Data access implementations frequently combine posting a single update, insert or delete statement with an immediate COMMIT RETAIN in a mechanism that is dubbed “Autocommit.” It is common for interface layers that implement Autocommit capability to “dumb out” explicit control of transactions by starting one invisibly in situations where the programmer-written code attempts to pass a statement without first starting a transaction itself.

Explicit transaction control is worth the extra effort, especially if you are using a connectivity product that exploits the flexible options provided by Firebird. In a busy environment, the COMMIT RETAIN option can save time and resources, but it has some serious disadvantages:

•  A snapshot transaction continues to hold the original snapshot in its view, meaning the user does not see the effects of committed changes from other
transactions that were pending at the start of the transaction.

•  As long as the same transaction continues being committed with RETAIN, resource “housekeeping” on the server is inhibited, resulting in excessive growth
of memory resources consumed by the TSB. This growth progressively slows down performance, eventually “freezing” the server and, under adverse operating system conditions, even causing it to crash.

•  No old record versions made obsolete by operations committed by a COMMIT RETAIN transaction can be garbage collected as long as the original transaction
is never “hard committed.”

#25 Re: mORMot 1 » Firebird transactions » 2021-05-14 16:34:11

vs

@AB
Thank you for the clarification about AutoFree().

@Michael,
Several tests have shown that only in such configuration record versions are not created.
Please, see an sample. vConnection.AutoCommit := False; and vConnection.Commit; After each Upadte/Delete request.

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils, ZConnection;

var
  vConnection: TZConnection;
  i: integer;

begin
  vConnection := TZConnection.Create(nil);
  try
    vConnection.HostName := 'localhost';
    vConnection.Database := 'd:\Projects\Tests\REC_VERSION_TEST.FDB';
    vConnection.Protocol := 'firebird';
    vConnection.User := 'sysdba';
    vConnection.Password := 'masterkey';
//    vConnection.Properties.Add('hard_commit=true');
//    vConnection.AutoCommit := True;
    vConnection.AutoCommit := False;

    vConnection.Connect;
    i := 0;
    while True do
      begin
        inc(i);
        if vConnection.ExecuteDirect('UPDATE SIMPLERECORD SET CHANGETIME = CURRENT_TIMESTAMP') then
          begin
            vConnection.Commit;
            Writeln('Udate #' + i.ToString);
          end;
        sleep(100);
      end;
    Readln;
  finally
    vConnection.Free;
  end;
end.

When vConnection.AutoCommit := True, value of field *CHANGETIME* is changing.
But, it looks like CommitRetaining not free the transaction environment, so not reset the ID of the oldest active transaction.

@AB
Is it possible at the ORM level to control the transactions behavior?
Thank you!

#26 Re: mORMot 1 » Firebird transactions » 2021-05-11 11:13:20

vs

Please follow the forum rules and don't post huge piece of code or logs.

I'm sorry for that. I will take it into account.
Is 120 lines of code a huge? Can I post a tweaked example here or do I need to save it to an external resource?

Anyway, changing AutoFree () to explicit Create/Free did not solve the problem with record versions.

@AB,
do you recommend not using AutoFree () anymore? (https://synopse.info/forum/viewtopic.ph … 947#p33947)
It was a very useful thing.
Thanks!

#27 Re: mORMot 1 » Firebird transactions » 2021-05-10 22:38:09

vs

After Server restarting

...
        Oldest transaction      119
        Oldest active           120
        Oldest snapshot         120
        Next transaction        122
...
SIMPLERECORD (128)
...
     Average version length: 9.00, total versions: 113, max versions: 113

So, Oldest active           120 was changed
but total versions: 113        remained the same since there was no read from the table SIMPLERECORD
When the client was started second time

Udate #15
Udate #16
Udate #17
Udate #18
Udate #19

PS>

...
        Oldest transaction      120
        Oldest active           121
        Oldest snapshot         121
        Next transaction        142
...
SIMPLERECORD (128)
...
    Average version length: 9.00, total versions: 19, max versions: 19

So, sweep has been performed.
But, the problem is that the server keeps an active transaction for the first connection, which gives rise record versions.

#28 Re: mORMot 1 » Firebird transactions » 2021-05-10 22:06:49

vs

@Michael,
@AB
Thanks alot for your help resolving this issue!
But from my point of view issue still exists.
I'm using mORMot 1.18.6286 (mORMot_and_Open_Source_friends_2021-05-06_064801_575871d143) and zeoslib-code-0-r7537-trunk
Source code of test app

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils, SynCommons, system.Classes, SynDB, SynTable, ZDbcIntfs, SynDBZeos, mORMot, mORMotDB, mORMotSQLite3,
  SynSQLite3Static, mORMotHttpServer, mORMotHttpClient;

type
  TSimpleRecord = class(TSQLRecord)
  private
    FChangeTime: TDateTime;
  published
    property ChangeTime: TDateTime read FChangeTime write FChangeTime;
  end;

var
  vCommandLine: TCommandLine;
  vMode: RawUTF8;

const
  cServerPort = '8080';
  cConnectStr = '{"Kind":"TSQLDBZEOSConnectionProperties","ServerName":"firebird://",' +
                '"DatabaseName":"d:\\Projects\\Tests\\REC_VERSION_TEST.FDB","User":"SYSDBA","Password":"HuYxjMYsqbe2"}'; //HuYxjMYsqbe2  = masterkey

procedure RunServer;
var
  vDBConnectionParams: TSQLDBZEOSConnectionProperties;
  vModel: TSQLModel;
  vRestServer: TSQLRestServerDB;
  vHttpServer: TSQLHttpServer;
  vSimpleRecord: TSimpleRecord;
begin
  Writeln('RunServer');
  TAutoFree.One(vDBConnectionParams, TSQLDBZEOSConnectionProperties.CreateFromJSON(cConnectStr) as TSQLDBZEOSConnectionProperties);
//  vDBConnectionParams.ZeosURL.Properties.Add('hard_commit=true');
  vModel := TSQLModel.Create([TSimpleRecord]);
  VirtualTableExternalRegisterAll(vModel, vDBConnectionParams, True);
  TAutoFree.One(vRestServer, TSQLRestServerDB.Create(vModel, ':memory:'));
  vRestServer.Model.Owner := vRestServer;
  vRestServer.AcquireExecutionMode[execORMGet] := amBackgroundORMSharedThread;
  vRestServer.AcquireExecutionMode[execORMWrite] := amBackgroundORMSharedThread;
  vRestServer.CreateMissingTables;
  if vRestServer.TableRowCount(TSimpleRecord) = 0 then
    begin
      TAutoFree.One(vSimpleRecord, TSimpleRecord.Create);
      vSimpleRecord.ChangeTime := Now;
      vRestServer.Add(vSimpleRecord, True);
    end;
  TAutoFree.One(vHttpServer, TSQLHttpServer.Create(cServerPort, [vRestServer], '+', useHttpApiRegisteringURI));
  vHttpServer.AccessControlAllowOrigin := '*';
  Writeln('Server is now running on http://localhost:' + cServerPort + '/root'#13#10#13#10+'Press [Enter] to quit');
  Readln;
end;

procedure RunClient;
var
  i: Integer;
  vModel: TSQLModel;
  vHttpClient: TSQLHttpClient;
  vSimpleRecord: TSimpleRecord;
begin
  Writeln('RunClient');
  vModel := TSQLModel.Create([TSimpleRecord]);
  TAutoFree.One(vHttpClient, TSQLHttpClient.Create('localhost', cServerPort, vModel));
  vHttpClient.Model.Owner := vHttpClient;
  TAutoFree.One(vSimpleRecord, TSimpleRecord.Create);
  vHttpClient.Retrieve(1, vSimpleRecord);
  i := 0;
  while true do
    begin
      inc(i);
      vSimpleRecord.FChangeTime := now;
      if vHttpClient.Update(vSimpleRecord) then
        Writeln('Udate #' + i.ToString);
      Sleep(100);
      if ConsoleKeyPressed(13) then break;
    end;
  Readln;
end;

begin
  TAutoFree.One(vCommandLine, TCommandLine.Create);
  vMode := vCommandLine.AsUTF8('mode', 'S', '');
  case vMode[1] of
    'S', 's': RunServer;
    'C', 'c': RunClient;
  end;
end.

Output of Project1.exe -mode C

....
Udate #96
Udate #97
Udate #98
Udate #99
Udate #100
Udate #101
Udate #102
Udate #103
Udate #104
Udate #105
Udate #106
Udate #107
Udate #108
Udate #109
Udate #110
Udate #111
Udate #112
Udate #113

PS>

Output of gstat -u sysdba -p masterkey -r d:\Projects\Tests\REC_VERSION_TEST.FDB

$ gstat -u sysdba -p masterkey -r d:\Projects\Tests\REC_VERSION_TEST.FDB

Database "d:\Projects\Tests\REC_VERSION_TEST.FDB"
Database header page information:
        Flags                   0
        Checksum                12345
        Generation              120
        Page size               4096
        ODS version             11.2
        Oldest transaction      1
        Oldest active           1
        Oldest snapshot         1
        Next transaction        118
        Bumped transaction      1
        Sequence number         0
        Next attachment ID      1
        Implementation ID       16
        Shadow count            0
        Page buffers            0
        Next header page        0
        Database dialect        3
        Creation date           May 10, 2021 23:51:00
        Attributes              force write

    Variable header data:
        *END*


Database file sequence:
File d:\Projects\Tests\REC_VERSION_TEST.FDB is the only file

Analyzing database pages ...
SIMPLERECORD (128)
    Primary pointer page: 168, Index root page: 169
    Average record length: 17.00, total records: 1
    Average version length: 9.00, total versions: 113, max versions: 113
    Data pages: 1, data page slots: 1, average fill: 73%
    Fill distribution:
         0 - 19% = 0
        20 - 39% = 0
        40 - 59% = 0
        60 - 79% = 1
        80 - 99% = 0

    Index NDXSIMPLERECORDID (1)
        Depth: 1, leaf buckets: 1, nodes: 1
        Average data length: 9.00, total dup: 0, max dup: 0
        Fill distribution:
             0 - 19% = 1
            20 - 39% = 0
            40 - 59% = 0
            60 - 79% = 0
            80 - 99% = 0

    Index RDB$PRIMARY1 (0)
        Depth: 1, leaf buckets: 1, nodes: 1
        Average data length: 9.00, total dup: 0, max dup: 0
        Fill distribution:
             0 - 19% = 1
            20 - 39% = 0
            40 - 59% = 0
            60 - 79% = 0
            80 - 99% = 0

Note for this value, please.

...
        Oldest transaction      1
        Oldest active           1
        Oldest snapshot         1
        Next transaction        118
...
SIMPLERECORD (128)
...
    Average version length: 9.00, total versions: 113, max versions: 113

Client was closed.
Server still running and hold
Oldest active            1
So, I have
total versions: 113, max versions: 113 equal to the number of updates to the record in the SIMPLERECORD table
Maybe I'm missing something?
Thanks!

#29 Re: mORMot 1 » Firebird transactions » 2021-04-26 16:40:40

vs

I found a solution. Switching to the FireDAC library.
If I try this code,

  while true do
    begin
      inc(i);
      vSQLRecordSample.LastLogin := now;
      if FRestServer.Update(vSQLRecordSample) then
        Writeln('Udate #' + i.ToString);
      Sleep(100);
      if ConsoleKeyPressed(13) then break;
    end;

the number of record versions is 1!

#30 Re: mORMot 1 » Firebird transactions » 2021-04-26 16:32:08

vs
ab wrote:

I suppose the automatic transaction used within a BATCH process works as expected, right?

I tried code like this

  while true do
    begin
      inc(i);
      vBatch := TSQLRestBatch.Create(FRestServer, TSQLRecordSample, 5);
      try
        vSQLRecordSample.LastLogin := now;
        vBatch.Update(vSQLRecordSample);
        vBatch.Update(vSQLRecordSample);
        vBatch.Update(vSQLRecordSample);
        vBatch.Update(vSQLRecordSample);
        vBatch.Update(vSQLRecordSample);
        FRestServer.BatchSend(vBatch);
      finally
        vBatch.Free;
      end;
        Writeln('Udate #' + i.ToString);
      Sleep(300);
      if ConsoleKeyPressed(13) then break;
    end;

Result remains the same. Amount of record versions is growing at each FRestServer.BatchSend

#31 Re: mORMot 1 » Firebird transactions » 2021-04-20 12:43:37

vs

Hi Michael!
Thanks for your help.

I can't see a zeos regression. Hope you use minimum version v7.2.10 or v8.0 from trunk.

Yes, I am using version 7.2.10-stable build at 2021-01-12 08:55:31

If you have such "old" transactions than you have open cursors somewhere propably

There is no explicit cursor control in my code.

Take care you read all results until no more row is returned. Another cause of many ransactions might be nested transactions with multiple connections

Only one record is read and only one client connection is established.

My test has a server and one client that executes code like this

    TAutoFree.One(vMySQLRecord, TMySQLRecord.Create);
    FHttpClient.Retrieve(1, vMySQLRecord);
    i := 0;
    while true do
      begin
        inc(i);
        vMySQLRecord.LastCheckTime := now;
        FHttpClient.Update(vMySQLRecord);
        Writeln('Udate #' + i.ToString);
        Sleep(300);
      end;

Actually there are two connections to the database.
First from the main server thread and second from the ThreadSafeConnection, which occurs when the client is selecting data for vMySQLRecord (FHttpClient.Retrieve(1, vMySQLRecord);)

Moreover, if after some time on the server side execute

FDBConnectionParams.ThreadSafeConnection.Disconnect;

For some reason first connections will close and client continues to successfully update vMySQLRecord through the second connection.
And the saddest thing, the ID of the oldest active transaction is equal to the ID of first transaction of the second connection. This is the reason lot of record versions.

Note a second call to IZConnection.StartTransaction creates a savepoint. First call to IZConnection.Commit/Rollback just releases/rollback the savepoint but not the transaction you wanna close.

This is interesting, but the data in the database are updated, therefore commit occurs and savepoint is released.

According the "hard_commit" option: If you are adding that option after creating a transaction then the option is a NOOP.

FDBConnectionParams.ZeosURL.Properties.Add('hard_commit=true')

This code is executed before the call

//SynDBZeos
constructor TSQLDBZEOSConnection.Create(aProperties: TSQLDBConnectionProperties);
var url: TZURL;
begin
  inherited Create(aProperties);
  url := (fProperties as TSQLDBZEOSConnectionProperties).fURL;
  fDatabase := DriverManager.GetConnectionWithParams(url.URL,url.Properties);
  // EG: setup the connection transaction behavior now, not once Opened in Connect
  //fDatabase.SetReadOnly(false); // is default
  // about transactions, see https://synopse.info/forum/viewtopic.php?id=2209
  //fDatabase.SetAutoCommit(true); // is default
  fDatabase.SetTransactionIsolation(tiReadCommitted); // will be swapped to tiSerialiable for SQLite
end;

In this case url.Properties.Commatext = 'codepage=UTF8,hard_commit=true'

#32 Re: mORMot 1 » Firebird transactions » 2021-04-19 21:30:05

vs

A little more details.
When the server is restarted, the first client connection and selection from this table of course started garbage collection (delete record versions).
But for a 24/7 service this is certainly not a good solution.
The question is, how to correctly and safetly close the transaction that occurs when a ThreadSafeConnection is created?
Thanks!

Addon.
Maybe this is more related to Zeos Database Objects transaction handling than ORM.
Please give your advice.
Thanks!

#33 Re: mORMot 1 » Firebird transactions » 2021-04-19 12:09:24

vs

Unfortunately, both

FDBConnectionParams.ZeosURL.Properties.Add('hard_commit=true')

and

FDBConnectionParams.ZeosURL.Properties.Add('hard_commit=false')

same result

#34 Re: mORMot 1 » Firebird transactions » 2021-04-18 17:50:50

vs

Hi.
Faced such a case.
There is a table with 50 records.
Each record is updated every minute from FHttpClient: TSQLHttpClient, 24/7.
As a result, a lot of record version accumulated for this table, since the ID of the oldest active transaction is equal to the ID of the first update transaction for this table at server start.

Maybe someone knows how to handle this?
Thanks!

#35 Re: mORMot 1 » Error on call TSQLHttpClientWinHTTP.UpdateField » 2019-10-21 16:21:43

vs

Looks like it's already fixed.
Please check https://synopse.info/fossil/info/530adf5475e9af5

ab,
Is it by design, that in mORMotDB.TSQLRestStorageExternal.EngineUpdateField/EngineUpdateFieldIncrement no update of TRecordVersion fields?
Thanks.

Board footer

Powered by FluxBB