Posts by h.hasenack

h.hasenack · mORMot 1

Never mind. It turned out to be my own mistake: I had added the AuthGroup and AuthUser tables to the model, and also provided Trie for handleautorhization=True to the TSQLRestClientDB.Create()

Sorry - Hans

h.hasenack · mORMot 1

Tracing into this stack:

Mormot_D17.mORMotSQLite3.TSQLVirtualTableModuleSQLite3.SetDB($FE7BE000)
Mormot_D17.mORMotSQLite3.TSQLVirtualTableModuleServerDB.Create(TSQLVirtualTableExternal,$FE6213C0)
Mormot_D17.mORMotSQLite3.TSQLRestServerDB.InitializeEngine
Mormot_D17.mORMotSQLite3.TSQLRestServerDB.Create($FE7CEEE0,$FE7BE000,True)
Mormot_D17.mORMotSQLite3.TSQLRestClientDB.Create($FE7CD560,$FE7CEEE0,$FE7BE000,TSQLRestServerDB,True)
Mormot_D17.mORMotSQLite3.TSQLRestClientDB.Create($FE7CD560,nil,':memory:',TSQLRestServerDB,True,'')
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonTest.SetUp
SGMormotTestSuitePkg_D17.uTestMormotDBBaseCommon.TMormotDBBaseCommonCRUDTest.SetUp

fails to initialize for the 2nd table initialized, being the AuthGroup table.
When tracing into the "sqlite3_check" routine, it reveals the return code is 21, and an exception is raised with a _very_ informative text: "not an error"

I have no idea how to proceed from here. Tips are very welcome.

regards - Hans

h.hasenack · mORMot 1

Y're welcome!

h.hasenack · mORMot 1

SOmething tells me you'll ask me to generate a ticket for this

Hans

h.hasenack · mORMot 1

Hi Ab

I sent you a mail with changed sourcecode, but not all changed mentioned we in it. (http://synopse.info/forum/viewtopic.php?id=983)

What's the status on it? - If things are slow because of the holidays: no problem.

Hans

h.hasenack · mORMot 1

1

h.hasenack · mORMot 1

I agree with jonjbar. We have currently invested quite much in our own project using Mormot, and we would hateto lose Ab's superb support and effort.

So... for our peace of mind it would be nice to know if we can rely in Mormot for, let';s say, the next 5 years...?

Regards - Hans

h.hasenack · mORMot 1

And now, I finally have a log, created by TestSQL3.exe in native 64bit mode.

This is what it looks like:

S:\Sources\LibSource\mORMot\SQLite3\bin\D17\Win64\TestSQL3.exe 0.0.0.0 (2012-12-18 17:08:47)Host=VMOBELIX-XE3 User=Hans CPU=4*9-21-258 OS=13.1=6.1.7601 Wow64=0 Freq=3579545Environment variables=ALLUSERSPROFILE=C:\ProgramData	APPDATA=C:\Users\Hans\AppData\Roaming	CommonProgramFiles=C:\Program Files\Common Files	CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files	CommonProgramW6432=C:\Program Files\Common Files	COMPUTERNAME=VMOBELIX-XE3	ComSpec=C:\Windows\system32\cmd.exe	FP_NO_HOST_CHECK=NO	HOMEDRIVE=C:	HOMEPATH=\Users\Hans	IBREDISTDIR=C:\Users\Public\Documents\InterBase\redist\InterBaseXE3	LOCALAPPDATA=C:\Users\Hans\AppData\Local	LOGONSERVER=\\VMOBELIX-XE3	MpConfig_ProductAppDataPath=C:\ProgramData\Microsoft\Windows Defender	MpConfig_ProductCodeName=AntiSpyware	MpConfig_ProductPath=C:\Program Files\Windows Defender	MpConfig_ProductUserAppDataPath=C:\Users\Hans\AppData\Local\Microsoft\Windows Defender	MpConfig_ReportingGUID=60AB9E58-A78C-4EBF-8CA4-1675C354FB2A	MYSRC=S:\Sources	NUMBER_OF_PROCESSORS=4	OS=Windows_NT	Path=C:\Program Files (x86)\CollabNet;C:\Program Files (x86)\Embarcadero\RAD Studio\10.0\bin;C:\Users\Public\Documents\RAD Studio\10.0\Bpl;C:\Program Files (x86)\Embarcadero\RAD Studio\10.0\bin64;C:\Users\Public\Documents\RAD Studio\10.0\Bpl\Win64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseSVN\bin;S:\Sources\bpl\D17\Win32;S:\Sources\Libsource\Steema9\Compiled\Delphi17.win32\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.win32\System;S:\Sources\bpl\D17\iOS;S:\Sources\Libsource\Steema9\Compiled\Delphi17.iOS\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.iOS\System;S:\Sources\bpl\D17\Win64;S:\Sources\Libsource\Steema9\Compiled\Delphi17.win64\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.win64\SystemS:\Sources\bpl\D17\Win64;	PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC	PROCESSOR_ARCHITECTURE=AMD64	PROCESSOR_IDENTIFIER=AMD64 Family 21 Model 1 Stepping 2, AuthenticAMD	PROCESSOR_LEVEL=21	PROCESSOR_REVISION=0102	ProgramData=C:\ProgramData	ProgramFiles=C:\Program Files	ProgramFiles(x86)=C:\Program Files (x86)	ProgramW6432=C:\Program Files	PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\	PUBLIC=C:\Users\Public	SESSIONNAME=Console	SystemDrive=C:	SystemRoot=C:\Windows	TEMP=C:\Users\Hans\AppData\Local\Temp	TMP=C:\Users\Hans\AppData\Local\Temp	USERDOMAIN=VMOBELIX-XE3	USERNAME=Hans	USERPROFILE=C:\Users\Hans	windir=C:\WindowsTSynLog 1.18 2012-12-18T17:11:0820121218 17110836 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064E640 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110836 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064ED44 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F0D4 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F17A 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F1FE 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 

..... (520KByte of the same line)

So I guess some work is still to be done

h.hasenack · mORMot 1

I had to apply a similar fix to the TGDIPlys.Destroy as there was for TGDIPlus.Create:

destructor TGDIPlus.Destroy;
begin
  if fToken<>0 then begin
    if Assigned(fStartupHook.UnHook) then // Avoid AV if unassigned
      fStartupHook.UnHook(fStartupHookToken);
    Shutdown(fToken);
    fToken := 0;
  end;
  UnLoad;
end;

h.hasenack · mORMot 1

As it turns out there was a very nasty bug in SynCOmmons.pas, procedure TSynLog.LogFileHeader;

a pointer is used to dump the envoironment strinngs, but the pointer is changed by the routine before FreeEnvironmentStringsA is called. thus with the wrong pointer, causeing the ugly hang. Here's the fixed routine:

procedure TSynLog.LogFileHeader;
var Env,P: PAnsiChar;
    L: Integer;
begin
  if not QueryPerformanceFrequency(fFrequencyTimeStamp) then begin
    fFamily.HighResolutionTimeStamp := false;
    fFrequencyTimeStamp := 0;
  end;
  ExeVersionRetrieve;
  if InstanceMapFile=nil then begin
    InstanceMapFile := TSynMapFile.Create;
    GarbageCollector.Add(InstanceMapFile);
  end;
  // array of const is buggy under Delphi 5 :( -> use fWriter.Add*()
  with ExeVersion, SystemInfo, OSVersionInfo, fWriter do begin
    AddString(ProgramFullSpec);
    AddShort(#13'Host='); AddString(Host);
    AddShort(' User=');   AddString(User);
    AddShort(' CPU='); Add(dwNumberOfProcessors); Add('*');
    Add(wProcessorArchitecture); Add('-'); Add(wProcessorLevel); Add('-');
    Add(wProcessorRevision);
    AddShort(' OS='); Add(ord(OSVersion)); Add('.'); Add(wServicePackMajor);
    Add('='); Add(dwMajorVersion); Add('.'); Add(dwMinorVersion); Add('.');
    Add(dwBuildNumber);
    AddShort(' Wow64='); Add(integer(IsWow64));
    AddShort(' Freq='); Add(fFrequencyTimeStamp);
    if IsLibrary then begin
      AddShort(' Instance='); AddNoJSONEscapeString(InstanceFileName);
    end;
    Add(#13);
    AddShort('Environment variables=');
    Env := GetEnvironmentStringsA;
    P:=Env;
    while P^<>#0 do begin
      L := StrLen(PUTF8Char(P));
      if (L>0) and (P^<>'=') then begin
        AddNoJSONEscape(P,L);
        Add(#9);
      end;
      // go to end-of-string
      while P^<>#0 do
        inc(P);
      // next string
      inc(P);
    end;
    FreeEnvironmentStringsA(Env);
    CancelLastChar; // trim last #9
    Add(#13);
    AddClassName(self.ClassType); AddShort(' '+SYNOPSE_FRAMEWORK_VERSION+' ');
    AddDateTime(Now); Add(#13,#13);
  end;
  QueryPerformanceCounter(fStartTimeStamp);
  fHeaderWritten := true;
end;

h.hasenack · mORMot 1

Sigh... its hard but we're getting there!

It's a bit of a log style, so you'd probably run into the same problems in the same order.

I finally managed to compile sqlite3.c into a .o (=ELF64) object file, AND managed to compile it into TestSQL3.exe.

Nevertheless I am still under the impression that using a SQLite3.DLL in combination with the new "delayed" directive available in XE2/XE3 might make it an easier task to compile mormot for Win32 and Win64. (Ckeck Delayed loading of libraries here http://docwiki.embarcadero.com/RADStudi … _Packages)

In order to create all required obj files I run following batch file, which requires a objconv tool as delphi64 is unable to link in the .o files generated by BCC64. (seriously)
http://www.agner.org/optimize/objconv.zip

@echo off

ECHO compile sqlite3.c into obj files

rem DROP pre- existing obj files

del sqlite3*.o
del sqlite3*.obj

ECHO Generate 32bit OBJ files

bcc32 -6 -O2 -c -d -DSQLITE_ENABLE_FTS3 -u- -o sqlite3fts3.obj sqlite3.c 
bcc32 -6 -O2 -c -d -u- -o sqlite3.obj sqlite3.c

ECHO generate 64bit ELF64 OBJ files

bcc64 -c -DSQLITE_ENABLE_FTS3 -o sqlite3fts3x64.o sqlite3.c
bcc64 -c -o sqlite3x64.o sqlite3.c

ECHO convert the ELF64 to 64bit COFF files, suitable for Delphi linking with leading underscores for symbols removed

objconv -ed2036 -fcoff -nu -nd sqlite3fts3x64.o sqlite3fts3x64.obj
objconv -ed2036 -fcoff -nu -nd sqlite3x64.o sqlite3x64.obj


@ECHO DONE!
pause

And off course I had to adapt SynSQLite3.pas:

{$ifdef INCLUDE_FTS3} // link SQlite3 database engine with FTS3/FTS4 + TRACE
  {$ifdef WIN64}
    {$L sqlite3fts3x64.obj}
  {$ELSE}
    {$L sqlite3fts3.obj}
  {$endif}
{$else} // link SQlite3 database engine
  {$ifdef WIN64}
    {$L sqlite3x64.obj}
  {$ELSE}
    {$L sqlite3.obj}
  {$endif}
{$endif}

Some adjustments had to be made to Mormot.pas:

in GetObjectComponent(..) around line 10780 there was a closing bracket missing

{$ifdef CPU64} // pointer(P) to call typinfo
      result := pointer(GetOrdProp(Obj, pointer(P))); {$else}
      result := pointer(P^.GetOrdValue(Obj));
{$endif}

In SYnSelfTest.pas, there was something wrong with the interface section/uses clause, the mormot uses list had to be moved outside the $ifdef

uses
  Windows,
  Classes,
{$ifdef USEVARIANTS}
  Variants,
{$endif}
  SysUtils,
{$ifndef FPC}
{$ifndef LVCL}
  Contnrs,
  SynDB,
{$endif}
{$ifndef CPU64}
  SynSQLite3,
{$ifndef DELPHI5OROLDER}
{$ifndef LVCL}
  SynDBSQLite3,
  mORMotDB,
{$endif}
{$endif}
{$endif}
{$endif}
  mORMot,
  mORMotSQLite3,
  mORMotHttpServer,
  mORMotHttpClient,
  SynCommons;

And next I ran into an Invalid Typecast in TSQLRest.Retrieve

...
  if (self=nil) or (RecordRef(Reference).ID=0) then // Invalid Typecast!!
    exit;
...

because TRecordReference actually is aPtrUInt, and DCC64 does not allow this typecasting . In order to fool the compiler I had to change

RecordRef(Reference).ID

into

RecordRef((@Reference)^).ID

My suggestion would be to provide a function that takes a VAR TRecordReference parameter and returns a RecordRef structure for it. This also allows for better readability and compiler handling I think.

I'm not sure though about the declaration of PtrUInt, as the UNICODE defines is used to identify a 64bit pointer...?:

// SYnCOmmons.pas
{$ifdef ISDELPHI2009}
  PtrUInt = cardinal; { see http://synopse.info/forum/viewtopic.php?id=136 }
{$else}
  PtrUInt = {$ifdef UNICODE}NativeUInt{$else}cardinal{$endif};
{$endif}

...
  TRecordReference = type PtrUInt; //Mormot.pas

And then , I ran into some inline assambler instructions in function TInterfacedObjectFake.FakeCall(var aCall: TFakeCallStack): Int64;

  smvDouble,smvDateTime: asm fld  qword ptr [result] end;  // in st(0)
  smvCurrency:           asm fild qword ptr [result] end;  // in st(0)

I am no expert, in assembler, so for now (and just compiler sake) I surrounded the instructions with {$ifndef WIN64}..{$endif}

And this took me to procedure CallMethod(var Params: TCallMethodArgs);
Which could not be compiled due to "[dcc64 Error] mORMot.pas(30148): E2116 Invalid combination of opcode and operands" on the first statement {push esi}.
For compiler sake I disabled the code again, and continued my quest...

And I returned to SynSelfTests.pas, around line 1269. The error makes perfect sense as V is a classtype (this an object pointer) and PtrInt is a regular integer.

  AVP.Init(TypeInfo(TSynValidates),AV);
  for i := 0 to 1000 do begin
    Check(AVP.Count=i);
    PtrInt(V) := i; // [dcc64 Error] SynSelfTests.pas(1269): E2064 Left side cannot be assigned to
    Check(AVP.Add(V)=i);
    Check(AVP.Count=i+1);
    Check(AV[i]=V);
  end;

Therefor I changed the definition in SynCrtSock like this:

{$ifndef FPC}
  {$ifdef UNICODE}
  PtrInt = NativeInt;
  PtrUInt = NativeUInt;
  {$else}
  PtrInt = integer;
  PtrUInt = Cardinal;
  {$endif}
  PPtrUInt = ^PtrUInt;
  PPtrInt = ^PtrInt;
{$endif}

Back to SynSelfTest... around line 2645, TCollTest appears to be undefined. So I moved the TCollTest class declaration above the $ifdef cPU64

And now it compiles...!

Switching back to Win32 mode & building TestSQL3 reveals I didn't break any existing code...

Now I'm trying to run the 64bit TestSQL3.exe, wich is NOT expected to work completely, but let's see how far we can get!

In SYnCommons.pas, There's an Assert(SizeOf(TSynTableData)=16); which I had to change since. (there are obviously 2 pointers in it, leading towards 8 extra bytes)

{$ifdef WIN64}
  Assert(SizeOf(TSynTableData)=24);
{$else}
  Assert(SizeOf(TSynTableData)=16);
{$endif}

Same goes for the initialization section of SynCrtStock.pas (I took sizes from the debugger, so I'm unsure if these actually are correct sizes):

{$ifdef WIN64}
  Assert((sizeof(HTTP_REQUEST)=848) and (sizeof(HTTP_SSL_INFO)=48) and
    (sizeof(HTTP_DATA_CHUNK_INMEMORY)=32) and
    (sizeof(HTTP_DATA_CHUNK_FILEHANDLE)=32) and
    (sizeof(HTTP_REQUEST_HEADERS)=688) and
    (sizeof(HTTP_RESPONSE_HEADERS)=512) and (sizeof(HTTP_COOKED_URL)=40) and
    (sizeof(HTTP_RESPONSE)=552) and (ord(reqUserAgent)=40) and
    (ord(respLocation)=23) and (sizeof(THttpHeader)=4));
{$else}
  Assert((sizeof(HTTP_REQUEST)=464) and (sizeof(HTTP_SSL_INFO)=28) and
    (sizeof(HTTP_DATA_CHUNK_INMEMORY)=24) and
    (sizeof(HTTP_DATA_CHUNK_FILEHANDLE)=32) and
    (sizeof(HTTP_REQUEST_HEADERS)=344) and
    (sizeof(HTTP_RESPONSE_HEADERS)=256) and (sizeof(HTTP_COOKED_URL)=24) and
    (sizeof(HTTP_RESPONSE)=280) and (ord(reqUserAgent)=40) and
    (ord(respLocation)=23) and (sizeof(THttpHeader)=4));
{$endif}

Next stop: SynGDIPlus (No idea why we need this ans I probably dont wanna know). Anyway, the Hook procedure is still NIL after calling startup so I added a small check in TGDIPlus.Create in order to get past the AV.

  Input.SuppressBackgroundThread := true;
  if Startup(fToken,Input,fStartupHook)<>stOk then begin
    fToken := 0;
    UnLoad;
    exit;
  end;
  if Assigned(fStartupHook.Hook) then // HH: Avoid AV
    fStartupHook.Hook(fStartupHookToken);

Next Stop: The initialization section of SynOleDB:

{$ifdef WIN64}
  assert(sizeof(TOleDBStatementParam)=sizeof(PTrUInt)*4);
{$else}
  assert(sizeof(TOleDBStatementParam)=sizeof(PTrUInt)*4+sizeof(Int64));
{$endif}

Jay! - The application starts and displays it's first assertion Passed! - and hangs in SynSelfTests.TTestLowLevelCommon._TDynArray, around line 1250:

  AIP.Reverse;
  for i := 0 to 50000 do
    Check(AI[i]=50000-i);
  AIP.Slice(AI2,2000,1000);
  Check(length(AI2)=2000); // Here it hangs!
  for i := 0 to 1999 do
    Check(AI2[i]=49000-i);
  AIP.AddArray(AI2,1000,2000);
  Check(AIP.Count=51001);

It hangs so badly I even have to abort RAD studio

Maybe you can shed some light on the changes I had to make. We will probably continue the 64bit story by the end of this or next week. Or next year ?

Regards!

Hans

h.hasenack · mORMot 1

ab wrote:

Perhaps you have some code to share?

Yes, we intend to share the Nexus "driver". Only I fear it's not so ready for production/serious beta testing yet. e.g. batch writes are missing from what I understood from Bascy. I'll ask Bascy anyway to send the unit to you anyway so you can chew on it .

ab wrote:

What are the performance feedback?

It's not bad at all, depending a bit on using the NexusDB embedded engine or the remote engine. I'm not sure wheter NexusDB allows batch update like Oracle: prepare one statement and next only send tons of data. I understand NexusDB will require a (prepared) query to be executed for each insert/update statement.

Anyway, Here's some performance output, NexusDB compared to Oracle in the same test.

NexusDB, Batch=False, Transaction=False
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 806 records added, avg speed 403 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 806 records fetched, avg speed 1,011 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 806 records modified, avg speed 580 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 806 records deleted, avg speed 2,575 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)

NexusDB, Batch=False, Transaction=True
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 891 records added, avg speed 446 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 891 records fetched, avg speed 1,018 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 891 records modified, avg speed 594 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 47 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 891 records deleted, avg speed 3,160 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)

NexusDB, Batch=True, Transaction=False
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 6,006 records added, avg speed 2,687 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 375 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 6,006 records fetched, avg speed 1,077 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 329 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 6,006 records modified, avg speed 2,301 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 297 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 6,006 records deleted, avg speed 3,231 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)

NexusDB, Batch=True, Transaction=True
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 8,008 records added, avg speed 3,635 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 485 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 8,008 records fetched, avg speed 1,077 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 391 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 8,008 records modified, avg speed 2,683 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 391 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 8,008 records deleted, avg speed 4,343 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)


Oracle, Batch=False, Transaction=False
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 1,003 records added, avg speed 502 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 16 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,003 records fetched, avg speed 558 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 15 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,003 records modified, avg speed 845 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 16 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,003 records deleted, avg speed 1,834 rps Batch=False Transaction=False Process SGTestrunner_D17.exe (2260)

Oracle, Batch=False, Transaction=True
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 1,267 records added, avg speed 634 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 16 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,267 records fetched, avg speed 513 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 0 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,267 records modified, avg speed 1,126 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 0 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 1,267 records deleted, avg speed 2,192 rps Batch=False Transaction=True Process SGTestrunner_D17.exe (2260)

Oracle, Batch=True, Transaction=False
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 17,017 records added, avg speed 8,509 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 266 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 17,017 records fetched, avg speed 1,129 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 250 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 17,017 records modified, avg speed 14,145 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 234 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 17,017 records deleted, avg speed 35,087 rps Batch=True Transaction=False Process SGTestrunner_D17.exe (2260)


Oracle, Batch=True, Transaction=True
Debug Output: ---CREATE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: 23,023 records added, avg speed 10,375 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---READ TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 391 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 23,023 records fetched, avg speed 1,121 rps  Process SGTestrunner_D17.exe (2260)
Debug Output: ---MODIFY TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 343 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 23,023 records modified, avg speed 15,349 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)
Debug Output: ---DELETE TEST--- Process SGTestrunner_D17.exe (2260)
Debug Output: Query took 328 ms Process SGTestrunner_D17.exe (2260)
Debug Output: 23,023 records deleted, avg speed 5,457 rps Batch=True Transaction=True Process SGTestrunner_D17.exe (2260)

ab wrote:

BATCH format uses an array of JSON objects...
Adding BLOB content would expect the content to be Base64 encoded. It will be fast (SynCommons.pas has very optimized functions), but perhaps won't be not very optimized for the bandwidth.
Updating BLOB fields should already be available, if you retrieve the fields using a FillPrepare (since updated fields will follow the one retrieved with the FillPrepare).

I'll have to check this, Ididn't see a blob on/off in fillprepare yet.

ab wrote:

But we may better add a generic way.
What about adding a FieldNames optional RawUTF8 parameter, with the field list as CSV?

I think something like this is already available isn't it?
Still I don't think fetching the blob's separately is a bad idea at all. I just feel the Fetch/Update/Delete methods are a bit oddly spread across the REST client and the TSQLRecord... Having a TSQLRecord.FillOne with a blob selection parameter as described earlier seems a nice solution to me. It might be hard to implement though, I read below it depends on the FillPrepare statement (which obviously prepares the fieldlist of the fetched fields).

ab wrote:

Or directly a PSQLFieldBits parameter, with a list of field index bits?

This seems a bit "enforced artificial" to me. I like the CSV ot TStrings solution better in that case.

ab wrote:

Or as an open array of RawUTF8 field names?

That could work, and requires no escape characters in any case. But wait, why not a TStrings type? That seems to very common practice to send over a list of strings between modules, controls etcetera.

h.hasenack · mORMot 1

The big issue was that we forgot to implement the ftsBlob handlking in our nexus driver
Once this was fixed, it worked like a charm

h.hasenack · mORMot 1

Finally I managed to update our nexusDB driver so it can read and write blob fields.

But I discovered there's no support for writing blob fields in batchadd/batchupdate (checked this NG too). What are the chances of getting this? I do understand batches can become quite large using large blobs, but it would seriously enhance my performance if available.

Hans

h.hasenack · mORMot 1

I have found TSQLRestServer.RetrieveBlobFields and .UpdateBlobfields by now, it was there al the time.

Still, I think the optional parameter for TSQLRecord.FillOne as described earlier would be a nice feature.
Same goes -off course- for the TSQLRecord.Add and TSQLRecord.Update methods for that matter.

h.hasenack · mORMot 1

I think I have worked around it. I'm using a TSQLRawBytes to store my Delphi stream. Previously I stored a TMemorySTream during runtime of by object) but this is just as efficient and also allows the blobreartieve and update routines to work as expected.

My error. - Sorry.

h.hasenack · mORMot 1

Hi Ab

I am fiddling to load and write an object that contains a Tpersistent derived property.

This property needs to go into a (probably custom) blob field as is size (and class) is only known during runtime. Usually I use the ObjectBinaryToText routines to generate text from my object resources.

I have tried using TSQLPropInfoCustom, but I run into the reader and writer parameters provided. I was expecting something like

  TOnSQLPropInfoRecord2Text = procedure(const Data; DataLen: integer; var Text: RawUTF8);
  TOnSQLPropInfoRecord2Data = procedure(const Text: RawUTF8; var Data);

but actually, the code in mormot.pas looks like:

  TOnSQLPropInfoRecord2Text = procedure(Data: pointer; DataLen: integer; var Text: RawUTF8);
  TOnSQLPropInfoRecord2Data = procedure(Text: PUTF8Char; var Data: RawByteString);

What way would you advise to load/save TPersistent objects together with my TSQLRecord?

Regards - Hans

h.hasenack · mORMot 1

Hi Ab

I can use a restserver-global parameter to fetch blobs (ForceBlobTransfert=True), or a field specific method to fetch blob data (RetrieveBlob/UpdateBlob).
What's missing here is a way to fetch a single object including all it's blobs depending on the parameter.

This would allow traversing the entire dataset, and only where required load the blobs without changing restserver [parameters or specifying specific blob field names.

Currently, I dont like RetrieveBlob/UpdateBlob that much because it requires a blob field name (=property name) as a parameter. It would already be an enhancement if omitting the property name would fetch all blob's for the provided TSQLRecord.

Also somehow I expected RetrieveBlob/UpdateBlob to be available as methods of TSQLRecord rather than as TRESTserver methods. Otherwise, FillOne was to be expected a method of TRestServer too...

So.

What I am looking for could be something like:

TMormotFetchMode=(fmDefault,fmExcludeBlobs,fmIncludeBlobs);

...

TSQLRecord.FillOne(aFetchMode:TMormotFetchMode=fmDefault)
...

And maybe also a routine something like

  TSQLRecord.FetchAllBlobs; // retrieves all blob fields for the currently loaded record

Whaddyathink?

Hans

h.hasenack · mORMot 1

Gr8!

h.hasenack · mORMot 1

I can find out using my debugger, but it would be so much better to have an 'GetLastError' solution, or maybe have an exception at client side raised that contains info about WHY the update failed.

Errors can haCOuld be e.g.
* the database rejected (key violation, invalid data etc) the update from the mOrmot
* mormot rejected the data
* communication failure/dropout

Currently it is quite hard (clienty side) to find out .

SUggestion:

find a way to catch/hold on to the "last" server side exception / exception message, and send it over to the client, either directly with the reply (not preferred due to bandwith considerations) or (preferably) in for form of a GetLastMotmotError:UTF8string / raiseLastMormotError solution that queries the server for the last exception/error info.

Hans

h.hasenack · mORMot 1

WOnderful. I'll keep you posted!

h.hasenack · mORMot 1

Yes I am definitely willing to share. I'll keep you posted.

h.hasenack · mORMot 1

I Moved the SYnLog definition outside defenitions and syncommons compiles well now.
Also I added USETYPEINFO define to x64 compilation

  {$ifdef CPUX64}
    {$define CPU64} // Delphi compiler for 64 bit CPU
    {$undef CPU32}
    {$define PUREPASCAL} // no x86 32 bit asm to be used
    {$define USETYPEINFO}  // in x64 we can onpy use official typeinfo structures
  {$else}

Next I run into trouble with the next code. Do you think my code proposal is valid?

function TSQLPropInfoRTTI.GetFieldAddr(Instance: TObject): pointer;
begin
  if Instance=nil then
    result := nil else
    {$ifndef USETYPEINFO}  // inx64 we can onpy use official typeinfo structures
    result := fPropInfo^.GetFieldAddr(Instance);
    {$else}
    begin
      if NativeInt(fPropInfo^.GetProc)<0 then // high bit set: there is no getter method so it's an offset. (should work in x64 as well as x32)
      begin
        result:=Instance;
        inc(PByte(Result),(integer(NativeInt(fPropInfo^.GetProc)) and $7fffffff));  add the offset to the object base pointer
      end
      else Result:=nil; // return nil for no pointer to data because there's a getter method
    end
    {$endif}
end;

Then again, this code could also be defined in the as a "pure pascal" routine...

{$ifndef USETYPEINFO}
function TPropInfo.GetFieldAddr(Instance: TObject): pointer;
asm
  cmp [eax].TPropInfo.GetProc.Byte[3],$FF // is it a field property?
  mov eax,[eax].TPropInfo.GetProc
  jnz @0
  // field - Getter is the field's offset in the instance data
  and eax,$00FFFFFF
  add eax,edx
  ret
@0:xor eax,eax // method -> return nil
end;
{$endif}

I seem to get a little stuck now. Can I ask you to put some focus on the x64 compilation? If you need I can provide you with a RDP session or Teamviewer session. Just send me an email and we'll find out how.

Hans

h.hasenack · mORMot 1

Hi Ab

I have managed to compile my sqlite3.c into 64 bit .o files. Now I'm trying to compile the Mormot sources into 64 bit dcus, but I'm running into trouble with SynCOmmonspas, regarding the SynLog rouytines.

This piece

var
  /// internal list of created TSynLog instance, one per each log file on disk
  // - do not use directly - necessary for inlining TSynLogFamily.SynLog method
  SynLogFile: TObjectList = nil;

is required at many other places, but is inside a {$ifdef NOEXCEPTIONINTERCEPT} block, causing compilation failes at several places where the global SynLogFile is addressed like:

function TSynLogFamily.CreateSynLog: TSynLog;
var i: integer;
begin
  if SynLogFile=nil then begin
    SynLogFile := TObjectList.Create;
    GarbageCollector.Add(SynLogFile);
  end;
  result := fSynLogClass.Create(self);
  i := SynLogFile.Add(result);
  if fPerThreadLog then
    SynLogFileIndex[fIdent] := i+1 else
    fGlobalLog := result;
end;

Is something wrong with the compiler defines or is it just me not understanding what needs to be done

(Using Delphi XE3)

h.hasenack · mORMot 1

Thanks!

h.hasenack · mORMot 1

Here's the Patch I came up with. Unsure though if it is safe (memory leaks) for the other parts where 'CreateInstance' is called.

My own unit tests turn up OK now, the unexpected destruction of my implementation object has disappeared.

My guess: You'd probably want to change CreateInstance so it returns an IInterface rather than an TInterfacedObject.
This way, you could leave the reference counting to the RTL. I tried this, but I had to change so much code that it made me feel uncomfortably.

Here's another idea for your TServiceFactoryServer: instead of registering classes, you could also register constructors (or functions that instantiate interfaces) rather than classes. This way, you are not limited to using TInterfacedObject derived classes for your interface implementations. It would also address the virtual constructor for the interface implementation class that we discussed a while ago.

function TServiceFactoryServer.CreateInstance: TInterfacedObject;
begin
  if fImplementationClassWithCustomCreate then
    result := TInterfacedObjectWithCustomCreateClass(fImplementationClass).Create else
    result := fImplementationClass.Create;
  // HH:Added to ensure reference count of at least 1
  (Result as IInterface)._AddRef;
end;

procedure TServiceFactoryServerInstance.SafeFreeInstance;
begin
  try
    InstanceID := 0;
    // HH:Added to use reference count to drop interfaces
    // rather than direct destructor calls.
    ((Instance as TInterfacedObject) as IInterface)._Release;

  //    FreeAndNil(Instance);
  except
    ; // just ignore any error in customer code -->>HH: Are you shure you dont want to catch 'fatal' exceptions here? or at least report them somewehere?
  end;
end;

destructor TServiceFactoryServer.Destroy;
var i: integer;
begin
  try // release any internal instance (should have been done by client)
    for i := 0 to fInstancesCount-1 do
      fInstances[i].SafeFreeInstance;
//      if fInstances[i].Instance<>nil then
//        fInstances[i].Instance.Free;
  except
    ; // better ignore any error in business logic code
  end;
  DeleteCriticalSection(fInstanceLock);
  inherited;
end;

h.hasenack · mORMot 1

The interface is a "registered" one, created by the factory on server side. My implementation object is derived from TInterfacedObjectWithCustomCreate, de method called is Harvest(True:boolean)

Here's a call stack to call of the interface method on the server side

uLCSSimulationImpl.TLCSSimulationImpl.Harvest(True)
uLCSSimulationImpl.TLCSSimulationImpl.CheckResultsChanged(1636016)
:0065119B Mormot::CallMethod(Params=????)
mORMot.CallMethod(???)
mORMot.TServiceMethod.InternalExecute((...),nil {#0},$FE431B20,'',[])
mORMot.TServiceFactoryServer.ExecuteMethod($18FACC,2,0,'[1636016','','','')
mORMot.TSQLRestServer.LaunchService($18FACC,'',400)
mORMot.TSQLRestServer.URI('LCSServer/CRMS6H2TKXSUAAVAIW34CCBIA/LCSSimulation.CheckResultsChanged?session_signature=0000004C002529B97A1A92F7','POST','[1636016'#0,'','',$18FA4B)
mORMot.TSQLRestServer.AnswerToMessage((74, (), 7474580, $18FDD4, 200))
SynCommons.WndProcMethod(8390460,74,7474580,1637844)
...

I checked out: When the constructor of the object is called, it reveals that an object (pointer) is stored rather than an interface instance. This causes the reference count to be 0 instead of 1.
Here's the calls stack when calling the constructor:

uLCSSimulationImpl.TLCSSimulationImpl.Create
mORMot.TServiceFactoryServer.CreateInstance
mORMot.AddNew
mORMot.TServiceFactoryServer.InternalInstanceRetrieve((1, 623915141, nil),2)
mORMot.TServiceFactoryServer.ExecuteMethod($18FACC,2,0,'[1636016]','','','')
mORMot.TSQLRestServer.LaunchService($18FACC,'',400)
mORMot.TSQLRestServer.URI('LCSServer/A1VY503GHHDKPBNN47C4W5MII/LCSSimulation.CheckResultsChanged?session_signature=0000004C00253030C328628D','POST','[1636016]','','',$18FA4B)
mORMot.TSQLRestServer.AnswerToMessage((74, (), 5244374, $18FDD4, 200))
SynCommons.WndProcMethod(22416962,74,5244374,1637844)
...

When debugging Mormot.pas, and checking out a watch TInterfacedObject(Inst.Instance).FRefCount it reveals the RefCOunt of the instantiated object is 0.

function TServiceFactoryServer.InternalInstanceRetrieve(
  var Inst: TServiceFactoryServerInstance; aMethodIndex: integer): boolean;
procedure AddNew;
var i: integer;
    P: ^TServiceFactoryServerInstance;
begin
  Inst.Instance := CreateInstance;
  if Inst.Instance=nil then <<====== Here the FRefcount of the instantiated object is 0
    exit;
  P := pointer(fInstances);
  for i := 1 to fInstancesCount do
    if P^.InstanceID=0 then begin
      P^ := Inst; // found an empty entry -> use it
      exit;
    end else
    inc(P);
  fInstance.Add(Inst); // append a new entry
end;

h.hasenack · mORMot 1

Hi Ab

I am still unit testing my own code and came across the strange phenomenon: During the processing of an (remote) interfaced call, the reference count of my TInterfacedObject (serverside) is 0, even on its direct call entry.

The nasty side effect of this is that when I pass on an interface of "self" to any subroutine, my object gets freed when the routine returns (due to the reference count getting 0 again).

May I suggest increasing the reference count at least during the call? Unfortunately, when dropping the refcount after the call this would free the object and I am not sure whether this is what you want.

I understand there's some kind of timeout dropping the interface with the restserver if the connection is lost somehow. This could also drop the reference count for the interface and thus free the object.

I hope you get what I mean.

Regards, Hans

h.hasenack · mORMot 1

I understand, and adjusting it is exactly what I did.

What I meant is that changing this setting does not affect the way mormot works, it only as an effect on the output file size. My suggestion was to completely leave out the setting as it is no requirement for mormot to compile/run.

Hans

h.hasenack · mORMot 1

Hi Ab

I am using class attributes to document my unit tests, and this works exceptionally fine. Except for the unit tests generated with {$I synopse.inc} in the dpr file. It took me several hours to find out why the same unit did have attributes in one exe and not in another.

It turned out that in synopse.inc you set {$RTTI EXPLICIT METHODS([]) PROPERTIES([]) FIELDS([])} to reduce rtti footprint for 2010/XE/XE2/XE3 code generation.

    {$if CompilerVersion >= 21.0}
      // Delphi 2010/XE: Reduce EXE size by disabling much RTTI
      {$define ISDELPHI2010}
      {$WEAKLINKRTTI ON}
      {$RTTI EXPLICIT METHODS([]) PROPERTIES([]) FIELDS([])} 
    {$ifend}

Though this is not a bad idea by itself, it is not required for Mormot compilation IMO. I would suggest leaving the compiler directive out by default and rely on the applications default settings.

Regards - Hans

h.hasenack · mORMot 1

It means if you perform 2 searches within 10 (or so?) seconds you have to wait.
It is intended to discourage robots. Nevertheless, I would suggest activating the delay only after 5 searches within 10 s or so.

But I have no idea how to accomplish this in your forum software.

h.hasenack · mORMot 1

OK, WIll try. I downloaded VS 2012 express. (And read somewehere it's obj files are NOT compatible with delphi.).

We'll see!

h.hasenack · mORMot 1

Hi Ab

2 things are still 'holding me back' from creating my simulation server in native 64 bit mode.

1) The 'Root' database engine, in our case NexusDB
2) The Mormot framework, or better said:
3) SQLite3.obj

I have seen that there are 32bit as well as 64bit DLL's available for SQLite, so I wondered, Maybe you can make Mormot use (optionally?) SQLite3 as a DLL instead of a obj file.
This would allow compiling the sqlite into a DLL using an arbitrary compiler (e.g. M$ VIsual Studio Express) and still use it natively.
I have checked if the obj file generated by M$ is compatibale with Delphi, and appartently it is not (found out on internet, did not check myself).

The 32 bit as well as the 64 bit DLL can be found on the internet. Then again, I wonder what quirks you might run into... :s

http://sourceforge.net/projects/sqlite- … p/download

I just hate waiting for the 64bit c++ compiler (BCC64.exe)

Regards - Hans

h.hasenack · mORMot 1

Yeup. We develop in XE3, and I even created a FMX client app using mormot tof communication to our (non-FMX) server app.
Apart from some regression issues (have been fixed now), it works like a charm.

h.hasenack · mORMot 1

1) Is good reasoning, I agree to that.
2) I mean the return value of type of TServiceFactoryServer.CreateInstance to be TObject rather than TInterfacedObject. The runtime error is avoided by the constructor insisting on a TInterfacedClass anyway.

Dont get me wrong, point 2 is very minor, now I feel like I'm nagging you . I also see now the point of reference counting you do with FSharedinstance. AMOF : just keep it as it is.

Except of course for the error mentioned earlier

h.hasenack · mORMot 1

COme to think of it...

TServiceFactoryServer.CreateInstance

could very well return a TObject, there is no reason I can think of to make it return a TInterfacedObject... do you?

h.hasenack · mORMot 1

Thx

h.hasenack · mORMot 1

I am pretty sure casting an unitialized result to an object will fail. I even tested it

function TServiceFactoryServer.CreateInstance: TInterfacedObject;
begin
  if fImplementationClassWithCustomCreate then
    result := TInterfacedObjectWithCustomCreate(result).Create else
    result := fImplementationClass.Create;
end;

What you probably mean is this:

type
  TInterfacedObjectWithCustomCreateClass=class of TInterfacedObjectWithCustomCreate;
...

function TServiceFactoryServer.CreateInstance: TInterfacedObject;
begin
  if fImplementationClass.InheritsFrom(TInterfacedObjectWithCustomCreate) then
    result := TInterfacedObjectWithCustomCreateClass(fImplementationClass).Create 
  else
    result := fImplementationClass.Create;
end;

My guess is you wont need the fImplementationClassWithCustomCreate private field either.

h.hasenack · mORMot 1

Really?
I am pretty sure that an objkect/interface assigned to a threadvar will NOT be released automatically.
It it is set to nil, it's probably because you programmed it that way, as the compiler does not do this for you.
The encapsulation tip was mainly about making your global vars readonly.
The finalization tip was about the termination of the application to catch faulty initialization calls.

BTW I know about threadvars, they are simply thread-local but application global variables.

h.hasenack · mORMot 1

Thx - I already had an idea it was thought of.

Here's a little encapsulation tip...

In these kind of global variable constructs, I usually create a function to return (and optionally initialize) the variable. In this way I am sure "no-one" changes my global variable. (thus read-only)

I also came up with this nifty trick for (non thread) global variables because I ran into calling the initialization routine when it was actually shutting down

interface
  function MyGlobalObject:TMyObject;
  procedure DropMyGlobalObject;

implementation

VAR vMyGlobalObject:TMyObject=nil;

function MyGlobalObject:TMyObject;
begin
  if NativeInt(vMyGlobalObject)=-1 then 
    raise EFinalizetionError.Create('Cannot initialize MyGlobalObject when shutting down') //  the global object was finalized!
  else if vMyGlobalObject=nil then
    vMyGlobalObject=TMyObject.Create; // the global object needs to be initialized
  Result:=vMyGlobalObject;
end;

procedure DropMyGlobalObject;
begin
  if NativeInt(vMyGlobalObject)<>-1 then
    FreeAndNil(vMyGlobalObject); // allows re-initialization. Comes in handy for unit tests.
end;

procedure FinalizeMyGlobalObject;
begin
  DropMyGlobalObject;
  NativeInt(vMyGlobalObject):=-1; // mark as finalized.
end;

initialization

finalization
  FinalizeMyGlobalObject;
end.

Off course, the same trick goes for interfaces too. And as objects are always (at least!) on 4byte boundaries, the object pointer is impossible to be -1 for any real object instance, so... it;'s pretty safe,

One drawback is that there is no exception handling available during shutdown, so you may consider using a different solution than raising an exception. In the debugger, however, it works like a charm to find initialization/finalization culprits.

It would be nice to have this also for thread variables, but unfortunately there is no safe/watertight way I know of to "catch" terminating threads.

h.hasenack · mORMot 1

Ah that's cool too... - Thx

h.hasenack · mORMot 1

Hi Ab

I did not get time yet to study and memorize the entire manual, but here's the thing. I might be asking for something obvious.

I have an interface at server side, that is called from client side. (Nothing new there, works like a charm). This interface however produces some data that I want to store in (memory) tables so it can be adressed from client side using the obvious CreateFillAndPrepare methods etcetera.

The first question is: (How) Can I store these records on server side for availability in the correct Client session (memory tables you know). I seem to have no access to the correct REST instance on server side from within my interfaced object at server side... These objects are instantiated without any reference to the "owning" rest server...

Maybe adding an (optional) ISQLRestOwner interface for registered interfaces can fix this? The REST server would then only set the REST owner in case the object supports the SQLRestOwner interface.

(BTW except from the "root" server, my "workbase" rest servers are created dynamically for each "project database" opened).

Also: I dont like the idea of my objects being packed to JSON and unpacked from JSON just to store them, whuile the data is actually in the same process (server side).

So the second question is: How do I get around this?

Hans

h.hasenack · mORMot 1

The current ServiceRegister uses a TInterfacedClass for aImplementationClass

It would clean up my code soooo much if the aImplementationClass would be of type... TMormotInterfacedClass

interface

type
  TMormotInterfacedObject=class(TInterfacedObject)
  public 
    constructor Create; virtual; 
  end;

implementation

constructor TMormotInterfacedObject.Create; 
begin
  inherited;
end;

As it would allow overriding the constructor to initialize my interfaced objects rather than overriding the AfterConstruction method...
Yes, unfortunately it will very likely break some code. (Well, unless a overloaded ServiceRegister method is created. I would not recommend the overloaded method for KISS 's sake - at least mark it as deprecated!)

Whaddyathink?

Hans

h.hasenack · mORMot 1

I didnt get the time to look at it yet, but my hands are itching to get started with it

Thanks

h.hasenack · mORMot 1

I also don't like the JSON/BLOB solution to store the objects in a stream.

I like the TSQLRecordReference solution a bit better (wich boils down to the same: storing the "special" class data somewehere else than in the "main table") more, apart from the fact that it is currently not very robust when the datamodel changes.

About the solution with one ID generator: this means every TSQLRecord has it's own ID (IMO not bad at all) , and resides somehow in the a (global) collection (Not that bad either). Nevertheless you will probably want to be able to "quick-select" all objects of a certain class:
When picking a base class, the derived objects should also be incorporated. When picking a derived class, the base class objects should NOT be incorporated. This seems doable, provided there is some smart table memorizing the class hierarchy. If you want a named collection, one can add a TSQLRecordCollection that holds the ID's of all objects belonging to that collection. In this case, the global collection simulates the "heap" whilst the NamedCollection simulates a TList. It also implies some serious trouble when an TSQLRecord object is deleted from the DB, as in all lists referring to it, it must be removed too.

About the multi table or single table solution for inheritance:
Well, first of all I don't care "that much" about how ORM stores it's data. (Well, off course I am interested in the solution, but it should be of no concern to me. My app interfaces with the ORM, not with the underlying DB.)

I do not specifically want/need a table-per-inheritance tree. As stated before : What I do want is: multiple (named) collections of objects (in this case TSQLRecord). And this TSQLNamedRecordCollection should be able to perform the CRUD actions. However it seems logical to me, to have the collection define a base class for storing the object. But I can also imagine the TSQLNamedRecordCollection being able to CRUD any TSQLRecord derived class. That would be more than perfect. In this case the TSQLNamedRecordCollection behaves like a TObjectList, and "Owns" the TSQLRecord objects in it.

h.hasenack · mORMot 1

I've downloaded and tested D7.

OMG... when adding a "record type" property, GetPropInfo() just returns nil, when examing the proplist returned by GetProplist it just isn't there, it is simply skipped. That's just plain stupid.
Thus: the only way to "stream" objects including record type properties is using the DefineProperties solution like the one defined in TCOmponent.
argh - that's just so.. f. Incomplete in the D7 rtti.

Let's just hope the new property framework will provide a solution for this one too. It would certainly be a great enhancement if the TSQLRecordReference could be resolved in a bit cleaner way. Another solution might be to define it as a class or interface and add some special handling for it as rtti will be generated for class based properties.

h.hasenack · mORMot 1

D7 again

I guess using a object instead of record (thus not a TObject class!, but the old thingy from turbo pascal) would not fix this either... sigh.

What about an Int64, of which the 1st 4 bytes are a hash of the classname? yes, there could be a collision between class names, but what are the odds... (1:4G). Would that be an escape route?

( gnargnar, I going to install D7, just to try it., )

h.hasenack · mORMot 1

This is the error I got when trying to post the new topic:

Error: Could not connect to smtp host "smtp.ictuswin.com" (0) (php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution).

h.hasenack · mORMot 1

TRecordReference - risky busyness?

I noticed this in SQLite3Commons.pas:

  /// a reference to another record in any table in the database Model
  // - stored as an 32 bits unsigned integer (i.e. a pointer=TObject)
  // - type cast any value of TRecordReference with the RecordRef object below
  // for easy access to its content
  // - use TSQLRest.Retrieve(Reference) to get a record value
  // - don't change associated TSQLModel tables order, since TRecordReference
  // depends on it to store the Table type in its highest bits
  TRecordReference = type PtrUInt;

So this means when I have another table added to my model (which might be inserted before or after the referred table, depending on it's name or the loaded plugins), my TRecordReference fields get to be invalid, simply because during downtime my table order may have changed. Using part of the integer as a table index is kind-a awkward IMO.

Anyway, this is pretty bad when building a plugin-based data model, as I cannot (and do not want to) guarantee that my tables are registered in the same order, or for that matter that they will still all be there once the app has been shutdown and restarted. Plugins may be (un)installed during downtime, causing the datamodel to change between sessions. Anyway, this should definitely NOT break my datamodel. (Well, at least not when a table that is not "used" by any record reference is removed/added)

Wouldnt it be much more sensible to have a "combined" link, similar to TMethod (ObjectInstancePointer,ProcPointer), where the 1st pointer points to the targeted class (or table) and the 2nd pointer points to the actual target record, or contains the record ID? Wouldn't it be a lot safer if it's something like this:

  TSQLRecordReference=record
  case type of
    Unresolved:
      TargetRecordID:TSQLRecordID;
      TargetTableClassName:UTF8string;
   Resolved:
      TargetRecord:TSQLRecord
   end;

It is probably not as efficient as an int32, but is surely a lot safer.

(PS I didnt check the syntax of the record, but I'm pretty sure you understand what I mean)
(PPS I failed to create a new topic on TRecordReference, i got sime SMTP timeout, so I'm trying to post it here. Previewe worked just fine)

h.hasenack · mORMot 1

I noticed this in SQLite3Commons.pas:

  /// a reference to another record in any table in the database Model
  // - stored as an 32 bits unsigned integer (i.e. a pointer=TObject)
  // - type cast any value of TRecordReference with the RecordRef object below
  // for easy access to its content
  // - use TSQLRest.Retrieve(Reference) to get a record value
  // - don't change associated TSQLModel tables order, since TRecordReference
  // depends on it to store the Table type in its highest bits
  TRecordReference = type PtrUInt;

So this means when I have another table added to my model (which might be inserted before or after the referred table, depending on it's name or the loaded plugins), my TRecordReference fields get to be invalid, simply because during downtime my table order may have changed. Using part of the integer as a table index is kind-a awkward IMO.

Anyway, this is pretty bad when building a plugin-based data model, as I cannot (and do not want to) guarantee that my tables are registered in the same order, or for that matter that they will still all be there once the app has been shutdown and restarted. Plugins may be (un)installed during downtime, causing the datamodel to change between sessions. Anyway, this should definitely NOT break my datamodel. (Well, at least not when a table that is not "used" by any record reference is removed/added)

Wouldnt it be much more sensible to have a "combined" link, similar to TMethod (ObjectInstancePointer,ProcPointer), where the 1st pointer points to the targeted class (or table) and the 2nd pointer points to the actual target record, or contains the record ID? Wouldn't it be a lot safer if it's something like this:

  TSQLRecordReference=record
  case type of
    Unresolved:
      TargetRecordID:TSQLRecordID;
      TargetTableClassName:UTF8string;
   Resolved:
      TargetRecord:TSQLRecord
   end;

It is probably not as efficient as an int32, but is surely a lot safer.

(BTW I didnt check the syntax of the record, but I'm pretty sure you understand what I mean).

mORMot Open Source

#51 Re: mORMot 1 » Trouble using in-memory tables with latest .obj fils (and pmy own .obj » 2013-01-02 15:48:12

#52 mORMot 1 » Trouble using in-memory tables with latest .obj fils (and pmy own .obj » 2013-01-02 15:02:24

#53 Re: mORMot 1 » Did you get around applying my proposed changes » 2013-01-02 12:32:10

#54 Re: mORMot 1 » Did you get around applying my proposed changes » 2013-01-02 10:00:42

#55 mORMot 1 » Did you get around applying my proposed changes » 2013-01-02 09:33:10

#56 Re: mORMot 1 » Business model and sponsored work » 2013-01-01 19:12:59

#57 Re: mORMot 1 » Business model and sponsored work » 2012-12-28 22:25:00

#58 Re: mORMot 1 » Movin' towards 64bit... » 2012-12-18 16:13:19

#59 Re: mORMot 1 » Movin' towards 64bit... » 2012-12-18 16:08:45

#60 Re: mORMot 1 » Movin' towards 64bit... » 2012-12-18 16:06:34

#61 mORMot 1 » Movin' towards 64bit... » 2012-12-18 15:22:42

#62 Re: mORMot 1 » Writing Blobs in BatchModes » 2012-12-18 08:09:56

#63 Re: mORMot 1 » Creating a TPersistent alike property in TSQLRecord » 2012-12-18 07:29:30

#64 mORMot 1 » Writing Blobs in BatchModes » 2012-12-17 14:49:48

#65 Re: mORMot 1 » Fetching blobs » 2012-12-17 11:35:35

#66 Re: mORMot 1 » Creating a TPersistent alike property in TSQLRecord » 2012-12-17 10:59:40

#67 mORMot 1 » Creating a TPersistent alike property in TSQLRecord » 2012-12-17 10:44:13

#68 mORMot 1 » Fetching blobs » 2012-12-17 09:34:55

#69 Re: mORMot 1 » Add/Update failed. How do I foind out why » 2012-12-15 12:37:26

#70 mORMot 1 » Add/Update failed. How do I foind out why » 2012-12-14 20:46:11

#71 Re: mORMot 1 » x64 compilation » 2012-12-12 08:24:12

#72 Re: mORMot 1 » x64 compilation » 2012-12-11 13:30:35

#73 Re: mORMot 1 » x64 compilation » 2012-12-11 13:26:09

#74 mORMot 1 » x64 compilation » 2012-12-11 12:46:51

#75 Re: mORMot 1 » Interface/object lifetime on serverside objects » 2012-12-06 14:02:03

#76 Re: mORMot 1 » Interface/object lifetime on serverside objects » 2012-12-06 09:04:44

#77 Re: mORMot 1 » Interface/object lifetime on serverside objects » 2012-12-06 07:45:08

#78 mORMot 1 » Interface/object lifetime on serverside objects » 2012-12-05 15:39:17

#79 Re: mORMot 1 » RTTI foot print reduction » 2012-12-05 15:31:21

#80 mORMot 1 » RTTI foot print reduction » 2012-12-05 10:45:45

#81 Re: mORMot 1 » Is it possible to reduce waiting time for reapeating search » 2012-11-20 12:11:23

#82 Re: mORMot 1 » Still moving towards 64bit... » 2012-11-19 13:12:33

#83 mORMot 1 » Still moving towards 64bit... » 2012-11-18 20:16:44

#84 Re: mORMot 1 » XE3 and FMX » 2012-11-09 08:19:21

#85 Re: mORMot 1 » TInterfacedObjectWithCustomCreate implementation error » 2012-11-06 09:48:19

#86 Re: mORMot 1 » TInterfacedObjectWithCustomCreate implementation error » 2012-11-06 09:21:36

#87 Re: mORMot 1 » Generating records at server side » 2012-11-06 09:12:18

#88 mORMot 1 » TInterfacedObjectWithCustomCreate implementation error » 2012-11-06 09:02:23

#89 Re: mORMot 1 » Generating records at server side » 2012-11-06 07:35:51

#90 Re: mORMot 1 » Generating records at server side » 2012-11-05 20:14:12

#91 Re: mORMot 1 » TSQLRestServer.ServiceRegister change request » 2012-11-05 20:12:30

#92 mORMot 1 » Generating records at server side » 2012-11-05 16:42:40

#93 mORMot 1 » TSQLRestServer.ServiceRegister change request » 2012-11-05 14:26:40

#94 Re: mORMot 1 » Support for a TGUID field » 2012-10-26 13:38:16

#95 Re: mORMot 1 » Inheritance and TSQLRecord » 2012-10-23 07:34:43

#96 Re: mORMot 1 » TRecordReference - risky busyness? » 2012-10-22 21:38:42

#97 Re: mORMot 1 » TRecordReference - risky busyness? » 2012-10-22 18:34:19

#98 Re: mORMot 1 » Inheritance and TSQLRecord » 2012-10-22 15:05:04

#99 Re: mORMot 1 » Inheritance and TSQLRecord » 2012-10-22 15:03:05

#100 mORMot 1 » TRecordReference - risky busyness? » 2012-10-22 14:57:28

Board footer