mORMot Open Source

h.hasenack

Member

From: Nijmegen, Netherlands

Registered: 2012-08-01

Posts: 173

Website

Movin' towards 64bit...

Sigh... its hard but we're getting there!

It's a bit of a log style, so you'd probably run into the same problems in the same order.

I finally managed to compile sqlite3.c into a .o (=ELF64) object file, AND managed to compile it into TestSQL3.exe.

Nevertheless I am still under the impression that using a SQLite3.DLL in combination with the new "delayed" directive available in XE2/XE3 might make it an easier task to compile mormot for Win32 and Win64. (Ckeck Delayed loading of libraries here http://docwiki.embarcadero.com/RADStudi … _Packages)

In order to create all required obj files I run following batch file, which requires a objconv tool as delphi64 is unable to link in the .o files generated by BCC64. (seriously)
http://www.agner.org/optimize/objconv.zip

@echo off

ECHO compile sqlite3.c into obj files

rem DROP pre- existing obj files

del sqlite3*.o
del sqlite3*.obj

ECHO Generate 32bit OBJ files

bcc32 -6 -O2 -c -d -DSQLITE_ENABLE_FTS3 -u- -o sqlite3fts3.obj sqlite3.c 
bcc32 -6 -O2 -c -d -u- -o sqlite3.obj sqlite3.c

ECHO generate 64bit ELF64 OBJ files

bcc64 -c -DSQLITE_ENABLE_FTS3 -o sqlite3fts3x64.o sqlite3.c
bcc64 -c -o sqlite3x64.o sqlite3.c

ECHO convert the ELF64 to 64bit COFF files, suitable for Delphi linking with leading underscores for symbols removed

objconv -ed2036 -fcoff -nu -nd sqlite3fts3x64.o sqlite3fts3x64.obj
objconv -ed2036 -fcoff -nu -nd sqlite3x64.o sqlite3x64.obj


@ECHO DONE!
pause

And off course I had to adapt SynSQLite3.pas:

{$ifdef INCLUDE_FTS3} // link SQlite3 database engine with FTS3/FTS4 + TRACE
  {$ifdef WIN64}
    {$L sqlite3fts3x64.obj}
  {$ELSE}
    {$L sqlite3fts3.obj}
  {$endif}
{$else} // link SQlite3 database engine
  {$ifdef WIN64}
    {$L sqlite3x64.obj}
  {$ELSE}
    {$L sqlite3.obj}
  {$endif}
{$endif}

Some adjustments had to be made to Mormot.pas:

in  GetObjectComponent(..) around line 10780 there was a closing bracket missing

{$ifdef CPU64} // pointer(P) to call typinfo
      result := pointer(GetOrdProp(Obj, pointer(P))); {$else}
      result := pointer(P^.GetOrdValue(Obj));
{$endif}

In SYnSelfTest.pas, there was something wrong with the interface section/uses clause, the mormot uses list had to be moved outside the $ifdef

uses
  Windows,
  Classes,
{$ifdef USEVARIANTS}
  Variants,
{$endif}
  SysUtils,
{$ifndef FPC}
{$ifndef LVCL}
  Contnrs,
  SynDB,
{$endif}
{$ifndef CPU64}
  SynSQLite3,
{$ifndef DELPHI5OROLDER}
{$ifndef LVCL}
  SynDBSQLite3,
  mORMotDB,
{$endif}
{$endif}
{$endif}
{$endif}
  mORMot,
  mORMotSQLite3,
  mORMotHttpServer,
  mORMotHttpClient,
  SynCommons;

And next I ran into an Invalid Typecast in TSQLRest.Retrieve

...
  if (self=nil) or (RecordRef(Reference).ID=0) then // Invalid Typecast!!
    exit;
...

because TRecordReference actually is aPtrUInt, and DCC64 does not allow this typecasting . In order to fool the compiler I had to change

RecordRef(Reference).ID

into

RecordRef((@Reference)^).ID

My suggestion would be to provide a function that takes a VAR TRecordReference parameter and returns a RecordRef structure for it. This also allows for better readability and compiler handling I think.

I'm not sure though about the declaration of PtrUInt, as the UNICODE defines is used to identify a 64bit pointer...?:

// SYnCOmmons.pas
{$ifdef ISDELPHI2009}
  PtrUInt = cardinal; { see http://synopse.info/forum/viewtopic.php?id=136 }
{$else}
  PtrUInt = {$ifdef UNICODE}NativeUInt{$else}cardinal{$endif};
{$endif}

...
  TRecordReference = type PtrUInt; //Mormot.pas

And then , I ran into  some inline assambler instructions in function TInterfacedObjectFake.FakeCall(var aCall: TFakeCallStack): Int64;

  smvDouble,smvDateTime: asm fld  qword ptr [result] end;  // in st(0)
  smvCurrency:           asm fild qword ptr [result] end;  // in st(0)

I am no expert, in assembler, so for now (and just compiler sake) I surrounded the instructions with {$ifndef WIN64}..{$endif}

And this took me to procedure CallMethod(var Params: TCallMethodArgs);
Which could not be compiled due to "[dcc64 Error] mORMot.pas(30148): E2116 Invalid combination of opcode and operands" on the first statement {push esi}.
For compiler sake I disabled the code again, and continued my quest...

And I returned to SynSelfTests.pas, around line 1269. The error makes perfect sense as V is a classtype (this an object pointer) and PtrInt is a regular integer. 

  AVP.Init(TypeInfo(TSynValidates),AV);
  for i := 0 to 1000 do begin
    Check(AVP.Count=i);
    PtrInt(V) := i; // [dcc64 Error] SynSelfTests.pas(1269): E2064 Left side cannot be assigned to
    Check(AVP.Add(V)=i);
    Check(AVP.Count=i+1);
    Check(AV[i]=V);
  end;

Therefor I changed the definition in SynCrtSock like this:

{$ifndef FPC}
  {$ifdef UNICODE}
  PtrInt = NativeInt;
  PtrUInt = NativeUInt;
  {$else}
  PtrInt = integer;
  PtrUInt = Cardinal;
  {$endif}
  PPtrUInt = ^PtrUInt;
  PPtrInt = ^PtrInt;
{$endif}

Back to SynSelfTest... around line 2645, TCollTest appears to be undefined. So I moved the TCollTest class declaration above the $ifdef cPU64

And now it compiles...!

Switching back to Win32 mode & building TestSQL3 reveals I didn't break any existing code...

Now I'm trying to run the 64bit TestSQL3.exe, wich is NOT expected to work completely, but let's see how far we can get!

In SYnCommons.pas, There's an Assert(SizeOf(TSynTableData)=16); which I had to change since. (there are obviously 2 pointers in it, leading towards 8 extra bytes)

{$ifdef WIN64}
  Assert(SizeOf(TSynTableData)=24);
{$else}
  Assert(SizeOf(TSynTableData)=16);
{$endif}

Same goes for the initialization section of SynCrtStock.pas (I took sizes from the debugger, so I'm unsure if these actually are correct sizes):

{$ifdef WIN64}
  Assert((sizeof(HTTP_REQUEST)=848) and (sizeof(HTTP_SSL_INFO)=48) and
    (sizeof(HTTP_DATA_CHUNK_INMEMORY)=32) and
    (sizeof(HTTP_DATA_CHUNK_FILEHANDLE)=32) and
    (sizeof(HTTP_REQUEST_HEADERS)=688) and
    (sizeof(HTTP_RESPONSE_HEADERS)=512) and (sizeof(HTTP_COOKED_URL)=40) and
    (sizeof(HTTP_RESPONSE)=552) and (ord(reqUserAgent)=40) and
    (ord(respLocation)=23) and (sizeof(THttpHeader)=4));
{$else}
  Assert((sizeof(HTTP_REQUEST)=464) and (sizeof(HTTP_SSL_INFO)=28) and
    (sizeof(HTTP_DATA_CHUNK_INMEMORY)=24) and
    (sizeof(HTTP_DATA_CHUNK_FILEHANDLE)=32) and
    (sizeof(HTTP_REQUEST_HEADERS)=344) and
    (sizeof(HTTP_RESPONSE_HEADERS)=256) and (sizeof(HTTP_COOKED_URL)=24) and
    (sizeof(HTTP_RESPONSE)=280) and (ord(reqUserAgent)=40) and
    (ord(respLocation)=23) and (sizeof(THttpHeader)=4));
{$endif}

Next stop: SynGDIPlus (No idea why we need this ans I probably dont wanna know). Anyway, the Hook procedure is still NIL after calling startup so I added a small check in TGDIPlus.Create in order to get past the AV.

  Input.SuppressBackgroundThread := true;
  if Startup(fToken,Input,fStartupHook)<>stOk then begin
    fToken := 0;
    UnLoad;
    exit;
  end;
  if Assigned(fStartupHook.Hook) then // HH: Avoid AV
    fStartupHook.Hook(fStartupHookToken);

Next Stop: The initialization section of SynOleDB:

{$ifdef WIN64}
  assert(sizeof(TOleDBStatementParam)=sizeof(PTrUInt)*4);
{$else}
  assert(sizeof(TOleDBStatementParam)=sizeof(PTrUInt)*4+sizeof(Int64));
{$endif}

Jay! - The application starts and displays it's first assertion Passed! - and hangs in SynSelfTests.TTestLowLevelCommon._TDynArray, around line 1250:

  AIP.Reverse;
  for i := 0 to 50000 do
    Check(AI[i]=50000-i);
  AIP.Slice(AI2,2000,1000);
  Check(length(AI2)=2000); // Here it hangs!
  for i := 0 to 1999 do
    Check(AI2[i]=49000-i);
  AIP.AddArray(AI2,1000,2000);
  Check(AIP.Count=51001);

It hangs so badly I even have to abort RAD studio

Maybe you can shed some light on the changes I had to make. We will probably continue the 64bit story by the end of this or next week. Or next year ?

Regards!

Hans

h.hasenack

Member

From: Nijmegen, Netherlands

Registered: 2012-08-01

Posts: 173

Website

Re: Movin' towards 64bit...

As it turns out there was a very nasty bug in SynCOmmons.pas, procedure TSynLog.LogFileHeader;

a pointer is used to dump the envoironment strinngs, but the pointer is changed by the routine before FreeEnvironmentStringsA is called. thus with the wrong pointer, causeing the ugly hang. Here's the fixed routine:

procedure TSynLog.LogFileHeader;
var Env,P: PAnsiChar;
    L: Integer;
begin
  if not QueryPerformanceFrequency(fFrequencyTimeStamp) then begin
    fFamily.HighResolutionTimeStamp := false;
    fFrequencyTimeStamp := 0;
  end;
  ExeVersionRetrieve;
  if InstanceMapFile=nil then begin
    InstanceMapFile := TSynMapFile.Create;
    GarbageCollector.Add(InstanceMapFile);
  end;
  // array of const is buggy under Delphi 5 :( -> use fWriter.Add*()
  with ExeVersion, SystemInfo, OSVersionInfo, fWriter do begin
    AddString(ProgramFullSpec);
    AddShort(#13'Host='); AddString(Host);
    AddShort(' User=');   AddString(User);
    AddShort(' CPU='); Add(dwNumberOfProcessors); Add('*');
    Add(wProcessorArchitecture); Add('-'); Add(wProcessorLevel); Add('-');
    Add(wProcessorRevision);
    AddShort(' OS='); Add(ord(OSVersion)); Add('.'); Add(wServicePackMajor);
    Add('='); Add(dwMajorVersion); Add('.'); Add(dwMinorVersion); Add('.');
    Add(dwBuildNumber);
    AddShort(' Wow64='); Add(integer(IsWow64));
    AddShort(' Freq='); Add(fFrequencyTimeStamp);
    if IsLibrary then begin
      AddShort(' Instance='); AddNoJSONEscapeString(InstanceFileName);
    end;
    Add(#13);
    AddShort('Environment variables=');
    Env := GetEnvironmentStringsA;
    P:=Env;
    while P^<>#0 do begin
      L := StrLen(PUTF8Char(P));
      if (L>0) and (P^<>'=') then begin
        AddNoJSONEscape(P,L);
        Add(#9);
      end;
      // go to end-of-string
      while P^<>#0 do
        inc(P);
      // next string
      inc(P);
    end;
    FreeEnvironmentStringsA(Env);
    CancelLastChar; // trim last #9
    Add(#13);
    AddClassName(self.ClassType); AddShort(' '+SYNOPSE_FRAMEWORK_VERSION+' ');
    AddDateTime(Now); Add(#13,#13);
  end;
  QueryPerformanceCounter(fStartTimeStamp);
  fHeaderWritten := true;
end;

h.hasenack

Member

From: Nijmegen, Netherlands

Registered: 2012-08-01

Posts: 173

Website

Re: Movin' towards 64bit...

I had to apply a similar fix to the TGDIPlys.Destroy as there was for TGDIPlus.Create:

destructor TGDIPlus.Destroy;
begin
  if fToken<>0 then begin
    if Assigned(fStartupHook.UnHook) then // Avoid AV if unassigned
      fStartupHook.UnHook(fStartupHookToken);
    Shutdown(fToken);
    fToken := 0;
  end;
  UnLoad;
end;

h.hasenack

Member

From: Nijmegen, Netherlands

Registered: 2012-08-01

Posts: 173

Website

Re: Movin' towards 64bit...

And now, I finally have a log, created by TestSQL3.exe in native 64bit mode.

This is what it looks like:

S:\Sources\LibSource\mORMot\SQLite3\bin\D17\Win64\TestSQL3.exe 0.0.0.0 (2012-12-18 17:08:47)Host=VMOBELIX-XE3 User=Hans CPU=4*9-21-258 OS=13.1=6.1.7601 Wow64=0 Freq=3579545Environment variables=ALLUSERSPROFILE=C:\ProgramData	APPDATA=C:\Users\Hans\AppData\Roaming	CommonProgramFiles=C:\Program Files\Common Files	CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files	CommonProgramW6432=C:\Program Files\Common Files	COMPUTERNAME=VMOBELIX-XE3	ComSpec=C:\Windows\system32\cmd.exe	FP_NO_HOST_CHECK=NO	HOMEDRIVE=C:	HOMEPATH=\Users\Hans	IBREDISTDIR=C:\Users\Public\Documents\InterBase\redist\InterBaseXE3	LOCALAPPDATA=C:\Users\Hans\AppData\Local	LOGONSERVER=\\VMOBELIX-XE3	MpConfig_ProductAppDataPath=C:\ProgramData\Microsoft\Windows Defender	MpConfig_ProductCodeName=AntiSpyware	MpConfig_ProductPath=C:\Program Files\Windows Defender	MpConfig_ProductUserAppDataPath=C:\Users\Hans\AppData\Local\Microsoft\Windows Defender	MpConfig_ReportingGUID=60AB9E58-A78C-4EBF-8CA4-1675C354FB2A	MYSRC=S:\Sources	NUMBER_OF_PROCESSORS=4	OS=Windows_NT	Path=C:\Program Files (x86)\CollabNet;C:\Program Files (x86)\Embarcadero\RAD Studio\10.0\bin;C:\Users\Public\Documents\RAD Studio\10.0\Bpl;C:\Program Files (x86)\Embarcadero\RAD Studio\10.0\bin64;C:\Users\Public\Documents\RAD Studio\10.0\Bpl\Win64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\TortoiseSVN\bin;S:\Sources\bpl\D17\Win32;S:\Sources\Libsource\Steema9\Compiled\Delphi17.win32\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.win32\System;S:\Sources\bpl\D17\iOS;S:\Sources\Libsource\Steema9\Compiled\Delphi17.iOS\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.iOS\System;S:\Sources\bpl\D17\Win64;S:\Sources\Libsource\Steema9\Compiled\Delphi17.win64\System;S:\Sources\Libsource\Steema9\TeeTree\Compiled\Delphi17.win64\SystemS:\Sources\bpl\D17\Win64;	PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC	PROCESSOR_ARCHITECTURE=AMD64	PROCESSOR_IDENTIFIER=AMD64 Family 21 Model 1 Stepping 2, AuthenticAMD	PROCESSOR_LEVEL=21	PROCESSOR_REVISION=0102	ProgramData=C:\ProgramData	ProgramFiles=C:\Program Files	ProgramFiles(x86)=C:\Program Files (x86)	ProgramW6432=C:\Program Files	PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\	PUBLIC=C:\Users\Public	SESSIONNAME=Console	SystemDrive=C:	SystemRoot=C:\Windows	TEMP=C:\Users\Hans\AppData\Local\Temp	TMP=C:\Users\Hans\AppData\Local\Temp	USERDOMAIN=VMOBELIX-XE3	USERNAME=Hans	USERPROFILE=C:\Users\Hans	windir=C:\WindowsTSynLog 1.18 2012-12-18T17:11:0820121218 17110836 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064E640 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110836 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064ED44 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F0D4 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F17A 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 
20121218 17110837 fail  TTestLowLevelCommon(00000000020DAA50) Low level common: TDynArray "" stack trace API 0000000000529129 000000000052EB92 000000000051DD97 000000000064F1FE 000000000051ED28 0000000000667E3B 000000000067077A 000000007720652D 000000007733C521 

..... (520KByte of the same line)

So I guess some work is still to be done

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,659

Website

Re: Movin' towards 64bit...

I've created a "meta-ticket" to prioritize 64 bit support.
See http://synopse.info/fossil/info/fb893df6fe

The corresponding sub-tickets have been moved to "Code defect" instead of "Feature request".
Some pending tickets you created will be removed, mainly by Sqlite3.dll use instead of static linking.