InternalSendRequest failure

h.hasenack · 2012-10-04 11:01:16

Using RAD studio XE3 (not knowning if this is the cause of the issue)
InternalSendRequest raises an EOSError Code 12030 on the WinHttpReceiveResponse call.
I have turned off my Firewall, and verified my server is running.

Even better: I am quite sure the server receives the responses as log entries are added to the log

C:\Users\Hans\Sources\LCSS\bin\D17\Win32\LCS_Server.exe 0.0.0.0 (2012-10-04 11:26:46)
Host=OBELIX User=Hans CPU=8*9-21-258 OS=13.1=6.1.7601 Wow64=1 Freq=4042841
TSQLLog 1.17 2012-10-04T12:55:48

20121004 12554824 srvr  TLCSRestServer(7EE747F0) GET LCSServer/TimeStamp -> 200
20121004 12555255 srvr  TLCSRestServer(7EE747F0) GET LCSServer/auth?UserName=Admin -> 200
20121004 12555357 srvr  TLCSRestServer(7EE747F0) GET LCSServer/auth?UserName=Admin -> 200
20121004 12555750 srvr  TLCSRestServer(7EE747F0) GET LCSServer/TimeStamp -> 200
20121004 12555949 srvr  TLCSRestServer(7EE747F0) GET LCSServer/TimeStamp -> 200
20121004 12560028 srvr  TLCSRestServer(7EE747F0) GET LCSServer/auth?UserName=Admin -> 200
20121004 12560104 srvr  TLCSRestServer(7EE747F0) GET LCSServer/auth?UserName=Admin -> 200

Nevertheless the client appears to be unable to receive the requests which are actually marked on the server as successfull (200). In the above I'm just trying to make an autorized connection.

I don't get what I'm doing wrong here. Maybe it's something with the PrepServer code ? - this routine runs in administrator mode without any trouble:

procedure PrepLCSServer;
VAR R:string;
begin
  R:=THttpApiServer.AddUrlAuthorize(cLCSServerName, cLCSDefaultNetworkPort, false);
  if R>'' then
    raise Exception.CreateFmt('PrepLCSServer failed. Error [%s]',[R]);
end;

COuld it have something to do with your recent changes to allow sub-urls?

FYI the same test routines work wothout a problem using the namedpipe and message clients, they use the same REST server instance so I think the REST server is not the real problem, but it is somewhere in the HTTP communication wrapper.

Help! I'm losing it

Hans

ab · 2012-10-04 12:38:49

The error is the following:

2030 ERROR_INTERNET_CONNECTION_ABORTED
The connection with the server has been terminated.

So I do not know what is wrong here.

Are the regression tests available in TestSQLite3.dpr running as expected?

I did not check with Delphi XE3, only with Delphi XE2, and I do not have the money nor need to upgrade to XE3.

h.hasenack · 2012-10-04 14:15:47

I fear some regression tests regarding HTTP Fail:

   Synopse mORMot Framework Automated tests
  ------------------------------------------


1. Synopse libraries

 1.1. Low level common:
  - System copy record: 22 assertions passed  161us
  - TDynArray: 519,427 assertions passed  151.46ms
  - TDynArrayHashed: 1,200,629 assertions passed  156.41ms
  - Fast string compare: 7 assertions passed  113us
  - IdemPropName: 10 assertions passed  104us
  - Url encoding: 105 assertions passed  1.00ms
  - IsMatch: 599 assertions passed  200us
  - Soundex: 35 assertions passed  89us
  - Numerical conversions: 785,498 assertions passed  133.56ms
  - Curr64: 20,039 assertions passed  1.46ms
  - CamelCase: 5 assertions passed  78us
  - Bits: 4,614 assertions passed  104us
  - Ini files: 7,004 assertions passed  23.94ms
  - Unicode - Utf8: 61,082 assertions passed  1.29s
  - Iso8601 date and time: 24,000 assertions passed  4.70ms
  - Url decoding: 1,100 assertions passed  278us
  - TSynTable: 873 assertions passed  4.87ms
  - TSynCache: 404 assertions passed  1.00ms
  - TSynFilter: 1,005 assertions passed  2.73ms
  - TSynValidate: 677 assertions passed  758us
  - TSynLogFile: 42 assertions passed  646us
  Total failed: 0 / 2,627,177  - Low level common PASSED  1.78s

 1.2. Low level types:
  - RTTI: 34 assertions passed  437us
  - Url encoding: 200 assertions passed  662us
  - Encode decode JSON: 250,233 assertions passed  167.71ms
  Total failed: 0 / 250,467  - Low level types PASSED  171.83ms

 1.3. Big table:
  - TSynBigTable: 19,232 assertions passed  67.76ms
  - TSynBigTableString: 16,054 assertions passed  31.64ms
  - TSynBigTableMetaData: 384,060 assertions passed  1.59s
  - TSynBigTableRecord: 452,185 assertions passed  4.02s
  Total failed: 0 / 871,531  - Big table PASSED  5.71s

 1.4. Cryptographic routines:
  - Adler32: 1 assertion passed  351us
  - MD5: 1 assertion passed  352us
  - SHA1: 5 assertions passed  353us
  - SHA256: 5 assertions passed  349us
  - AES256: 6,372 assertions passed  72.21ms
  - Base64: 11,994 assertions passed  119.05ms
  Total failed: 0 / 18,378  - Cryptographic routines PASSED  194.34ms

 1.5. Compression:
  - In memory compression: 12 assertions passed  221.80ms
  - Gzip format: 19 assertions passed  408.56ms
  - Zip format: 36 assertions passed  757.35ms
  - SynLZO: 3,006 assertions passed  84.67ms
  - SynLZ: 13,016 assertions passed  282.07ms
  Total failed: 0 / 16,089  - Compression PASSED  1.75s

 1.6. Synopse PDF:
  - TPdfDocument: 4 assertions passed  6.42ms
  - TPdfDocumentGDI: 6 assertions passed  14.27ms
  Total failed: 0 / 10  - Synopse PDF PASSED  21.50ms


2. mORMot

 2.1. Basic classes:
  - TSQLRecord: 47 assertions passed  356us
  - TSQLRecordSigned: 200 assertions passed  4.90ms
  - TSQLModel: 3 assertions passed  380us
  Total failed: 0 / 250  - Basic classes PASSED  6.41ms

 2.2. File based:
  - Database direct access: 10,138 assertions passed  220.81ms
  - Virtual table direct access: 12 assertions passed  3.17ms
  - TSQLTableJSON: 19,030 assertions passed  47.97ms
  - TSQLRestClientDB: 599,030 assertions passed  3.51s
  Total failed: 0 / 628,210  - File based PASSED  3.78s

 2.3. File based WAL:
  - Database direct access: 10,138 assertions passed  224.36ms
  - Virtual table direct access: 12 assertions passed  1.42ms
  - TSQLTableJSON: 19,030 assertions passed  43.01ms
  - TSQLRestClientDB: 599,030 assertions passed  3.44s
  Total failed: 0 / 628,210  - File based WAL PASSED  3.71s

 2.4. Memory based:
  - Database direct access: 10,136 assertions passed  197.53ms
  - Virtual table direct access: 12 assertions passed  1.30ms
  - TSQLTableJSON: 19,030 assertions passed  37.69ms
  - TSQLRestClientDB: 667,323 assertions passed  4.05s
  Total failed: 0 / 696,501  - Memory based PASSED  4.29s

 2.5. Client server access:
  - TSQLite3HttpServer: 21 assertions passed  8.01ms
     using THttpApiServer
!  - TSQLite3HttpClient: 1 / 1 FAILED  26.68ms
!  - Http client keep alive: 48 / 84 FAILED  23.16ms
     first in 22.18ms,
!  - Http client multi connect: 48 / 84 FAILED  14.61ms
     first in 13.65ms,
  - Named pipe access: 3,085 assertions passed  600.30ms
     first in 60.84ms, done in 139.89ms i.e. 7148/s, aver. 139us, 33.4 MB/s
  - Local window messages: 3,084 assertions passed  65.89ms
     first in 1.34ms, done in 61.64ms i.e. 16221/s, aver. 61us, 75.8 MB/s
  - Direct in process access: 3,052 assertions passed  59.11ms
     first in 816us, done in 58.13ms i.e. 17201/s, aver. 58us, 80.4 MB/s
  Total failed: 97 / 9,411  - Client server access FAILED  802.38ms

 2.6. Service oriented architecture:
  - Weak interfaces: 56 assertions passed  382us
  - Service initialization: 127 assertions passed  2.21ms
  - Direct call: 596,163 assertions passed  38.47ms
  - Server side: 596,173 assertions passed  38.40ms
  - Client side REST: 596,175 assertions passed  546.46ms
  - Client side JSONRPC: 596,173 assertions passed  621.30ms
  - Client side synchronized REST: 596,173 assertions passed  1.27s
  - Security: 135 assertions passed  1.11ms
  - Custom record layout: 596,173 assertions passed  567.17ms
  Total failed: 0 / 3,577,348  - Service oriented architecture PASSED  3.09s

 2.7. External database:
  - External records: 1 assertion passed  612us
  - Auto adapt SQL: 168 assertions passed  10.60ms
  - Crypted database: 253,272 assertions passed  265.75ms
  - External via REST: 243,436 assertions passed  908.07ms
  - External via virtual table: 243,436 assertions passed  1.65s
  Total failed: 0 / 740,313  - External database PASSED  2.84s


Synopse framework used: 1.17
SQlite3 engine used: 3.7.12.1
Generated with: Delphi XE3 compiler

Time elapsed for all tests: 28.19s
Tests performed at 4-10-2012 16:12:32

Total assertions failed for all test suits:  97 / 10,063,895

! Some tests FAILED: please correct the code.
Done - Press ENTER to Exit

I'd be more than happy to fix it for you in XE3. What things should I consider looking for..?

BTW I can confirm no errors are reported in XE2

Hans

Last edited by h.hasenack (2012-10-04 14:31:35)

ab · 2012-10-04 16:15:26

1. Try to use TSQLite3HttpClientWinINet or TSQLite3HttpClientWinSock clients instead of the TSQLite3HttpClientWinHTTP client - e.g. by changing the default "TSQLite3HttpClient = TSQLite3HttpClientWinHTTP;" in unit SQLite3HttpClient.

2. Check the changes of WinInet.pas or Windows.pas of all used constants and function prototypes between XE2 and XE3.
Perhaps some corrections are needed in SynCrtSock.pas.

h.hasenack · 2012-10-05 08:19:41

I have added a CheckOSError() routine that verifies the result of Http.SentHttpResponse, and it tells me the parameter is invalid. SO I guess something is messed up ad server side and not the client side. I'l investigate it some more.

procedure CheckOSError(aResult:HResult); inline;
begin
  if aResult<>NO_ERROR then
    raiseLastOSError(aResult)
end;

* Update there's a CheckOSError routine in the system.sysutils, but it's not inline

Last edited by h.hasenack (2012-10-05 08:20:48)

h.hasenack · 2012-10-05 09:28:19

I have found 1 error in SynCRTStock: the pHttpResonse should NOT be a VAR parameter but rather a CONST parameter

    SendHttpResponse: function(ReqQueueHandle: THandle; RequestId: HTTP_REQUEST_ID;
      Flags: integer; {var} const pHttpResponse: HTTP_RESPONSE; pReserved1: pointer;
      var pBytesSent: cardinal; pReserved2: pointer=nil; Reserved3: ULONG=0;
      pOverlapped: pointer=nil; pReserved4: pointer=nil): HRESULT; stdcall;

And something strange is going on here too, around line 3780 in SynCrtStock

            if integer(fCompress)<>0 then

FCompress is a THttpSocketCompressRecDynArray, so nothing like an ordinal. Mayby you mean length(FCompress)<>0 or maybe you mean a completely different field of some object?

Adding a FALSE AND makes no difference in the XE2 compiled app (still works fine) and on the XE3 app (still fails).

ab · 2012-10-05 09:48:48

There is no difference in the generated asm code between a var or const parameter.
So it won't change anything here.

fCompress can be type-casted to a pointer or even an integer (NativeInt / PtrInt) - so the code will have an issue in 64 bit mode, but it is correct (and confusing, for sure) under 32 bit.
I've made some code cleaning in the unit about 64 bit compilation.
See http://synopse.info/fossil/info/619937d947
But I suspect it won't fix anything for your issue with Delphi XE3.

h.hasenack · 2012-10-05 10:03:45

yes in 64bit mode it surely breaks. But for codereadability Id's trongly suggest using <>nil or even better length() when checking (dynamic) arrays.

Back to the XE3 problem
I cannot find (significant) difference between the Resp^ structures in XE2 and XE3. Also the FRequestID and Req^.RequestID seem to be fine.

Which leaves me frowning ... WTF is missing here... Do you know what to look for (apart from the obvious) of SendHTTPResponse parameters being reported as Invaid param,eters (error 87)

h.hasenack · 2012-10-05 10:39:25

Ab would you like to debug on my XE3 using TeamViewer?

h.hasenack · 2012-10-08 12:51:35

I have managed to slaughter the beast. After googlong a lot, I found out there were same issues with the translation of the HTT_DATA_CHUNK translation because of alignment differences between MS-C++ and Delphi. Obviously, in XE3 these alignment difderences have been changed, as applying the following patch makes everything work again. Ab I guess you will take this up a bit more cleanly using the $Include file and it's defines. I have verified this is working on both XE3 and XE2 now.

Regards - Hans

{$ifdef VER240} // XE3
  HTTP_DATA_CHUNK = record
    case DataChunkType: THttpChunkType of
    hctFromMemory: (
    FromMemory: record
      pBuffer: pointer;
      BufferLength: ULONG;
      Reserved1: ULONG;
      Reserved2: ULONG;
      Reserved3: ULONG;
    end; );
    hctFromFileHandle: (
    FromFileHandle: record
      ByteRange: HTTP_BYTE_RANGE;
      FileHandle: THandle;
    end; );
    hctFromFragmentCache: (
    FromFragmentCache: record
      FragmentNameLength: word;      // in bytes not including the #0
      pFragmentName: PWideChar;
    end; );
  end;
{$else} // not XE3
  HTTP_DATA_CHUNK = record
    case DataChunkType: THttpChunkType of
    hctFromMemory: (
    FromMemory: record
      Reserved1: ULONG;
      pBuffer: pointer;
      BufferLength: ULONG;
      Reserved2: ULONG;
      Reserved3: ULONG;
    end; );
    hctFromFileHandle: (
    FromFileHandle: record
      ByteRange: HTTP_BYTE_RANGE;
      FileHandle: THandle;
    end; );
    hctFromFragmentCache: (
    FromFragmentCache: record
      FragmentNameLength: word;      // in bytes not including the #0
      pFragmentName: PWideChar;
    end; );
  end;
{$endif}
  PHTTP_DATA_CHUNK = ^HTTP_DATA_CHUNK;

ab · 2012-10-08 15:44:13

This is very good news!

I've committed a potential fix.
See http://synopse.info/fossil/info/5e5353438a

In fact, there is a regression in Delphi XE3 about alignment of record with variable length (i.e. record with a nested case).
My fix proposal is to use 3 distinct records, according to the kind of HTTP_DATA_CHUNK needed.

Thanks for all your effort to slaughter the beast.

h.hasenack · 2012-10-08 15:59:25

Y're welcome

I'd rather suggest keeping the variable length record as this best reflects the (union) struct in the M$ http.h definitions. But it's up to you off course. Thanks for the quick fix!

I would make me very happy if you add the necessary CheckOSError() calls to the Http.SendHttpResponse() calls as well. And to any other place where it would make sense.

Regards

Hans

ab · 2012-10-08 17:29:44

Check the latest version from repository: you will find out that the error check was already included!

About several records, this was the most "elegant" way of handling the issue I found out.

Thanks for your finding and suggested fix!

mORMot Open Source

#1 2012-10-04 11:01:16

InternalSendRequest failure

#2 2012-10-04 12:38:49

Re: InternalSendRequest failure

#3 2012-10-04 14:15:47

Re: InternalSendRequest failure

#4 2012-10-04 16:15:26

Re: InternalSendRequest failure

#5 2012-10-05 08:19:41

Re: InternalSendRequest failure

#6 2012-10-05 09:28:19

Re: InternalSendRequest failure

#7 2012-10-05 09:48:48

Re: InternalSendRequest failure

#8 2012-10-05 10:03:45

Re: InternalSendRequest failure

#9 2012-10-05 10:39:25

Re: InternalSendRequest failure

#10 2012-10-08 12:51:35

Re: InternalSendRequest failure

#11 2012-10-08 15:44:13

Re: InternalSendRequest failure

#12 2012-10-08 15:59:25

Re: InternalSendRequest failure

#13 2012-10-08 17:29:44

Re: InternalSendRequest failure

Board footer