Double header? Problem with Ctxt.Call.InBody

hnb · 2015-08-26 14:01:56

Hi,

sometimes i have got problem with received data. Ctxt.Call.InBody contains wrong data (for published methods in TSQLRestServerDB). Part of "InBody" data is equal to header (I think that header in some way has been sent twice). Problem exist in low layer of framework, part of Context.InContent/InBody data after this code (inside module SynCrtSock in method THttpApiServer.Execute):

              Err := Http.ReceiveRequestEntityBody(fReqQueue,Req^.RequestId,flags,
                BufRead,InContentLength-InContentLengthRead,BytesRead);

is equal to something like this:

'POST /root/SendReceiptPosition HTTP/1.1'#$D#$A'Host: XXX.XXX.XX.XXX:XXX'#$D#$A'Accept: */*'#$D#$A'User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows; Synopse mORMot 1.18.1819 THttpClientSocket)'#$D#$A'Keep-Alive: 20000'#$D#$A'Connection: Keep-Alive'#$D#$A'Content-Encoding: synlz'#$D#$A'Content-Length: 1303'#$D#$A'Content-Type: application/json; charset=UTF-8'#$D#$A'Accept-Encoding: synlz'

after this part of "InContent/InBody" I have got incomplete compressed by synlz part of right data...

Server is compiled with Delphi XE2 (Windows 7).
Client is compiled with FPC trunk (Android ARMv6)

How is looking sending of data:

    with TJSONSerializer.CreateOwnedStream do
    try
      Add('[');
      for i := 0 to FCachedBillItemRecordList.Count - 1 do
      begin
        WriteObject(FCachedBillItemRecordList[i], [woDontStoreDefault]);
        Add(',');
      end;
      CancelLastComma;
      Add(']');

      SetText(LValue);
      if GClient.CallBack(mPOST, 'SendReceiptPosition', LValue, LResult) = HTML_SUCCESS then
        ShowMessage('OK');
    finally
      Free;
    end;

where GClient is TSQLHttpClientWinSock.

Best regards,
Maciej Izak

ab · 2015-08-26 14:35:33

Did you check GClient.CallBack() parameters order?

hnb · 2015-08-26 14:50:50

Yes. The error occurs from time to time (especially when I run the server - maybe some uninitialized pointer?). In most cases, I got the correct data for Ctxt.Call.InBody. Anyway Ctxt.Call.InBody should be always consistent.

At server side (THttpApiServer.Execute) I found only small bug:

fillchar(Req^,sizeof(HTTP_REQUEST),0); // should be fillchar(Req^,sizeof(HTTP_REQUEST) + 16384,0);

sadly, this patch does not help...

Last edited by hnb (2015-08-26 14:51:18)

ab · 2015-08-26 15:34:05

IMHO the fillchar() is correct.
The 16KB of data (which may be bigger later on) is safely re-filled at each request.
Only the HTTP_REQUEST structure is to be reset to 0.

hnb · 2015-08-27 07:09:10

Screens from Fiddler Web Debugger

correct (most of positions):
https://drive.google.com/file/d/0B4PZhd … sp=sharing

incorrect (from time to time):
https://drive.google.com/file/d/0B4PZhd … sp=sharing

any idea?

hnb · 2015-08-27 07:47:56

Looks like non standard compression like synlz in header make big difference. All works fine when in client is set:

GClient.Compression := [hcDeflate];

This client/server application was tested on real/life scenario (not localhost). Maybe some hardware can't deal with something like synlz :\ .

ab · 2015-08-27 09:30:22

Looks like if a HTTP proxy on the wire (e.g. a packet inspection / antivirus daemon) does not know the synlz compression scheme, so is messing things.
I'm afraid the same behavior may occur with hcSynShaAes, which is also proprietary.
This is an issue on the infrastructure side.

Perhaps, for a Delphi client/server, the websockets protocol is the best.
It is encrypted, compressed, and supports real-time callbacks if necessary.
This is what we use internally to communicated between nodes in our cloud projects.
AFAIK WebSockets, if upgrade is possible (it is sometimes blocked by proxies - see http://websocketstest.com and http://www.infoq.com/articles/Web-Sockets-Proxy-Servers ), is a good solution for communication between mORMot programs.

I've changed the default TSQLHttpClientGeneric.Compression to [hcDeflate], and updated the documentation for more accurate information about this point.
See http://synopse.info/fossil/info/4fafcc6d2c

Thanks a lot for this feedback.

hnb · 2015-08-27 11:11:20

Thanks for client/server proposition, but I need to keep current solution.

Looks like Android client (Nexus 5, client compiled for ARMv6 with FPC trunk) has problem with synlz compression scheme

Tested with android equivalent of Fiddler

bug mentioned above (from time to time)
https://drive.google.com/file/d/0B4PZhd … sp=sharing

and most of positions (correct):
https://drive.google.com/file/d/0B4PZhd … sp=sharing

also buggy session received from android (synlz):
https://drive.google.com/file/d/0B4PZhd … sp=sharing

I can't achieve any of above bugs with gzip. Maybe there is some pointer bug in android version of synlz compression?

Last edited by hnb (2015-08-27 11:13:34)

hnb · 2015-08-27 11:26:20

Looks like Android System or mORMot bug rather than issue on the infrastructure side (probably).

Last edited by hnb (2015-08-27 11:27:42)

ab · 2015-08-27 12:38:40

I did not test SynLZ on ARM... so perhaps there is an issue here.

hnb · 2015-08-27 13:40:45

It was mORMot bug. Please undo changes introduced in http://synopse.info/fossil/info/4fafcc6d2c
SynLZ works like a charm and it should be used as default Compression! It was SndBuf fault.

Console log for bug (from Android):

https://drive.google.com/file/d/0B4PZhd … sp=sharing

Patch attached:

https://drive.google.com/file/d/0B4PZhd … sp=sharing

Best regards ^^ ,
Maciej Izak

ab · 2015-08-27 14:04:40

Should have been fixed by http://synopse.info/fossil/info/7387bd5eb2

And reverted to default compression hcSynzip by http://synopse.info/fossil/info/267f83567e

Thanks for the patch!

mORMot Open Source

#1 2015-08-26 14:01:56

Double header? Problem with Ctxt.Call.InBody

#2 2015-08-26 14:35:33

Re: Double header? Problem with Ctxt.Call.InBody

#3 2015-08-26 14:50:50

Re: Double header? Problem with Ctxt.Call.InBody

#4 2015-08-26 15:34:05

Re: Double header? Problem with Ctxt.Call.InBody

#5 2015-08-27 07:09:10

Re: Double header? Problem with Ctxt.Call.InBody

#6 2015-08-27 07:47:56

Re: Double header? Problem with Ctxt.Call.InBody

#7 2015-08-27 09:30:22

Re: Double header? Problem with Ctxt.Call.InBody

#8 2015-08-27 11:11:20

Re: Double header? Problem with Ctxt.Call.InBody

#9 2015-08-27 11:26:20

Re: Double header? Problem with Ctxt.Call.InBody

#10 2015-08-27 12:38:40

Re: Double header? Problem with Ctxt.Call.InBody

#11 2015-08-27 13:40:45

Re: Double header? Problem with Ctxt.Call.InBody

#12 2015-08-27 14:04:40

Re: Double header? Problem with Ctxt.Call.InBody

Board footer