You are not logged in.
thank you very much ab, will do some dig, if it's not good maybe Ubuntu@Virtualbox will be my last choice.
It's good to hear that 2.3 stable become LTS, it's an important step to build bussiness software!
https://blog.synopse.info/?post/2024/10 … 2.3-Stable
and after download the branch, use new fpcupdeluxe(2.4.0e) build new FPC&Lazarus, here are some test:
1. fixes FPC3_2 & Lazarus stable, build mormot LTS2.3 mormottests to linux-x64, run the mormot2tests on WSL ubuntu, it will produce hundreds of fail.
2. Both FPC & Lazarus trunck, build mormot LTS2.3 mormottests to linux-x64, run the mormot2tests on WSL ubuntu, still some fail (which contains some network-failure) , but seems much better .
according to git history, there seems a lot of commits are about FPC3.3. so my question is , If I want to work with 2.3LTS, which FPC version is better?
First time I checked mormot was 5 years ago. The Mormot v1 was a huge investment to start using it. I gave up.
Two years ago, with the 2nd version, I started to read the code and it was really helpful to have different units as it is now.
The documentation is probably the main problem for a new adopter now.
Dividing mormot to different repositories is not going to help more. The increased time it will need, it is better to be invested in more documentation with more examples.
me too, v1 was too difficult to work with, v2 seems more friendly. Split mormot into sub-modules may please someone's taste, or maybe not.
IMHO some more documents and more realistic examples may help newbies to mormot world! And they will be amazed by this great framework!!
beautiful scenery & exquisite room :-) wish you a good journey
maybe this function helps
in-cds, your clientdataset contains modified data
in-rest clientrest, funciton will loop all cds delta (insert update delete), then invoke rest-batch uri call
in-oc tableOrmclass
function YGGCDSApplyREST(cds: TClientDataSet; const rest: TRestClientURI; oc: TOrmClass):Boolean;
var
prev: record
BM: TBookMark;
Active: Boolean;
ReadOnly: Boolean;
LogChanges: Boolean;
AfterScroll: TDataSetNotifyEvent;
end;
change: TClientDataset;
o: TOrm;
m,n :integer;
del: TID;
IDs: TIDDynArray;
insertBatchIdx: Array of Integer;
insertBM: Array of TBookMark;
batch : TRestBatch;
UpdateCSV: RawUTF8;
function oprops(isUpdate: Boolean): RawUTF8;
var
f: integer;
fn: String;
v: Variant;
u: RawUTF8;
begin
Result:= ''; //csv, updated fields. eg FP_Name,FP_Code
for f := 0 to cds.Fields.Count-1 do
begin
fn:= ( cds.Fields[f].FieldName );
u:= LowerCase( StringToUTf8(fn) );
if SameText(UpperCase(fn), 'ID') then continue; // skip id
if isUpdate and (
VarSameValue(
cds.Fields[f].OldValue,
cds.Fields[f].NewValue)
) then Continue; //skip field if not modify
v:= cds.Fields[f].Value;
if (1=1) and
(o.OrmProps.Fields.ByRawUtf8Name(u).OrmFieldType = oftUtf8Text) and
( (cds.Fields[f].DataType= ftString) or
(cds.Fields[f].DataType= ftWideString) ) and
Varisnull(v) then
ClearVariantForString(v);
o.SetFieldVariant(fn, v);
AddToCsv(u, Result);
end;
end;
begin
Result:= False;
if cds.ChangeCount =0 then Exit;
Prev.BM:= cds.Bookmark;
prev.AfterScroll:= cds.AfterScroll;
cds.AfterScroll:=nil;
change := TClientDataset.Create(nil);
batch := TRestBatch.Create(rest.Orm, oc);
try
SetLength(insertBatchIdx, 0);
rest.BatchStartAny(0);
cds.First;
while not cds.Eof do
begin
case cds.UpdateStatus of
usUnmodified: begin
end;
usModified: begin
o:= oc.Create;
try
o.IDValue:= cds.FieldByName('ID').AsInteger;
UpdateCSV:= oprops(True);
batch.Update(o, UpdateCSV);
finally
o.Free;
end;
end;
usInserted: begin
o:= oc.Create;
try
oprops(False);
SetLength(insertBatchIdx , Length(insertBatchIdx)+1);
insertBatchIdx[Length(insertBatchIdx)-1]:=
batch.Add(o, True);
SetLength(insertBM , Length(insertBM)+1);
insertBM[Length(insertBM)-1]:= cds.GetBookmark;
finally
o.Free;
end;
end;
usDeleted: begin
end;
end;
cds.Next;
end;
Change.Data:= cds.Delta;
Change.First;
while not Change.Eof do
begin
case Change.UpdateStatus of
usDeleted: begin
del:= Change.FieldByName('ID').OldValue;
batch.Delete(del);
end;
end;
Change.Next;
end;
Result:= rest.Orm.BatchSend(batch, IDs) = HTTP_SUCCESS;
if not Result then
DatabaseError(UTF8ToString(rest.LastErrorMessage));
if Length(insertBatchIdx) >0 then begin // fill REST-Returned Inserted IDs
Change.Data:= cds.Data;
Change.LogChanges:= False;
Change.MergeChangeLog;
for m := Low(IDS) to High(IDs) do // [200, 200, 15,16,17, 404]
begin
for n:= Low(insertBatchIdx) to High(insertBatchIdx) do // [2,3,4]
if m= insertBatchIdx[n] then
begin
Change.Bookmark:= insertBM[m];
Change.Edit;
Change.FieldByName('ID').AsInteger:= IDS[m];
Change.Post;
end;
end;
cds.Data:= Change.Data;
end;
if Assigned(Prev.BM) then
cds.Bookmark:= Prev.BM;
Result:= True;
finally
cds.AfterScroll:= prev.AfterScroll;
batch.Free;
change.Free;
end;
end;
wish you a pleasant holiday :-)
problem exists with mormot2, tested below:
D7 + mormot
D7 + mormot2
Congratulations!!! thank you ab!
you can try this url
https://drive.google.com/file/d/14hRTB7 … sp=sharing
hi EMartin , made a mormot2 version TSynRestDataset :
just verify the possiblity, my sample-exe was build with mormot static SQlite3 Engine, place the FTS-tokenizer-extension-dll to exe folder
the sample-exe comes from : mormot2/ex/extdb-bench/PerfTestCase
sqlite3.db_config(Client.DB.DB, SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION, 1);
Load:= Client.DB.SQLite3Library.load_extension(Client.DB.DB, 'simple.dll', 0, Msg);
if not (Load= SQLITE_OK) then
raise Exception.Create('Error Message');
then some where you can create a table with new tokenizer
Client.DB.Execute('CREATE VIRTUAL TABLE t33 USING fts5(text, tokenize = ''simple''); ');
In mormot2, add this line before Client.DB.SQLite3Library.load_extension() will work as expected.
sqlite3.db_config(Client.DB.DB, SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION, 1);
according to sqlite3.h, the root cause is :
** ^Extension loading must be enabled using
** [sqlite3_enable_load_extension()] or
** [sqlite3_db_config](db,[SQLITE_DBCONFIG_ENABLE_LOAD_EXTENSION],1,NULL)
** prior to calling this API,
** otherwise an error will be returned.
and thanks to AB for this amazing framework and lasting enthusiasm
in fact I wants to load a FTS5 tokenizer extension, which mainly deals with Chinese characters. As Chinese differs a lot with alphabetic language (English French ...) , tokenizer is very complicated , so some professional tokenizer extension are indispensable.
tried with mormot1 also , modify sources of SynDBExplorer(mormot 1 sample 12 )to load an external SQLite3.dll (3.39.0 , this dll can load extension, verified by SQLiteExpert tool ),
then excute SQL: select load_extension('./simple.dll');
got Exception: "not authorized " too.
You need to call the sqlite3_load_extension() function.
So perhaps it is easier to define in sqlite3mc.c and recompile it:
#define SQLITE_ENABLE_ICU
A custom LIKE function calling the ICU comparison available in SynFPCLinux.pas may be a better alternative.
Hi ab, just tried to load a tokenizer extension, in mormot2 \mORMot2\ex\extdb-bench\Perftestcases , add this line:
if not Client.DB.SQLite3Library.load_extension(Client.DB.DB, 'simple', '', Msg)= SQLITE_OK then
raise Exception.Create('Error Message');
the load_extension returns SQLITE_OK, but Msg contains 'not authorized' , am I missed something ?
scared me haha
restClient.Server.DB.Synchronous := smOff;
restClient.Server.DB.LockingMode := lmExclusive;
guess maybe helpful?
Hi, ab, Let's consider this situation :
Client A,Client B both wants to update the same record (eg: TSQLtask.workername ), maybe at same time.
0:00 ClientA REST-retrive table-task, id is 27
0:01 ClientB REST-retrive table-task, id is 27
0:05 ClientA REST-update table-task 27, set workername=A
0:06 CilentB REST-update table-task 27, set workername=B
now , ClientA & ClientB they all feels like gets the task, but ClientA is failed actually.
So, edwinsn's proposal is reasonable, if there is some method like :
TSQLRESTClient.Update(Value: TSQLRecord; wherefields: array of const; wherevalues: array of const; out rowsaffected:Integer ).
We call call this new method, then eventually mormot.pas will send SQL like: update task set workername=A where id =27 and workername='' , and we'll check if rowsaffected =1, then everything will work as expected.
It seems now, mormot level does't care about syndb's ISQLDBStatement.UpdateCount, every REST Request (Create&Upate) the server received will turned as Insert&Update SQL , then server return HTTPOK if there is no exception, regardless how many rows affected. so I add a rowsaffected parameter above.
And IMHO almost every ORM framework takes care of the actual rows affected, it's import to developers. Syndb is so powerful, and REST C/S ORM is a unique &flexible feature, but this little function insufficient may requires developer to write a SOA method to fulfil the data integrity job. Hope this feature worth a consideration.
Hi mpv,
Thank you for your help with the linux problems I've posted. And I've met some thread sync problems with SynDBOracle@Linux too, maybe some transaction-deadlock, still trying to find out the root cause.
and Did you tried with the synDBODBC , will this one be more stable?
Thanks mpv, your code works fine , it helps gracefully quit app. And fpPause will just sleep mainthread but not respond to later sub-thread synchorize , and CheckSynchronize(10) will do the work.
uh , my low level bug, thank you ab.
used Syncommons.ConsoleWaitForEnterKey , which will give mainthread chance to work on windows, so works well on windows; but this function just Readln on linux, so mainthread is starved to end.
And is there some util-function can do this work ?
Hi ab, I'm trying to add a new restserver which exposes interface service running in Mainthread. But if client call the new service, server will hang on this request till timeout.
Server side compiled with FPC(tried with: 3.2.0-beta-r41498 & 3.1.1-r39235 ) + mormot (latest code on github) running on Linux x64.
server code like this:
aServer.ServiceRegister(TServiceRemoteSQL,[TypeInfo(IRemoteSQL)], sicPerThread).SetOptions([], []);
bServer.ServiceRegister(TServiceRemoteTS,[TypeInfo(IRemoteTS)], sicClientDriven).SetOptions([], [optExecInMainThread,optFreeInMainThread]);
aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer, bServer]);
Request URL:
http://172.16.13.122:6610/boot/RemoteTS … 595fe9b2ca //this request fine
http://172.16.13.122:6610/boot/RemoteTS … 59a13f2640 //this request will hang
server log :
20190308 02331860 # srvr mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30) User 172.18.0.3 POST root/RemoteSQL.ExeBin Interface=200 out=397 B in 3.95ms
20190308 02331860 # - 00.004.030
20190308 02331901 # + mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0).URI POST boot/RemoteTS._instance_?session_signature=6a48c0cf007f71595fe9b2ca in=2 B
20190308 02331901 # auth mORMot.TSQLRestRoutingREST(00007F9CEE8F5D30) User/1783152847 172.18.0.3
20190308 02331901 # debug mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0) TServiceFactoryServer.InternalInstanceRetrieve: Adding RemoteTS(00007F9CF07B2600) instance (id=1) count=1
20190308 02331901 # srvr mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0) User 172.18.0.3 POST boot/RemoteTS._instance_ Interface=200 out=12 B in 124us
20190308 02331901 # ret mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0) {"result":1}
20190308 02331901 # - 00.000.234
20190308 02331903 # + mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0).URI POST boot/RemoteTS.TransactionStart/1?session_signature=6a48c0cf007f7159a13f2640 in=2 B
20190308 02331903 # auth mORMot.TSQLRestRoutingREST(00007F9CEE8F5D30) User/1783152847 172.18.0.3
20190308 02334961 $ info SetThreadName 00007F9CEFB17700=TSQLHttpServer 8888/root boot TSynThreadPoolSubThread
20190308 02334961 $ trace mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30) BeginCurrentThread(TSynThreadPoolSubThread) root=root ThreadID=00007F9CEFB17700 ThreadCount=3
20190308 02334961 $ trace mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0) BeginCurrentThread(TSynThreadPoolSubThread) root=boot ThreadID=00007F9CEFB17700 ThreadCount=3
20190308 02334961 $ + mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0).URI POST boot/RemoteTS._free_/1?session_signature=6a48c0cf007f71d255b6dea8 in=2 B
20190308 02334961 $ auth mORMot.TSQLRestRoutingREST(00007F9CED18C970) User/1783152847 172.18.0.3
20190308 02342161 % info SetThreadName 00007F9CEFA96700=TSQLHttpServer 8888/root boot TSynThreadPoolSubThread
20190308 02342161 % trace mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30) BeginCurrentThread(TSynThreadPoolSubThread) root=root ThreadID=00007F9CEFA96700 ThreadCount=4
20190308 02342161 % trace mORMot.TSQLRestServerFullMemory(00007F9CF08E6DF0) BeginCurrentThread(TSynThreadPoolSubThread) root=boot ThreadID=00007F9CEFA96700 ThreadCount=4
20190308 02342161 % + mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30).URI GET root/Auth?UserName=User&Session=520179731&session_signature=1f015013007f724f984fdeb5 in=0 B
20190308 02342161 % auth mORMot.TSQLRestRoutingREST(00007F9CED124BF0) User/520179731 172.18.0.3
...
20190308 02405509 % + mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30).URI GET root/timestamp in=0 B
20190308 02405509 % srvr mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30) 172.18.0.3 GET root/timestamp Method=200 out=12 B in 39us
20190308 02405509 % ret mORMot.TSQLRestServerFullMemory(00007F9CF08E4D30) 135502113335
20190308 02405509 % - 00.000.133
As @Chaa stated , your server code users Restful auth, but C# send HTTP Basic auth, so you get 403
see docs https://synopse.info/files/html/Synopse … #TITLE_535
@EgonHugeist
my server fetch data by ISQLDBRows.FetchAllToBinary(a Response:TRawByteStringStream as a parameter) and return Response ,
then client will create Tclientdataset uses SynDBMidasVCL.ToClientDataSet() funtion.
the difference comes from FetchAllToBinary, below is the partition about column definition in the byteString:
SynDBOracle
.. 41 4D 45 06 0B 0A
ZEOS
.. 41 4D 45 06 00 0A
the same three byte (41 4D 45) represents Fields name, then type & field size. we can see ZEOS lost 0B, which means length 11.
@EgonHugeist
for Oracle- mycolumn: varchar2(10)
SyndbOracle will produce dataset contains
<FIELD fieldtype="string.uni" attrname="mycolumn" WIDTH="22"/>
while SynDBZeos will produce dataset like:
<FIELD attrname="mycolumn" fieldtype="bin.hex" SUBTYPE="Text"/>
Thank you very much , ab, for so quick response.
as you stated, modify function TSynAnsiConvert.AnsiBufferToUnicode,
original code result := Dest+SourceChars;
fixed code result := Dest+length(tmp);
Works fine now , got exactly same data with D7@win32.
And I'm curios is it depends on OS? My ubuntu server installed Chinese locales, if it's not, will the Ansi->unicode conversion OK?
No it's a very old database, all string column is varchar2. And my server code fetch data uses: ISQLDBRows.FetchAllToBinary, return binary to client.
the situation looks like :
oracle-column: varchar2(10)
Delphi7@win32 got value(hex):
E7 94 B7, these three bytes are one character in Chinese means "man", and it's correct.
FPC@linux got value(hex):
E7 94 B7 E7 BA A7, the last three bytes are not expected.
thank you mpv, seems some FPC ansiconvert works failed, but not got a clue. Any hint is helpful.
SynDBOracle.pas when compiled with FPC target linux64, some varchar2 column value are larger ( 2 widechar or 5-6 bytes ) then expected, and the content encoding is correct (oracle-db is codepage 936, the client-got utf8 correctly) , but something more weired is some column-value are just as exptected.
The server-db connection info:
20181204 05594455 # SQL SynDBOracle.TSQLDBOracleStatement(00007FD647A5C830) SELECT NLS_CHARSET_ID(PROPERTY_VALUE) FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME='NLS_CHARACTERSET'
20181204 05594456 # DB SynDBOracle.TSQLDBOracleStatement(00007FD647A5C830) 1 row(s) in 290us
20181204 05594456 # info SynDBOracle.TSQLDBOracleConnection(00007FD647AEB220) Connected to 172.16.16.100/orcl as haha with libclntsh.so rev. 11.2.0.4, codepage 936 (852/ZHS16GBK)
some other tests:
1. Same code & same sql compiled with Delphi7 running on windows is OK
2. Same code( just repacle syndboracle with zdbc) & same sql compiled with FPC running on linux 64 column-length is OK
3. btw , the zdbc resultset seems lost field length info, so it's not an alternative to me .
Meet same problem here, Syndb support fetch unlimted columns, but this FetchAllToBinary break the completeness.
And FetchAllToJSON() will loose ColumnValueDBSize (original DBMS declared fieldsize ) information , FetchAllToBinary is the only way in my enviorment. hope it will be corrected.
thanks for reply pvn0, tried but failed. Add FClient.Services.Resolve on OnAuthentificationFailed event, then framework will post _free_ _instances_ , but both will get 403 Forbidden. Guess may because after server restart, client will need auth agian.
#1 http://10.20.40.254:8888/root/RemoteSQL.ExeBin/2?session_signature=6ba3482b00090f5843ebd140 Get 200 OK, then server restarted.
#2 http://10.20.40.254:8888/root/RemoteSQL.ExeBin/2?session_signature=6ba3482b00090f7180a5ae84 Get 403 Forbidden
#3 http://10.20.40.254:8888/root/RemoteSQL._free_/2?session_signature=6ba3482b00090f7a01c51054 FClient.Services.Resolve , but Get 403 Forbidden
#4 http://10.20.40.254:8888/root/RemoteSQL._instance_?session_signature=6ba3482b00090f7a0909a6ce FClient.Services.Resolve , but Get 403 Forbidden
Currently, I'm force FService:=nil and FreeAndNil(RestClient), then wait my app recreate Restclient&doInit on next-request.
Backend server maybe restarted without notifcation to clients. so just after server restart, the next-client-request contains the old-ClientDrivenID will get an resonse-error: 403 Forbidden.
My first try was on restclient.OnAuthentificationFailed event, add some code:
fClient.SetUser('User','synopse');
fClient.ServiceContainer.Info(TypeInfo(IRemoteSQL)).Get(fService);
Result:= False;
but meets some curious problem, about the ClientDrivenID, seems later request totally ignore-ClientDrivenID. And finally I lost in TServiceFactoryClient.InternalInvoke:
...
if (status=HTTP_UNAUTHORIZED) and (clientDrivenID<>'') and
(fInstanceCreation=sicClientDriven) and (aClientDrivenID<>nil) then begin
{$ifdef WITHLOG}
log.Log(sllClient,'% -> try to recreate ClientDrivenID',[resp],self);
{$endif}
aClientDrivenID^ := 0;
uri := baseuri;
fRest.ServicesRouting.ClientSideInvoke(uri,aMethod,aParams,'',sent);
status := aClient.URI(uri,'POST',@resp,@head,@sent).Lo;
end;
...
Here , can we have an optional recreate judgement? or some callbacks?
...
if aServiceCustomAnswer=nil then begin
....
end else begin
// custom answer returned in TServiceCustomAnswer
fRest.InternalLog('TServiceCustomAnswer(%) returned status=% len=%',
[head,status,length(resp)],sllServiceReturn);
aServiceCustomAnswer^.Status := status;
aServiceCustomAnswer^.Header := head;
aServiceCustomAnswer^.Content := resp;
if aClientDrivenID<>nil then
aClientDrivenID^ := 0;
end;
...
And here, why just let aClientDrivenID^ := 0 ? My backend-service function returned TServiceCustomAnswer. it seems this line occurs the problem.
Original TSynTableStatement has a property: OrderByDesc:Boolean, it will remember whether last word=desc.
But think if a SQL like this: select name,price,year from cata where 1=1 order by price desc, year, name . Then TSynTableStatement will work incorrectly, it will ignore , year,name . So I have made some changes , add a property: OrderByDescField:TSQLFieldIndexDynArray, it will remember every SQL-Field marked with "DESC" .
Test passed with TestSQL3 on FPC&Delphi7 on windows.
I have created a P/R on github, https://github.com/synopse/mORMot/pull/123, Please review it, thanks ab.
Maybe you can use fiddler (webdev proxy software ) record your client http packet, then replay the packet from fiddler, see if it also takes 20 sec , may help you get clue (whether network problem or winhttplib or RESTClient code flaw)
thanks a lot mpv, just have made nginx_proxy_mormot works fine on linux, and performance seems great!
Tested with Jmeter (60 thread target 6 service-url ,and test 3 round), backend mormot is SOA-service, mainly issue sql query aginst oracle-db then return JSON to Jmeter.
client-Jmeter keepalive, Post to mormot, result is: 466KB/sec, 7QPS;
client-Jmeter keepalive, Post to nginx_proxy_mormot, result is: 13MB/sec, 190QPS;
client-Jmeter shortconnect , Post to mormot, result is: 15MB/sec, 225QPS;
client-Jmeter shortconnect , Post to nginx_proxy_mormot, result is: 14MB/sec, 207QPS;
when service on windows HTTP.sys:
client-Jmeter keepalive, Post to mormot(HTTP.sys), result is: 4MB/sec, 70QPS;
good news for nginx&mormot@linux
@mpv Could you pls paste your nginx configuration ? Same keep-alive performance prob occurs, and not familiar with nginx, That will be very helpful to me !
In order to stick SOA-method thread enviorment within TSynThreadPool , trying modify THREADPOOL_MAXWORKTHREADS = 0,
then test with Jmeter (100 client-thread concurrent ) , totaly 1000 http-request, result in about 30% broken connection ( Non HTTP response message: Software caused connection abort: recv failed ).
maybe the problem occurs at function TSynThreadPool.Push(aContext: pointer): boolean;
{$ifdef USE_WINIOCP}
result := PostQueuedCompletionStatus(fRequestQueue,0,0,aContext);
{$else}
and now the syncrtsock-IOCP is not fullfeature, https://synopse.info/forum/viewtopic.php?id=4049.
It seems, HTTPAPI @windows + SOA sicPerthread is more proper.
HTTPAPI mode works fine, 32 thread serves all 100-Jmeter clients, DB connection is reuesed by server thread.
The prev problem seems occurs with THttpServer (socket based ), the thread pool only serve reqest, but further calc&rest logic is dispatched with dedicate THttpServerResp thread everytime.
So when THttpServer working with sicPerThread mode SOA-service, the SOA-interface instance lifetime is so short.
/// a simple Thread Pool, used for fast handling HTTP requests of a THttpServer
// - will handle multi-connection with less overhead than creating a thread
// for each incoming request
// - will create a THttpServerResp response thread, if the incoming request is
// identified as HTTP/1.1 keep alive, or HTTP body length is bigger than 1 MB
TSynThreadPoolTHttpServer = class(TSynThreadPool)
// here aContext is a pointer(TSocket=THandle) value
procedure Task(aCaller: TSynThread; aContext: Pointer); override;
if (fServer.ServerKeepAliveTimeOut>0) and
(fServer.fInternalHttpServerRespList.Count<THREADPOOL_MAXWORKTHREADS) and
(ServerSock.KeepAliveClient or
(ServerSock.ContentLength>THREADPOOL_BIGBODYSIZE)) then begin
// HTTP/1.1 Keep Alive (including WebSockets) or posted data > 1 MB
// -> process in dedicated background thread
fServer.fThreadRespClass.Create(ServerSock,fServer);
and my question is why THttpServer internally desinged like this ?
Hi ab, Some sicPerThread problems
code based on Sample-16, I made little modification on sicPerThread, the result is not expected.
{ TServiceRemoteSQL }
procedure TServiceRemoteSQL.Connect(aEngine: TRemoteSQLEngine;
const aServerName, aDatabaseName, aUserID, aPassWord: RawUTF8);
..
begin
if fProps<>nil then
raise Exception.Create('Connect called more than once');
if TYPES[aEngine]=nil then
raise Exception.CreateFmt('aEngine=%s is not supported',
[GetEnumName(TypeInfo(TRemoteSQLEngine),ord(aEngine))^]);
fProps := TYPES[aEngine].Create(aServerName,aDatabaseName,aUserID,aPassWord);
end;
function TServiceRemoteSQL.Execute(const aSQL: RawUTF8; aExpectResults, aExpanded: Boolean): RawJSON;
var res: ISQLDBRows;
begin
if fProps=nil then
Connect(rseOracle, '10.20.40.251/orcl', '', 'ydjg508_mid', 'pde');
res := fProps.ExecuteInlined(aSQL,aExpectResults);
if res=nil then
result := '' else
result := res.FetchAllAsJSON(aExpanded);
end;
destructor TServiceRemoteSQL.Destroy;
begin
FreeAndNil(fProps);
inherited;
end;
aServer := TSQLRestServerFullMemory.Create(aModel,false);
aServer.ServiceRegister(TServiceRemoteSQL,[TypeInfo(IRemoteSQL)],sicPerThread);
aHTTPServer := TSQLHttpServer.Create(PORT_NAME,[aServer],'+',useHttpSocket);
The server log displayed by Synlogview image:
The server log :
20180522 09343026 "O trace mORMot.TSQLRestServerFullMemory(020F29A0) BeginCurrentThread(THttpServerResp) root=root ThreadID=00001D40 ThreadCount=126
20180522 09343908 "O trace mORMot.TSQLRestServerFullMemory(020F29A0) EndCurrentThread(THttpServerResp) ThreadID=00001D40 ThreadCount=102
20180522 09345313 !$ trace mORMot.TSQLRestServerFullMemory(020F29A0) BeginCurrentThread(THttpServerResp) root=root ThreadID=00001D40 ThreadCount=111
20180522 09345650 !$ trace mORMot.TSQLRestServerFullMemory(020F29A0) EndCurrentThread(THttpServerResp) ThreadID=00001D40 ThreadCount=111
Client is apache JMeter, with 100 thread simultaneously, each thread will send 5 HTTP request body (JSON encoded SQL expected by the RemoteSQL.Execute URL).
It seems like RESTServer randomly pick one thread then create TServiceRemoteSQL and process DB logics then release it.
With my expectation, the second picture #68 thread with threadID=00001D40 should run TServiceRemoteSQL.Execute and the fProps should be assigned .
But it seems the TServiceRemoteSQL instance is destroyed by previous #175 thread.
Do some research again, there are some misleading in my answer. ab you are right, The framework has no strip chunked logic. But the winhttp has.
https://stackoverflow.com/questions/251 … tpreaddata
https://msdn.microsoft.com/en-us/librar … 2147217396
Starting in Windows Vista and Windows Server 2008, WinHttp enables applications to perform chunked transfer encoding on data sent to the server. When the Transfer-Encoding header is present on the WinHttp response, WinHttpReadData strips the chunking information before giving the data to the application.
API-WinHttpReadData() auto strip the chunked-response, then send the data to application. Only when we wants to send chunked request, some manually code is needed before we call WinHttpSendRequest().
So my last correct output is a coincidence, winhttp does it. then CompressGZip() can decompress correctly.
Got it , SynCommons.IsContentCompressed SynZip.CompressGZip can do it.
use THttpClientSocket / TWinhttp to grab some html-content, but response is chunked.
searched in syncommons.pas & syncrtsock.pas without any clue, any hint? thank you !
thank you ab, for so quick response and continuous passion on mormot !
just submit a pullrequest on github
Fiddler is a wonderful debug tools which could catch every winhttp/winitnet call , and when it startsup , it could set winhttp proxy automatically. so it is very convenient on momort c/s communication debugging.
but now in the SynCrtSock.pas , it will bypass the system proxy
procedure TWinHTTP.InternalConnect(ConnectionTimeOut,SendTimeout,ReceiveTimeout: DWORD);
begin
if fProxyName='' then
// add https://msdn.microsoft.com/en-us/library/windows/desktop/aa384122 ?
OpenType := WINHTTP_ACCESS_TYPE_NO_PROXY else
OpenType := WINHTTP_ACCESS_TYPE_NAMED_PROXY;
fSession := WinHttpOpen(pointer(Ansi7ToUnicode(fUserAgent)), OpenType,
pointer(Ansi7ToUnicode(fProxyName)), pointer(Ansi7ToUnicode(fProxyByPass)), 0);
and MSDN ( https://msdn.microsoft.com/en-us/librar … s.85).aspx ) told us : WINHTTP_ACCESS_TYPE_DEFAULT_PROXY -- this option is deprecated on Windows 8.1 and newer. Use WINHTTP_ACCESS_TYPE_AUTOMATIC_PROXY instead.
so I wonder if it is possible to change the implemention here to : if client >8.1 then use WINHTTP_ACCESS_TYPE_AUTOMATIC_PROXY . it could help coding&debuging a lot.
https://github.com/lminuti/Delphi-OpenSSL
https://github.com/cybexr/Delphi-unit-OpenSSL
both are OpenSSL delphi wrapper , maybe helpful
mORMot framework is definitely productive framework and highly optimized ,to build an enterprise software with compiling programming tools (C C++ Pascal Java .net ) , maybe mORMot is the best. But somehow, it is not friendly to newbie (lacks of production-demo; SAD is comprehensive but feels like a well-builded churh, grandness but scaffold is disassembled, we can't figure out how it was builded).
eg: the mormot.pas is so complicatd , there are cornerstone class(TSQLRest, TSQLRecord),and also lots of aided classes(TSQLRestServerMonitor, TSQLRecordServiceLog). from my personal perspective, make some separation between the core and the assistant maybe seems like more reader-friendly and more maintainable.
Good idea ! Like apple reinvent phone, mORMot is a great framework, we can say ab reinvent Delphi :-)
As ab said, The deep integration of the mORMot framework makes it unusual and it seems unbelievable. The Java Spring project divied into sring MVC, spring Cloud, spring Data, spring Security ... and each of them can be used alone. Well refactoring will make mORMot more developer friendly and more persuasively.
maybe such subprojects could be introduced:
mORMot RTL - rawutf8 dynarray docvariant ... from SynCommons.pas
mORMot JSON- fast Json enc-dec from SynCommons.pas
mORMot REST- core C/S ORM, TSQLRest client server storage .. from mormot.pas
mORMot SOA- interface based service from mormot.pas
mORMot SQL- remote db ... from syndb.pas
mORMot noSQL- mongo db
mORMot NET- core http-communication, http.sys, iocp epoll ...
mORMot security- AES HMAC SHA JWT... from SynCrypto.pas
mORMot C- sqlite3 openssl ecc .. some C project, like FPC said write once compile anywhere, C&Pascal are best friend.
mORMot Report - pdf based report generation
mORMot MVC- mustache based MVC
fContract JSON content on windows:
{"contract":"ComplexCalculator","implementation":"single","methods":[{"method":"Add","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"integer"},{"argument":"n2","direction":"in","type":"integer"},{"argument":"Result","direction":"out","type":"integer"}]},{"method":"Multiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"int64"},{"argument":"n2","direction":"in","type":"int64"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"Subtract","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"double"},{"argument":"n2","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"double"}]},{"method":"ToText","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Value","direction":"in","type":"currency"},{"argument":"Result","direction":"both","type":"utf8"}]},{"method":"ToTextFunc","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Value","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"utf8"}]},{"method":"Swap","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"both","type":"double"},{"argument":"n2","direction":"both","type":"double"}]},{"method":"StackIntMultiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"integer"},{"argument":"n2","direction":"in","type":"integer"},{"argument":"n3","direction":"in","type":"integer"},{"argument":"n4","direction":"in","type":"integer"},{"argument":"n5","direction":"in","type":"integer"},{"argument":"n6","direction":"in","type":"integer"},{"argument":"n7","direction":"in","type":"integer"},{"argument":"n8","direction":"in","type":"integer"},{"argument":"n9","direction":"in","type":"integer"},{"argument":"n10","direction":"in","type":"integer"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"StackFloatMultiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"double"},{"argument":"n2","direction":"in","type":"double"},{"argument":"n3","direction":"in","type":"double"},{"argument":"n4","direction":"in","type":"double"},{"argument":"n5","direction":"in","type":"double"},{"argument":"n6","direction":"in","type":"double"},{"argument":"n7","direction":"in","type":"double"},{"argument":"n8","direction":"in","type":"double"},{"argument":"n9","direction":"in","type":"double"},{"argument":"n10","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"SpecialCall","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Txt","direction":"in","type":"utf8"},{"argument":"Int","direction":"both","type":"integer"},{"argument":"Card","direction":"both","type":"cardinal"},{"argument":"field","direction":"in","type":"TSynTableFieldTypes"},{"argument":"fields","direction":"in","type":"TSynTableFieldTypes"},{"argument":"options","direction":"both","type":"TSynTableFieldOptions"},{"argument":"Result","direction":"out","type":"TSynTableFieldTypes"}]},{"method":"ComplexCall","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Ints","direction":"in","type":"TIntegerDynArray"},{"argument":"Strs1","direction":"in","type":"TRawUTF8DynArray"},{"argument":"Str2","direction":"both","type":"TWideStringDynArray"},{"argument":"Rec1","direction":"in","type":"TVirtualTableModuleProperties"},{"argument":"Rec2","direction":"both","type":"TSQLRestCacheEntryValue"},{"argument":"Float1","direction":"in","type":"double"},{"argument":"Float2","direction":"both","type":"double"},{"argument":"Result","direction":"out","type":"TSQLRestCacheEntryValue"}]},{"method":"Substract","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"TComplexNumber"},{"argument":"n2","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"TComplexNumber"}]},{"method":"IsNull","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"boolean"}]},{"method":"TestBlob","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"TServiceCustomAnswer"}]},{"method":"TestVariants","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Text","direction":"in","type":"utf8"},{"argument":"V1","direction":"in","type":"variant"},{"argument":"V2","direction":"both","type":"variant"},{"argument":"Result","direction":"out","type":"variant"}]},{"method":"Collections","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Item","direction":"in","type":"TCollTest"},{"argument":"List","direction":"both","type":"TCollTestsI"},{"argument":"Copy","direction":"out","type":"TCollTestsI"}]},{"method":"GetCurrentThreadID","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Result","direction":"out","type":"cardinal"}]},{"method":"GetCustomer","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"CustomerId","direction":"in","type":"integer"},{"argument":"CustomerData","direction":"out","type":"TCustomerData"},{"argument":"Result","direction":"out","type":"boolean"}]},{"method":"FillPeople","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"People","direction":"both","type":"TSQLRecordPeople"}]}]}
fContract JSON content on linux:
{"contract":"ComplexCalculator","implementation":"single","methods":[{"method":"Add","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"integer"},{"argument":"n2","direction":"in","type":"integer"},{"argument":"Result","direction":"out","type":"integer"}]},{"method":"Multiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"int64"},{"argument":"n2","direction":"in","type":"int64"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"Subtract","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"double"},{"argument":"n2","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"double"}]},{"method":"ToText","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Value","direction":"in","type":"currency"},{"argument":"Result","direction":"both","type":"utf8"}]},{"method":"ToTextFunc","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Value","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"utf8"}]},{"method":"Swap","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"both","type":"double"},{"argument":"n2","direction":"both","type":"double"}]},{"method":"StackIntMultiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"integer"},{"argument":"n2","direction":"in","type":"integer"},{"argument":"n3","direction":"in","type":"integer"},{"argument":"n4","direction":"in","type":"integer"},{"argument":"n5","direction":"in","type":"integer"},{"argument":"n6","direction":"in","type":"integer"},{"argument":"n7","direction":"in","type":"integer"},{"argument":"n8","direction":"in","type":"integer"},{"argument":"n9","direction":"in","type":"integer"},{"argument":"n10","direction":"in","type":"integer"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"StackFloatMultiply","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"double"},{"argument":"n2","direction":"in","type":"double"},{"argument":"n3","direction":"in","type":"double"},{"argument":"n4","direction":"in","type":"double"},{"argument":"n5","direction":"in","type":"double"},{"argument":"n6","direction":"in","type":"double"},{"argument":"n7","direction":"in","type":"double"},{"argument":"n8","direction":"in","type":"double"},{"argument":"n9","direction":"in","type":"double"},{"argument":"n10","direction":"in","type":"double"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"SpecialCall","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Txt","direction":"in","type":"utf8"},{"argument":"Int","direction":"both","type":"integer"},{"argument":"Card","direction":"both","type":"cardinal"},{"argument":"field","direction":"in","type":"TSynTableFieldTypes"},{"argument":"fields","direction":"in","type":"TSynTableFieldTypes"},{"argument":"options","direction":"both","type":"TSynTableFieldOptions"},{"argument":"Result","direction":"out","type":"TSynTableFieldTypes"}]},{"method":"ComplexCall","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Ints","direction":"in","type":"TIntegerDynArray"},{"argument":"Strs1","direction":"in","type":"TRawUTF8DynArray"},{"argument":"Str2","direction":"both","type":"TWideStringDynArray"},{"argument":"Rec1","direction":"in","type":"TVirtualTableModuleProperties"},{"argument":"Rec2","direction":"both","type":"TSQLRestCacheEntryValue"},{"argument":"Float1","direction":"in","type":"double"},{"argument":"Float2","direction":"both","type":"double"},{"argument":"Result","direction":"out","type":"TSQLRestCacheEntryValue"}]},{"method":"Substract","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n1","direction":"in","type":"TComplexNumber"},{"argument":"n2","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"TComplexNumber"}]},{"method":"IsNull","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"boolean"}]},{"method":"TestBlob","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"n","direction":"in","type":"TComplexNumber"},{"argument":"Result","direction":"out","type":"TServiceCustomAnswer"}]},{"method":"TestVariants","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Text","direction":"in","type":"utf8"},{"argument":"V1","direction":"in","type":"variant"},{"argument":"V2","direction":"both","type":"variant"},{"argument":"Result","direction":"out","type":"variant"}]},{"method":"Collections","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Item","direction":"in","type":"TCollTest"},{"argument":"List","direction":"both","type":"TCollTestsI"},{"argument":"Copy","direction":"out","type":"TCollTestsI"}]},{"method":"GetCurrentThreadID","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"Result","direction":"out","type":"int64"}]},{"method":"GetCustomer","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"CustomerId","direction":"in","type":"integer"},{"argument":"CustomerData","direction":"out","type":"TCustomerData"},{"argument":"Result","direction":"out","type":"boolean"}]},{"method":"FillPeople","arguments":[{"argument":"Self","direction":"in","type":"self"},{"argument":"People","direction":"both","type":"TSQLRecordPeople"}]}]}
Original interface definition :
/// returns the thread ID running the method on server side
function GetCurrentThreadID: TThreadID;
which on linux64 produce : {"argument":"Result","direction":"out","type":"int64"}
but on win32 produce: {"argument":"Result","direction":"out","type":"cardinal"}
also on win64 produce : {"argument":"Result","direction":"out","type":"int64"}
So, strictly speaking, it's not a bug ,just an implemention flaw.
Thank you, ab, for your so quick replying .
Tested with :
FPC svn 52403 on Windows
target windows ,return A9C202E36B8F93AB
crosscompile to linux64 then running on ubuntu, return 0063F5DAFC95F854
FPC svn 55405 on Ubuntu
retrun 0063F5DAFC95F854
Same service code, both compiled with FPC, but service-contract differs . the client expected contract is the version on windows.
Linux-retruns {"result":["0063F5DAFC95F854"]}
windows-retruns {"result":["A9C202E36B8F93AB"]}