Client-Server remote access to any method (DataSnap-like)

ab · 2010-07-18 15:34:43

You certainly knows about the new DataSnap Client-Server features, based on JSON, introduced in Delphi 2010.
http://docwiki.embarcadero.com/RADStudi … plications

We added such communication in our SQLite3 Framework, in a KISS (i.e. simple) way: no expert, no new unit or new class. Just add a published method Server-side, then use easy functions about JSON or URL-parameters to get the request encoded and decoded as expected, on Client-side.

We'll implement the same example as in the official Embarcadero docwiki page above. Add two numbers. Very useful service, isn't it?

We'll first code the Server-side:

First the declaration of the class:

  TSQLRestServerTest = class(TSQLRestServerDB)
  published
    function Sum(aRecord: TSQLRecord; aParameters: PUTF8Char;
      const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
  end;

This method name will be used for the URL encoding, and will be called here with ModelRoot/Sum URL.

This method, like all Server-side methods, MUST have all these parameters, and follow the TSQLRestServerCallBack prototype:

type
  TSQLRestServerCallBack = function(aRecord: TSQLRecord;
      aParameters: PUTF8Char; const aSentData: RawUTF8;
      var aResp, aHead: RawUTF8): Integer of object;

Then we implement this method:

function TSQLRestServerTest.Sum(aRecord: TSQLRecord; aParameters: PUTF8Char;
  const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
var a,b: Extended;
begin
  if not UrlDecodeNeedParameters(aParameters,'A,B') then
  begin
    result := 404; // invalid Request
    exit;
  end;
  while aParameters<>nil do
  begin
    UrlDecodeExtended(aParameters,'A=',a);
    UrlDecodeExtended(aParameters,'B=',b,@aParameters);
  end;
  aResp := JSONEncodeResult([a+b]);
  // same as : aResp := JSONEncode(['result',a+b],TempMemoryStream);
  result := 200; // success
end;

Not difficult to follow, isn't?

OK, now the client-side:

function Sum(aClient: TSQLRestClientURI; a, b: double): double;
var err: integer;
begin
  val(aClient.CallBackGetResult('sum',['a',a,'b',b]),Result,err);
end;

And... that's all!

You have to create the server instance, and the corresponding TSQLRestClientURI, with the same database model, just as usual...

Note that this Client-Server protocol uses JSON here, but you can serve any kind of data, binary, HTML, whatever...
The usual protocols of our framework can be used: HTTP/1.1, Named Pipe, Windows GDI messages, direct in-memory/in-process access.

Just to be noticed that the data transmitted is a valid JSON content, like this one:

{ "result":3.141592653}

So you can consume these services, implemented Server-Side in fast Delphi code, with any AJAX application Client-Side.

Of course, these services can be related to any table/class of our ORM framework, so you would be able to create easily any RESTful compatible requests on URL like ModelRoot/TableName/ID/MethodName. For example, here we return a BLOB field content as hexa:

function TSQLRestServerTest.DataAsHex(aRecord: TSQLRecordPeople; aParameters: PUTF8Char; 
  const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
var aData: TSQLRawBlob;
begin
  result := 404; // invalid Request
  if (self=nil) or (aRecord=nil) or not aRecord.InheritsFrom(TSQLRecord) or
    (aRecord.ID<0) then
    exit; // we need a valid record and its ID
  if not RetrieveBlob(TSQLRecordPeople,aRecord.ID,'Data',aData) then
    exit; // impossible to retrieve the Data BLOB field
  aResp := JSONEncodeResult([SynCommons.BinToHex(aData)]);
  // idem: aResp := JSONEncode(['result',BinToHex(aRecord.fData)],TempMemoryStream);
  result := 200; // success
end;

Full source code is available in our Source Code Repository.
It should work from Delphi 7 to Delphi 2010.

ab · 2010-07-18 16:58:46

I've added a Client/Server sample to the source code repository (in "SQLite3\Samples\06 - Remote JSON REST Service" subfolder).

You can download the compiled version directly from http://synopse.info/files/samples/JSONService.zip

Both Server and Client executables are only 70 KB big, thanks to our LVCL units...

When you'll compile the sample Server code, you'll have warning about abstract classes, because TSQLRestServer doesn't implement any database engine, but it doesn't matter here, since we don't test the ORM aspect of the framework, but the remote Service provider.

longge007 · 2010-07-23 08:01:08

thanks your hard work,it's successful

Last edited by longge007 (2010-07-23 08:02:14)

ab · 2010-08-27 08:40:45

You have new JSONEncodeArray procedures in the SynCommons unit, to create JSON array content from supplied Delphi arrays (handle RawUTF8 text, double or integer arrays).

It can be used server-side, in some cases, when the data to be transmitted is an array.

Available in SQLite3 Framework updated to 1.9.1.

ab · 2010-08-27 08:43:36

Perhaps it's worth saying again, as stated above, that the server-side methods must be declared as published.

type
  TSQLRestServerWave=class(TSQLRestServerDB)
  published
     function DataAsHex(aRecord: TSQLSampleRecord; aParameters: PUTF8Char;
      const aSentData: RawUTF8; var aResp, aHead: RawUTF8): Integer;
  end;

Otherwize, the RTTI is not generated, and this method is never called by the server!

See http://synopse.info/forum/viewtopic.php?pid=479#p479

ab · 2011-07-12 13:07:15

And... that the implementation must be thread-safe!

I've updated the documentation to explicitly states this requirement:

An important point is to remember that the implementation of the callback method must be thread-safe. In fact, the TSQLRestServer. URI method expects such callbacks to handle the thread-safety on their side. It's perhaps some more work to handle a critical section in the implementation, but, in practice, it's the best way to achieve performance and scalability: the resource locking can be made at the tiniest code level.

ab · 2011-08-01 06:41:03

Per-query authentication has been added to our Client-Server implementation.
See http://blog.synopse.info/post/2011/05/2 … entication

The original DataSnap was found not to be secured. See http://www.sandon.it/?q=node/52
And lack of such authentication http://www.sandon.it/?q=node/57 even if latest XE version did introduce HTTPS.

(troll on)That's why sometimes an OpenSource project could have more features than a "paid" project. And you won't have to pay a new version of the IDE to have an upgrade of the library. (troll off)

ab · 2011-11-24 16:11:16

Note that the prototype of the callback methods has been modified, to supply an additional aSession: cardinal parameter: this is a CODE BREAK change and you shall refresh ALL your server-side code to match the new signature.

See http://synopse.info/forum/viewtopic.php?id=520 for details.

(to be included in revision 1.16)

profh · 2012-02-10 07:23:49

how to implement it using HTTP ,without named pipe?

ab · 2012-02-10 07:53:13

Please download the documentation available from http://synopse.info/fossil/wiki?name=SQLite3+Framework
You'll find in the SAD document all explanations about using HTTP protocol.
You just have to use the right classes.

See also the http://synopse.info/fossil/dir?name=SQL … ent-Server sample.

Extracted from this sample code:

// on the client:
  Form1.Database := TSQLite3HttpClient.Create(Server,'8080',Form1.Model);
  TSQLite3HttpClient(Form1.Database).SetUser('User','synopse');
// on the server:
  Model := CreateSampleModel;
  DB := TSQLRestServerDB.Create(Model,ChangeFileExt(paramstr(0),'.db3'),true);
  DB.CreateMissingTables(0);
  Server := TSQLite3HttpServer.Create('8080',[DB]);

That's all.

profh · 2012-02-10 10:00:05

thanks.

sample04 and sample06 are all fine.
i hope sample6 with HTTP but no named pipe,not just sample4.
it is better if samples9 , sample6 and sample4 are all built into one, and the server side uses just one ip address and one port.

ab · 2012-02-10 10:11:10

It is left to you as an exercise.

profh · 2012-02-10 10:18:06

thanks again, and i will finish it soon.

Last edited by profh (2012-02-10 10:19:48)

profh · 2012-02-16 17:38:38

excise is ok,samples9 , sample6 and sample4 are all built into one, and the server side uses just one ip address and one port.
but the memory keeps increasing...

for samples4,add a timer event with the following script:

procedure TForm1.Timer1Timer(Sender: TObject);
begin
FindButton.Click;
end;

add a big text record and then fire the Timer1,the Project04Server.exe memory keeps increasing...

ab · 2012-02-16 18:12:50

I think your code is leaking memory.

6,000,000 regression tests are running every time I commit the source code, and with no memory leak detected (with FastMM4).

Use FastMM4 memory leak features to guess where the leak comes from in your code.

profh · 2012-02-17 01:29:04

i just add a timer1 to the Project04Client of "04 - HTTP Client-Server" and the following script:

procedure TForm1.Timer1Timer(Sender: TObject);
begin
FindButton.Click;
end;

ab · 2012-02-17 08:05:22

Use FastMM4 memory leak features to guess where the leak comes from in your code.

Leander007 · 2012-02-17 12:30:33

ab wrote:

I think your code is leaking memory.
6,000,000 regression tests are running every time I commit the source code, and with no memory leak detected (with FastMM4).
Use FastMM4 memory leak features to guess where the leak comes from in your code.

Hm, 6 million tests, this must be wrongly counted, because line count of all your .pas sources is less then 400000 .

For memory leakage from Delphi 2006 up next simple setting could be used as first line in dpr file:

begin
ReportMemoryLeaksOnShutdown := True;
(...)
end.

This is usually enough, at least for me.

ab · 2012-02-17 15:04:43

Leander007 wrote:

Hm, 6 million tests, this must be wrongly counted, because line count of all your .pas sources is less then 400000 .

Perhaps you are making a confusion between test coverage and test counts.

I just run the regression tests:

   Synopse mORMot Framework Automated tests
  ------------------------------------------


1. Synopse libraries

 1.1. Low level common:
  - System copy record: 22 assertions passed  189us
  - TDynArray: 519,427 assertions passed  212.30ms
  - TDynArrayHashed: 1,200,023 assertions passed  181.12ms
  - Fast string compare: 7 assertions passed  130us
  - IdemPropName: 10 assertions passed  151us
  - Url encoding: 105 assertions passed  1.24ms
  - IsMatch: 599 assertions passed  296us
  - Soundex: 29 assertions passed  128us
  - Numerical conversions: 777,968 assertions passed  285.07ms
  - Curr64: 20,031 assertions passed  2.57ms
  - CamelCase: 5 assertions passed  138us
  - Bits: 4,614 assertions passed  163us
  - Ini files: 7,004 assertions passed  35.57ms
  - Unicode - Utf8: 61,082 assertions passed  1.30s
  - Iso8601 date and time: 24,000 assertions passed  6.25ms
  - Url decoding: 1,100 assertions passed  397us
  - TSynTable: 873 assertions passed  5.00ms
  - TSynCache: 404 assertions passed  1.17ms
  - TSynFilter: 1,005 assertions passed  3.21ms
  - TSynValidate: 677 assertions passed  961us
  - TSynLogFile: 20 assertions passed  774us
  Total failed: 0 / 2,619,005  - Low level common PASSED  2.04s

 1.2. Low level types:
  - RTTI: 34 assertions passed  211us
  - Url encoding: 200 assertions passed  917us
  - Encode decode JSON: 126,962 assertions passed  110.97ms
  Total failed: 0 / 127,196  - Low level types PASSED  117.72ms

 1.3. Big table:
  - TSynBigTable: 19,265 assertions passed  175.40ms
  - TSynBigTableString: 16,171 assertions passed  37.41ms
  - TSynBigTableMetaData: 384,060 assertions passed  450.10ms
  - TSynBigTableRecord: 452,185 assertions passed  752.93ms
  Total failed: 0 / 871,681  - Big table PASSED  1.42s

 1.4. Cryptographic routines:
  - Adler32: 1 assertion passed  1.17ms
  - MD5: 1 assertion passed  823us
  - SHA1: 5 assertions passed  805us
  - SHA256: 5 assertions passed  803us
  - AES256: 8,292 assertions passed  319.58ms
  - Base64: 11,994 assertions passed  108.23ms
  Total failed: 0 / 20,298  - Cryptographic routines PASSED  440.19ms

 1.5. Compression:
  - In memory compression: 12 assertions passed  226.21ms
  - Gzip format: 19 assertions passed  413.91ms
  - Zip format: 36 assertions passed  741.05ms
  - SynLZO: 3,006 assertions passed  96.13ms
  - SynLZ: 13,016 assertions passed  312.48ms
  Total failed: 0 / 16,089  - Compression PASSED  1.79s

 1.6. Synopse PDF:
  - TPdfDocument: 4 assertions passed  8.80ms
  - TPdfDocumentGDI: 6 assertions passed  26.66ms
  Total failed: 0 / 10  - Synopse PDF PASSED  38.23ms


2. mORMot

 2.1. Basic classes:
  - TSQLRecord: 47 assertions passed  658us
  - TSQLRecordSigned: 200 assertions passed  5.71ms
  - TSQLModel: 3 assertions passed  1.34ms
  Total failed: 0 / 250  - Basic classes PASSED  11.84ms

 2.2. File based:
  - Database direct access: 10,124 assertions passed  1.58s
  - Virtual table direct access: 12 assertions passed  156.08ms
  - TSQLTableJSON: 19,030 assertions passed  370.93ms
  - TSQLRestClientDB: 599,029 assertions passed  9.16s
  Total failed: 0 / 628,195  - File based PASSED  11.28s

 2.3. File based WAL:
  - Database direct access: 10,124 assertions passed  1.27s
  - Virtual table direct access: 12 assertions passed  73.80ms
  - TSQLTableJSON: 19,030 assertions passed  220.65ms
  - TSQLRestClientDB: 599,029 assertions passed  6.16s
  Total failed: 0 / 628,195  - File based WAL PASSED  7.73s

 2.4. Memory based:
  - Database direct access: 10,123 assertions passed  375.96ms
  - Virtual table direct access: 12 assertions passed  2.94ms
  - TSQLTableJSON: 19,030 assertions passed  59.75ms
  - TSQLRestClientDB: 667,322 assertions passed  5.39s
  Total failed: 0 / 696,487  - Memory based PASSED  5.83s

 2.5. Client server access:
  - TSQLite3HttpServer: 2 assertions passed  473.79ms
     using THttpApiServer
  - TSQLite3HttpClient: 3 assertions passed  285.59ms
  - Http client keep alive: 3,008 assertions passed  1.56s
     first in 105.72ms, done in 1.45s i.e. 687/s, aver. 1.45ms, 3.2 MB/s
  - Http client multi connect: 3,008 assertions passed  1.51s
     first in 63.51ms, done in 1.44s i.e. 690/s, aver. 1.44ms, 3.2 MB/s
  - Named pipe access: 3,010 assertions passed  753.62ms
     first in 187.50ms, done in 181.87ms i.e. 5498/s, aver. 181us, 25.7 MB/s
  - Local window messages: 3,009 assertions passed  138.25ms
     first in 51.14ms, done in 82.21ms i.e. 12162/s, aver. 82us, 56.8 MB/s
  - Direct in process access: 3,008 assertions passed  110.65ms
     first in 42.02ms, done in 68.32ms i.e. 14635/s, aver. 68us, 68.4 MB/s
  Total failed: 0 / 15,048  - Client server access PASSED  4.84s

 2.6. Service oriented architecture:
  - Service initialization: 92 assertions passed  2.77ms
  Total failed: 0 / 92  - Service oriented architecture PASSED  3.91ms

 2.7. External database:
  - TSQLRecordExternal: 1 assertion passed  729us
  - Crypted database: 253,272 assertions passed  1.05s
  - External via REST: 243,326 assertions passed  1.86s
  - External via virtual table: 243,326 assertions passed  2.58s
  Total failed: 0 / 739,925  - External database PASSED  5.51s


Synopse framework used: 1.16
SQlite3 engine used: 3.7.10
Generated with: Delphi 7 compiler

Time elapsed for all tests: 41.13s
Tests performed at 2/17/2012 3:58:08 PM

Total assertions failed for all test suits:  0 / 6,362,471

 All tests passed successfully.
Done - Press ENTER to Exit

Exact count is 6,362,471 calls to the TSynTestCase.Check() method during the regression tests.

Several tests are made over random content, and perform a lot of checks at high speed.
For instance, RawUTF8, numerical conversions, cypher or compression are performed over a huge number of different text.

See http://synopse.info/fossil/finfo?name=S … stSQL3.dpr for the supplied regression tests.

Leander007 · 2012-02-18 10:52:33

My second thought was that this number is the count of calls and even in that case 6 million is huge number comparing to "size" of core source code.
You have done really excellent work .

mORMot Open Source

#1 2010-07-18 15:34:43

Client-Server remote access to any method (DataSnap-like)

#2 2010-07-18 16:58:46

Re: Client-Server remote access to any method (DataSnap-like)

#3 2010-07-23 08:01:08

Re: Client-Server remote access to any method (DataSnap-like)

#4 2010-08-27 08:40:45

Re: Client-Server remote access to any method (DataSnap-like)

#5 2010-08-27 08:43:36

Re: Client-Server remote access to any method (DataSnap-like)

#6 2011-07-12 13:07:15

Re: Client-Server remote access to any method (DataSnap-like)

#7 2011-08-01 06:41:03

Re: Client-Server remote access to any method (DataSnap-like)

#8 2011-11-24 16:11:16

Re: Client-Server remote access to any method (DataSnap-like)

#9 2012-02-10 07:23:49

Re: Client-Server remote access to any method (DataSnap-like)

#10 2012-02-10 07:53:13

Re: Client-Server remote access to any method (DataSnap-like)

#11 2012-02-10 10:00:05

Re: Client-Server remote access to any method (DataSnap-like)

#12 2012-02-10 10:11:10

Re: Client-Server remote access to any method (DataSnap-like)

#13 2012-02-10 10:18:06

Re: Client-Server remote access to any method (DataSnap-like)

#14 2012-02-16 17:38:38

Re: Client-Server remote access to any method (DataSnap-like)

#15 2012-02-16 18:12:50

Re: Client-Server remote access to any method (DataSnap-like)

#16 2012-02-17 01:29:04

Re: Client-Server remote access to any method (DataSnap-like)

#17 2012-02-17 08:05:22

Re: Client-Server remote access to any method (DataSnap-like)

#18 2012-02-17 12:30:33

Re: Client-Server remote access to any method (DataSnap-like)

#19 2012-02-17 15:04:43

Re: Client-Server remote access to any method (DataSnap-like)

#20 2012-02-18 10:52:33

Re: Client-Server remote access to any method (DataSnap-like)

Board footer