Getting Started Guide?

paulh · 2012-05-12 17:53:52

Hi, I'm an experienced Delphi developer used to writing client/server Delphi applications that run on LAN with an Oracle back-end.
I need to write an ultra-thin client app with a central server for thousands of concurrent users to access. The app is not very sophisticated - mainly data entry and viewing - typical CRUD application.
My natural inclination is to create the app using DevExpress or TMS components (I need to use a rich grid control with filtering, column resizing, reordering, filtering etc) and to transmit the data to/from the client using JSON to an ISAPI DLL I'd write in Delphi. The ISAPI DLL would implement the server-side code with the database access/persistence to Oracle. I already have the database - it is from a legacy app my app will replace but I want to keep the underlying database.
I came across your Marmot SDK when looking for JSON components. I have to deliver the application before the end of May so I don't have time to learn a difficult framework. I like the overview parts of the 720 page doc but I don't have time to go through the doc - I just need a "Getting Started", some good demos and a reference to the classes.
I can't see any obvious "Getting Started" guide or suite of sample apps - seems to be a "Main Demo" but that doesn't seem to be documented as part of a "Getting Started" style doc? If you have these then please point me to them but if not then don't worry I will reuse existing ISAPI code and add a JSON component.
Please advise.
Regards
Paul
pheffernan@gmail.com

paulh · 2012-05-12 20:51:38

I've now downloaded the framework and I can see there are actually 10 sample projects which hopefully will give me what I need. Sorry for the premature request for help!
Paul

ab · 2012-05-13 06:22:08

Yes, you are not forced to read all documentation!
And some mORMot users uses the framework not for the ORM part of it, just for the RESTful HTTP access, of thousands of concurrent users (kernel-mode http.sys component scales very well). See http://synopse.info/forum/viewtopic.php?pid=4194#p4194

In the SAD document, take a look at the first part, which presents how our ORM and SOA features work.
Then you have a "getting started" part, when detailing the "Main Demo", in the same document (at the 2/3).
See the table of contents.

For your purpose, I suspect you would need:
- To create a http.sys based mORMot server;
- Not use our ORM directly (if you have an existing DB layout), but use an SOA approach instead: so see the "interface-based services";
- Take a look at the SynDBOracle classes: they allow fast and light direct access to Oracle, from the server side.

I suspect that interface-based services is what you need.
Just define the classes and interfaces, then access from a thin client to those methods.
All JSON / HTTP / Security features will be handled by the framework.

So use the latest version from the source code repository - 1.16 at this time.
Not the latest stable release - 1.15, which lacks some of the needed features, and bug fixes.

mpv · 2012-05-13 13:11:01

If you need a thousands of concurrent users ISAPI is not a best choice. The best performance is to use http.sys + IOCP (via mORMot THttpApiServer) and SynDBOracle for database access. In this case you:
1) do not need IIS and oracle client - only WinXP SP2+ and oci.dll
2) can easy implement in-memory server cache, session information storage and other feature
3) get VERY good performance

paulh · 2012-06-05 15:50:38

Hi Arnaud, I'm trying to test the framework as you described just using interface-based services.
I looked at the sample code for project 14 but it is too simplistic just invoking an Add method returning an integer.
I want to test this passing objects in both directions. I don't need the ORM as you identified, so what sample code can I see to simply demonstrate passing an object from client to server and a call from client requesting objects to be sent back?
If I can quickly get a simple example that works I'd be delighted to use the framework.
Thanks in advance.
Paul

paulh · 2012-06-05 17:05:17

Arnaud, I have modified the sample project 14 to have another function that passes two subclasses of TPersistent named TPatient in order to try and have the server update the objects.
The client code is as follows (a small modification to the sample code to include call to my new function:

if Client.Services['Calculator'].Get(I) then
begin
lblResult.Caption := IntToStr(I.Add(a,b));
// My new code to get a test object and pass to server
aPatient := TPatient(TPatient.testObject);
bPatient := nil;
c := I.TP(aPatient, bPatient);
showMessage('Patient 1 name='+aPatient.FirstName);
if assigned(bPatient) then
showMessage('Patient 2 name='+bPatient.FirstName+' '+bPatient.Surname)
else
showMessage('Patient retrieval failed!');
end;

The server code is as follows:

function TServiceCalculator.TP(n1: TPatient; n2: TPatient): integer;
begin
n2 := TPatient(TPatient.testObject);
TPatient(n1).FirstName := 'Bert';
result := 1;
end;

Note that the server is trying to update a field of the first object and to assign the second object.

The code runs, the server TP method gets invoked but the objects are unchanged after the call in the client.

How do I get my client to call the server and update and return new objects? These objects are simple subclasses of TPersistent with published simple properties.

Thanks

Paul

ab · 2012-06-05 18:00:15

Use var and out for the directions.

paulh · 2012-06-05 18:55:23

Hi Arnaud, when I tried to use effectively:

function TServiceCalculator.TP(var n1: TPatient; var n2: TPatient): integer;

or

function TServiceCalculator.TP(out n1: TPatient; out n2: TPatient): integer;

I get the following error raised from your code:
Invalid "Calculator" interface call: returned object.

And besides, an object parameter is a pointer to a pointer so it should be superfluous to declare an object parameter as var as if you want to change the object instance you change what the outer pointer points to.

I'm running the latest version of the code like you told me to. Do you want me to email you a version of the sample 14 project that is broken so you can see this first hand? If you email me paul @ westsw.com I can email you back the simple code... This is a very basic and simple requirement so it must have been implemented? I can serialise the object as a string myself and pass it that way if needed but I'm sure that is unnecessary???

Paul

The code that raises the exception is:

function TInterfacedObjectFake.FakeCall(var aCall: TFakeCallStack): Int64;

    if ValueDirection in [smdVar,smdOut,smdResult] then begin
      V := Value[a];
      case ValueType of
      smvObject: begin
        R := JSONToObject(V^,R,valid);
        if not valid then
          RaiseError('returned object');
        IgnoreComma(R);
      end;

(AB note: I edited your question since we do not need the whole framework code here - just he name of the method is enough)

Last edited by ab (2012-06-06 05:32:03)

ab · 2012-06-06 07:04:43

About the var/out fact use with a class, you are right of course, when you deal internally with Delphi code.
But it was by design, to specify the service transmission needs.
When an interface is designed, the const / var / out kind of parameter is used to determine what are in / out / in-out parameters to transmit.
It is stated by the documentation - see http://blog.synopse.info/post/2012/03/0 … a-contract

They can be defined as const, var or out - in fact, const and var parameters values will be sent from the client to the server as JSON, and var and out parameters values will be returned as JSON from the server;

So it does mean that JSONToObject is unable to unserialize the transmitted JSON content.
Could you put a breakpoint in the R := JSONToObject(V^,R,valid); line and check before executing JSONToObject what is the content of R - i.e. the JSON content?

How is your TPatient class defined?
It should be either a TSQLRecord or TPersistent sub class. Or any other class, if you supply a custom serializer - see http://blog.synopse.info/post/2012/04/1 … -any-class

Use of classes serialization is tested in SynSelfTests unit via the following interface:

  /// a test interface, used by TTestServiceOrientedArchitecture
  // - to test remote service calls with objects as parameters (its published
  // properties will be serialized as standard JSON objects)
  // - since it inherits from ICalculator interface, it will also test
  // the proper interface inheritance handling (i.e. it will test that
  // ICalculator methods are also available)
  IComplexCalculator = interface(ICalculator)
    ['{8D0F3839-056B-4488-A616-986CF8D4DEB7}']
    /// purpose of this method is to substract two complex numbers
    // - using class instances as parameters
    procedure Substract(n1,n2: TComplexNumber; out Result: TComplexNumber);
    /// purpose of this method is to check for boolean handling
    function IsNull(n: TComplexNumber): boolean;
    /// this will test the BLOB kind of remote answer
    function TestBlob(n: TComplexNumber): TServiceCustomAnswer;
  end;

The result is defined as out Result: TComplexNumber, and it works as expected.

paulh · 2012-06-06 10:26:07

Hi Arnaud, I can send you the code - it is a tiny extension to your Sample 14 projects and would make a good example for your users if you make it work. Please email me an address I can send it to and you can test it yourself - will be quicker than reading and responding to all these comments ;-)

My TPatient is a subclass of TPersistent - just publishing a couple of RawUTF8 fields for the patient's name.

If I remove the var/out parameters your code works in that the object gets encoded and transported from the client to the server. When I add the var option it parses the parameters and correctly as per the code rejects it because there is an object parameter with an "out" label. That is what you have coded it to do it seems.

I can manually serialise the object using ObjectToJSON at client and recreate at Server with JSONToObject, make the changes and then save back using ObjectToJSON - however I'm sure that is what you had intended to implement and would be a lot cleaner than me adding additional code to do this...

Please email me and I'll send you a zip of the sample projects.

Regards

Paul

ab · 2012-06-06 13:25:14

Note that the output TPersistent parameter (e.g. either declared as VAR or OUT), SHALL be defined before call.

I guess you set the "var n2: TPatient" parameter to NIL, which is not to be made.
Instantiate an instance before call, then retrieve the content after call.
In this case, the interface won't complain about the returned content (which is probably NULL).

ab · 2012-06-06 18:34:53

I just opened your sample code.
As I just suppose, you are passing NIL to bParent, which is the cause of the issue.
You should pass one instance, not a NIL pointer, in order to have properties to be retrieved from the remote server.
If you define the interface as

function TServiceCalculator.TP( n1: TPatient; out n2: TPatient; var ss: RawUTF8): integer;
begin
  n2.SetTestObject;
  TPatient(n1).FirstName := 'Bert';
  ss := ss+' changed';
  result := 1;
end;

In mORMot, class serialization are to be used as such: the instances shall be created before call, and the client library won't create any instance, just read or write properties using serialization.
Those classes are "value object", in the language of Domain Driven Design.
You may also use records (with custom serialization if necessary), since they may be easier to work with.

Some other (minor) remarks:
- It is not necessary to initalize all properties to '' after a Create: this is done by the compiler for you - all content is set to 0 or '';
- You should better use dedicated constructors instead of such functions, and - even better have a reusable SetTestObject at hand, e.g. to be used in the case above;
- Always take a look at the existing test code - e.g. IComplexCalculator test would have helped you;
- We may change the memory management in the future, but I guess this is not the best option, since creating class instances (which sounds easy) could be a bit difficult to properly implement, if you deal with dependency-injection or similar features.

paulh · 2012-06-07 02:51:50

Hi Arnaud, okay, that is fine - thanks for the help - it is working now.

The other pattern that I have is that I want to have the client request the server for a collection of patients. Which is the best mormot calling structure to have the server return a collection/array/list of patient objects back?

for example:

procedure TServiceCalculator.GetMalePatients( forRegion: integer; out patientList: TList);

or perhaps:

function TServiceCalculator.GetMalePatients( forRegion: integer ): TList;

Paul

ab · 2012-06-07 08:32:08

Just use a TCollection to store a collection of TPersistent.
Or a dynamic array to store a collection of records (my favorite).

A TList is not handled, because it lacks the Add method necessary to handle item creation.
So you'll need to implement a TPatientList inheriting from TCollection: it will transmitted as a JSON array of objects.

procedure TServiceCalculator.GetMalePatients( forRegion: integer; out patientList: TPatientCollection);

In this case also, the patientList parameter shall be allocated before call: "out" keyword is here not to state that an object is to be allocated, but that some data is to be retrieved from the server.
And you'll need to use it as an additional parameter, not as a result function, since classes can't be a function result.

paulh · 2012-06-07 09:03:24

Awesome Arnaud - that will be great to be able to use the TPatientList subclass of TCollection.
I'm kind of OO old-fashioned in that I never use records and arrays - always classes as even if you think you won't add behaviour to your data record, usually at some point you do end up doing so and nothing beats a class for encapsulation.

Well, I think I'm all set now to finish the implementation with mORMot's interface-based services. I do hope for a later project I can expand to learn the ORM implementation too.

Your documentation is extensive and well written. However, some constructive criticism is that it is written like a research paper that explains much more the theory than the practice. It is great to have the theory but in order to make a decision whether to use some technology or not, one needs a practical "Getting Started" guide that shows the user how to get up and running with the technology (the how) rather than the why. I'm glad that I can use this technology for my project but if I'm to learn the ORM side I think it will take a lot of trial and error.

Thanks for the very proactive and comprehensive support as well - it is reassuring to know you are close at hand and your English is so good!

ab · 2012-06-07 09:23:01

You are right, documentation was written from theory (some part is even published as blog articles of general high-level points) up to low level stuff. It uses sometimes regression tests as sample code, which is good for use coverage, but far from perfect when compared to "real world" use. And documentation is also some kind of reference, i.e. it tries to covers all features available... sometimes letting it difficult to guess how to use it in your code!

The "Main demo" part (in the SAD document) is pretty practical, but it covers only the ORM and UI part of mORMot, not the newest features, like interface based services.

I do not think my English is so good... documentation is full of mistakes, and I'm quite sure any native will smile (or even LOL) more than once.
But I'm pretty convinced that no documentation is worse than not perfect documentation. A lot of (Delphi) project - not only OpenSource - do lack of proper documentation.
For a framework like mORMot, it would be a disaster if no documentation would be available.
But since some users do not know much about some technique used in the framework (like ORM, SOA, MVC, REST, sharding, DDD or even n-tier), I thought it would be a good idea to put high-stuff within.
Most of the time, high-level articles were written for me, just to re-phrase and ensure that I've well used properly the corresponding patterns within mORMot.

I'd like to create some new demo project including most of the new features of mORMot, a new "Main demo", and try to use it as "Getting Started" guide.

Well I'm very happy that mORMot meets your requirements of service implementation.
If you have any idea or need, you can ask them here: some features comes directly from customer requests.
And you are free to contribute to the projects, and also make a visit to the forum from time to time to help others - I'm very happy that I'm not the only one doing support here!

paulh · 2012-06-07 14:42:45

BTW, have you used live bindings for linking user interface controls to objects?
I'm using bidirectional bindings with a few helper methods that allow me to map object fields to controls like the following and hey presto I have a UI and object kept in sync:

procedure TfrmNewReferral.doBindings;
begin
// Create a binding expression.
bindTogether(self, bsPatient, 'FirstName', aePatientFirstName);
bindTogether(self, bsPatient, 'Surname', aePatientSurname);
bindTogetherPopulateCombo(self, bsPatient, 'Title', cbPatientTitle, TPersonTitle);
bindTogether(self, bsPatient, 'Gender', cbPatientGender);
cbPatientGender.Items.Text := TGender.combolist;
bindTogether(self, bsPatient, 'Marital', cbPatientMaritalStatus);
cbPatientMaritalStatus.Items.Text := TMaritalStatus.combolist;
bindTogether(self, bsPatient, 'Dob', dpPatientDOB, 'Date');
bindTogether(self, bsPatient, 'HomePhone', mePatientHomePhone);
bindTogether(self, bsPatient, 'MobilePhone', mePatientMobilePhone);
bindTogether(self, bsPatient, 'Email', aePatientEmail);
bindTogether(self, bsPatient, 'NhsNumber', mePatientNHS);
bindTogether(self, bsPatient, 'Postcode', aePatientPostcode);
bindTogether(self, bsPatient, 'Address1', aePatientAddress1);
bindTogether(self, bsPatient, 'Address2', aePatientAddress2);
bindTogether(self, bsPatient, 'Town', aePatientTown);
bindTogetherPopulateCombo(self, bsPatient, 'County', cbPatientCounty, TCounty);
bindTogetherPopulateCombo(self, bsPatient, 'Country', cbPatientCountry, TCountry);
end;

ab · 2012-06-07 15:27:56

I had rather use auto-generated UI directly from the class layout.

THere are dedicated units for UI in our framework.

Live binding sounds a bit complicated to me.

paulh · 2012-06-07 16:59:33

Hi Arnaud, other than for a test app I could never think of a time when an auto-generated UI could be preferable to one that is created specifically for the application. If you looked at the richness of the applications I'm creating you'd see that it would be a waste of time to try and create an auto-generated UI.

Live bindings is SO simple. You design your form as you want things to be, you associate the controls with the fields in your class and then you can have your app retrieve an object and the fields immediately are displayed in the UI. If they select a different value in a combo and select the "Ok" button to apply the UI to the object there is a single call to do that. You then validate the object (not the UI) but flag errors back into the UI.

There is now never a need to use data aware controls now that live bindings exists - any control becomes object field aware and you never have to worry about keeping fields and components in sync - you can focus on getting a really nice UI and rich business logic.

paulh · 2012-06-07 17:04:49

Hi Arnaud, I need to use HTTPS rather than HTTP for the communication between client and server - do you have a reference of how I configure that?
Paul

ab · 2012-06-08 09:15:08

You'll need to set the HTTPS flag to true on both client and server side, AND deploy the HTTPS certificate on both sides (i.e. both server and client).

See for instance http://msdn.microsoft.com/en-us/library/ms733768.aspx and http://msdn.microsoft.com/en-us/library/ms733791.aspx
And http://cgeers.com/2009/08/07/wcf-over-https/
(WCF relies on the same http.sys kernel-mode API than mORMot)

It is not so easy to do - but it is Microsoft's fault!
Security is not fair, from the developer point of view.

I'd probably add an encryption feature within the mORMot communication layer. There is already SynLZ or Deflate/GZip compression built-in for HTTP, and it would be very easy to add such an encryption.
That is, it won't use HTTPS, just plain HTTP (nothing to configure), but all content will be encrypted during communication.
It will be less secure than a full HTTPS, but it would be very easy to configure, fast, and transparent.

If you have steps to follow, please post them here so that I could update the documentation, to help other users.

ab · 2012-06-08 15:28:59

I've added one entry to the official RoadMap:

Support content encryption at the mORMot level, allowing safe and fast transfer without the speed overhead and unpleasant signature configuration on both sides.

See http://synopse.info/fossil/wiki?name=RoadMap

paulh · 2012-06-14 01:36:00

Arnaud, I have tried to get your suggestion working of using either a TCollection or an array of records and in both cases when the client calls the server passing either a collection or array I get a runtime error. I've emailed you a modification of your sample project 14 that demonstrates the issue.
What I'm trying to do is get the client to request from the server a set of objects/records. It will typically pass an empty container and needs the server to populate the container and pass the information back.
Can you please let me know what I need to do to make this work. Please email me back the modified project with code that you have made work as this is getting fairly time critical now for me to deliver a working solution.
Paul

ab · 2012-06-14 06:56:56

I took a look at the code.
You are passing a plain TCollection type.

type
  ICalculator = interface(IInvokable)
    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
    function Add(n1,n2: integer): integer;
    function TP(var n1: TPatient; out n2: TCollection; var ss: RawUTF8): integer;
  end;

In Delphi, this is not the way you do it: you have to inherit from TCollection, and provide the corresponding methods (e.g. Add) for the given class.
See for instance http://www.atug.com/andypatterns/collections.htm

Or, as usual, see how the unit tests are defined in SQLite3commons.pas:

type
  TCollTest = class(TCollectionItem)
  private
    FLength: Integer;
    FColor: Integer;
    FName: RawUTF8;
  published
    property Color: Integer read FColor write FColor;
    property Length: Integer read FLength write FLength;
    property Name: RawUTF8 read FName write FName;
  end;

  TCollTests = class(TCollection)
  private
    function GetCollItem(Index: Integer): TCollTest;
  public
    function Add: TCollTest;
    property Item[Index: Integer]: TCollTest read GetCollItem; default;
  end;

You you'll have to pass your specific TCollection type, as such:

type
  ICalculator = interface(IInvokable)
    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
    function Add(n1,n2: integer): integer;
    function TP(var n1: TPatient; out n2: TCollTests; var ss: RawUTF8): integer;
  end;

and it will work as expected.

paulh · 2012-06-14 13:55:45

Hi Arnaud, in Delphi you don't need to subclass from TCollection and implement Add and Item methods to work with collections - just passing the TCollectionItem subclass to the Create method of TCollection allows it to add items of the correct subclass when you invoke the Add method on TCollection.
I'm happy to do it your way if that is what is required to make it work with mormot but it is not the standard way. The article you reference subclasses TCollection (THairs) as a convenience so that the Add and Item methods return the correct subclass but they are NOT mandatory to allow TCollection to work and the code I sent you just using a plain TCollection is correctly instantiating the correct subclass when you invoke its Add method.
I shall use this now on the assumption that you've tested this and it works. However, I also got a pretty similar error when using your recommended approach with records and dynamic arrays - it might be useful for anyone who does prefer records/arrays to know how to make that work correctly?
Thanks again for the prompt responses - great support.
Paul

paulh · 2012-06-14 14:23:00

Arnaud, I have implemented the TCollection subclass as you requested and I am still getting exactly the same error as before.
Do you want me to send you my project code?
Or perhaps you could send me your amended sample project code you tested as working passing these TCollections?
Thankfully the ObjectToJSON and JSONToObject do work with TCollection [without having to subclass ;-) ] so I can serialise and deserialise myself between calls so I'm not stuck but given that you feel this should work it would be nice to use the higher level methods.
Paul

ab · 2012-06-14 14:30:18

You are right about TCollection.

The problem relies on the Server side.
On the Server side, the n2 parameter is to be created.
It will use the DEFAULT Create constructor, without any parameter.
If you define the interface parameter just as a TCollection, the server side has no mean of specifying the corresponding constructor.
That's why a dedicated TCollection sub type is needed.
I can't see any potential workaround here.

paulh · 2012-06-14 14:39:54

Hi, but I'm testing it with the n2 parameter having been created with the TCollTest and just trying to invoke the call from client to server causes the error even without trying to add items on the server side.
The same error occurs if I pass a dynamic array - even if I don't try and manipulate the array on the server side.
I can use the ObjectToJSON/JSONToObject but it can't be anything too tricky to resolve as all I'd imaging you're doing here is using those methods to serialise and deserialise the parameters before passing the data across the HTTP connection?
Anyway, I'm not stuck but let me know if you have a workaround for the issue I can use.
Thanks
Paul

ab · 2012-06-14 17:11:59

Only a dynamic array of records will work (a dynamic array of classes won't work, by design).

I've identified an implementation issue, thanks to your feedback.

Due to the current implementation pattern of the TCollection type in Delphi, it was not possible to implement directly this kind of parameter.

In fact, the TCollection constructor is defined as such:

constructor Create(ItemClass: TCollectionItemClass);

And, on the server side, we do not know which kind of TCollectionItemClass is to be passed. Therefore, the TServiceFactoryServer is unable to properly instantiate the object instances, supplying the expected item class.

So a dedicated TInterfacedCollection abstract type has been defined:

TInterfacedCollection = class(TCollection)
protected
function GetClass: TCollectionItemClass; virtual; abstract;
public
constructor Create; reintroduce; virtual;
end;

In order to use a collection of objects, you will have to define at least the abstract method, for instance:

TCollTests = class(TInterfacedCollection)
protected
function GetClass: TCollectionItemClass; override;
end;
function TCollTests.GetClass: TCollectionItemClass;
begin
result := TCollTest;
end;

Or, if you want a more complete / convenient implementation:

TCollTests = class(TInterfacedCollection)
private
function GetCollItem(Index: Integer): TCollTest;
protected
function GetClass: TCollectionItemClass; override;
public
function Add: TCollTest;
property Item[Index: Integer]: TCollTest read GetCollItem; default;
end;

Then you will be able to define a contract as such:

procedure Collections(Item: TCollTest; var List: TCollTests; out Copy: TCollTests);

A typical implementation of this contract may be:

procedure TServiceComplexCalculator.Collections(Item: TCollTest;
var List: TCollTests; out Copy: TCollTests);
begin
CopyObject(Item,List.Add);
CopyObject(List,Copy);
end;

That is, it will append the supplied Item object to the provided List content, then return a copy in the Copy content:
- Setting Item without var or out specification is doing the same as const: it will be serialized from client to server (and not back from server to client);
- Setting List as var parameter will let this collection to be serialized from client to server, and back from server to the client;
- Setting Copy as out parameter will let this collection to be serialized only from server to client.

I've also made an enhancement: CopyObject() procedure now handle TCollection kind of object not only as sub properties, but as main parameters.

Note that const / var / out kind of parameters are used at the contract level in order to specify the direction of serialization, and not as usual (i.e. to define if it is passed by value or by reference). All class parameters shall be instantiated before method call: you can not pass any object parameter as nil (nor use it in a function result): it will raise an error.

See http://synopse.info/fossil/info/9a0b706eb9

paulh · 2012-07-04 12:09:52

Hi Arnaud, in answer #21 you said:

ab wrote:

There is already SynLZ or Deflate/GZip compression built-in for HTTP, and it would be very easy to add such an encryption.

I'd like to use my own compression and encryption of string values that I transmit between client and server - how to I turn off and on the "SynLZ or Deflate/GZip compression" compression that you mention is built-in?

Thanks in advance.

Paul

ab · 2012-07-04 15:32:35

Take a look at unit SQLite3HttpServer and unit SQLite3HttpClient.

You have two defines:

{$define COMPRESSSYNLZ}
{ if defined, will use SynLZ for content compression
  - SynLZ is much faster than deflate/zip, so is preferred
  - can be set global for Client and Server applications
  - with SynLZ, the 440 KB JSON for TTestClientServerAccess._TSQLite3HttpClient
    is compressed into 106 KB with no speed penalty (it's even a bit faster)
    whereas deflate, even with its level set to 1 (fastest), is 25 % slower
  - is defined by default for a Delphi client }

{.$define COMPRESSDEFLATE}
{ if defined, will use deflate/zip for content compression
  - can be set global for Client and Server applications
  - SynLZ is faster but only known by Delphi clients: you can enable deflate
    when the server is connected an AJAX application (not defined by default)
  - if you define both COMPRESSSYNLZ and COMPRESSDEFLATE, the server will use
    SynLZ if available, and deflate if not called from a Delphi client }

It uses two compression/encryption functions, depending on the algorithm used.

Then the compression is enabled in constructor TSQLite3HttpServer.Create:

{$ifdef COMPRESSSYNLZ}
  fHttpServer.RegisterCompress(CompressSynLZ);
{$endif}
{$ifdef COMPRESSDEFLATE}
  fHttpServer.RegisterCompress(CompressDeflate);
{$endif}

And the same for the client side.

So for your purpose, you need to define a compression/uncompression function as defined by this prototype:

  /// event used to compress or uncompress some data during HTTP protocol
  // - should always return the protocol name for ACCEPT-ENCODING: header
  // e.g. 'gzip' or 'deflate' for standard HTTP format, but you can add
  // your own (like 'synlzo' or 'synlz')
  // - the data is compressed (if Compress=TRUE) or uncompressed (if
  // Compress=FALSE) in the Data variable (i.e. it is modified in-place)
  // - to be used with THttpSocket.RegisterCompress method
  // - type is a generic AnsiString, which should be in practice a
  // RawByteString or a TSockData 
  THttpSocketCompress = function(var Data: AnsiString; Compress: boolean): AnsiString;

paulh · 2012-07-04 19:09:43

Hi Arnaud, I have another issue.

When I call a function on the server and pass back a string, if the string is 2460657 characters it returns okay but as soon as I try and return a string with 2461055 characters I get an error raised:
'Error calling ERefServerInterface.getSiloData remote method'.

The function has following structure:
function getSiloData(const userId: RawUTF8; var inSiloS: RawUTF8): integer;

There must be some limitation on the inSiloS string length? I'm using it to encode my collection of objects using "ObjectToJSON" which is quite happy returning strings 20M characters. How do I overcome this issue to use the client/server calls to return longer strings?

Paul

ab · 2012-07-04 21:03:22

Did you try with direct connection, or only via Http?
Did you try disabling compression, or shutdown your antivirus?

If it works with direct connection, so the framework core is not involved (what I think).
Perhaps you reached some unexpected memory limit of the HTTP server, in the default configuration.

Could you use the debugger, and see where the error is triggered, i.e. when TServiceFactoryClient.CallClient is exited?
I suspect there is an exception within the server side process, or some parsing issue.

paulh · 2012-07-08 03:01:02

Hi Arnaud, I found that the problem with passing back the string was not due to the size of the string, it was the content of the string.
My function on the server side returns a string which is an encoding of a TCollection full of objects by calling "ObjectToJSON(silo)" where silo is the TCollection and the string returned by the ObjectToJSON is passed back to the client using a "var" parameter.

There is one object in the collection of 17,000 objects which causes the call from the client to the server to fail if that object is included in the collection that is encoded. If I exclude the object the call works.

If I do an ObjectToJSON on the object, it is the ReferralId property that causes the problem - they are binary GUIDS and this particular one has a double quote character in it which ObjectToJSON is escaping but yet it still manages to cause the problem:
{"ReferralId":"殯￶\"ㅉ壴⇮䚰","ReferralRef":"692.8476","ReferralDate":"2011-09-12T00:00:00","PatientRef":"8439.692","PatientName":"Paul,Jones","NHSNo":"123 238 6123","ReferredBy":"","RefPractice":"Vision Opticians","RefReason":"Glaucoma","ActionRqd":"Routine","PatAddress":"1, A Road, A Town, ","DOB":"1928-01-01T00:00:00","AreAttachments":false,"ObjectId":0}

I have been able to get around the issue by Base64 encoding the string on the server side and decoding it on the client side but you may want to look into this as I'm sure your code is probably encoding as well and other than this particular object the other 17,000 all work fine so it must be a small bug.

Thanks for the information about the compression and encryption.

Paul

ab · 2012-07-08 10:44:18

1. Why not just pass the "silo" variable as parameter to the interface, and not use manual serialization via ObjectToJSON/JSONToObject?
Manual serialization will reduce speed process and increase transmitted content size (a JSON string containing a serialized object has to be escaped).

2. How is your ReferralId property defined?
IMHO it should be better defined as RawByteString, so the serialization will automatically use Base64 encoding for this property content.
Or perhaps set the property as TGUID? I'm not sure if TGUID kind of record is handled as such (even at the interface level), but I could manually add it as special case to the JSON serialization process.

paulh · 2012-07-08 20:58:01

Hi Arnaud, I can't just pass the silo variable as a TCollection as you'd said above that I would need to subclass it and add methods for adding TCollectionItem subclasses to the collection. That's just too much work for all the types of containers I need to pass across and I don't really understand why Mormot requires this when the serialisation/deserialisation of TCollections works just fine.

I'd not come across RawByteString but when I tried to define an interface between the client and server with a RawByteString parameter it causes the server to crash immediately on startup. I don't have time to investigate why.

I'm finding performance to be acceptable so it is not a problem for me to base64 encode/decode it if it makes it work as it does but it would be great if you can find and fix the bug to avoid me having to!

Paul

ab · 2012-07-09 04:33:21

Without code to reproduce it won't be possible. It dépends on your code .

About TCollection I aready gave the reason.
Please read my answer above.

ab · 2013-01-18 16:37:01

You can now register any TCollection type for proper JSON serialization.
See http://synopse.info/fossil/info/730c07e487

And http://blog.synopse.info/post/2013/01/1 … ialization article.

mORMot Open Source

#1 2012-05-12 17:53:52

Getting Started Guide?

#2 2012-05-12 20:51:38

Re: Getting Started Guide?

#3 2012-05-13 06:22:08

Re: Getting Started Guide?

#4 2012-05-13 13:11:01

Re: Getting Started Guide?

#5 2012-06-05 15:50:38

Re: Getting Started Guide?

#6 2012-06-05 17:05:17

Re: Getting Started Guide?

#7 2012-06-05 18:00:15

Re: Getting Started Guide?

#8 2012-06-05 18:55:23

Re: Getting Started Guide?

#9 2012-06-06 07:04:43

Re: Getting Started Guide?

#10 2012-06-06 10:26:07

Re: Getting Started Guide?

#11 2012-06-06 13:25:14

Re: Getting Started Guide?

#12 2012-06-06 18:34:53

Re: Getting Started Guide?

#13 2012-06-07 02:51:50

Re: Getting Started Guide?

#14 2012-06-07 08:32:08

Re: Getting Started Guide?

#15 2012-06-07 09:03:24

Re: Getting Started Guide?

#16 2012-06-07 09:23:01

Re: Getting Started Guide?

#17 2012-06-07 14:42:45

Re: Getting Started Guide?

#18 2012-06-07 15:27:56

Re: Getting Started Guide?

#19 2012-06-07 16:59:33

Re: Getting Started Guide?

#20 2012-06-07 17:04:49

Re: Getting Started Guide?

#21 2012-06-08 09:15:08

Re: Getting Started Guide?

#22 2012-06-08 15:28:59

Re: Getting Started Guide?

#23 2012-06-14 01:36:00

Re: Getting Started Guide?

#24 2012-06-14 06:56:56

Re: Getting Started Guide?

#25 2012-06-14 13:55:45

Re: Getting Started Guide?

#26 2012-06-14 14:23:00

Re: Getting Started Guide?

#27 2012-06-14 14:30:18

Re: Getting Started Guide?

#28 2012-06-14 14:39:54

Re: Getting Started Guide?

#29 2012-06-14 17:11:59

Re: Getting Started Guide?

#30 2012-07-04 12:09:52

Re: Getting Started Guide?

#31 2012-07-04 15:32:35

Re: Getting Started Guide?

#32 2012-07-04 19:09:43

Re: Getting Started Guide?

#33 2012-07-04 21:03:22

Re: Getting Started Guide?

#34 2012-07-08 03:01:02

Re: Getting Started Guide?

#35 2012-07-08 10:44:18

Re: Getting Started Guide?

#36 2012-07-08 20:58:01

Re: Getting Started Guide?

#37 2012-07-09 04:33:21

Re: Getting Started Guide?

#38 2013-01-18 16:37:01

Re: Getting Started Guide?

Board footer