Help with implementation Publish/Subscribe with lots of objects

VojkoCendak · 2012-09-09 21:13:25

Hi, (Delphi 2009ent)
We have a several TCollections of some Objects, which mainly are small in size like:
TOPCtag=Class
published
property Value:variant;
property TagName:String;
property Quality:WORD;
property TimeStamp:TDateTime;
....
end;

So clients subscribe to objects dynamicaly (not all) in groups, like
Client.Connect(['Tag1','Tag2',...]) and then server notifies Clients on DataChange with the whole Goup.

So we had problems with RTC, when we polled tags in goups around 2000 tags the CPU got too high almost 100%.

The scenaio has to be lightweight and responsive ideally . It does good job with COM as OPC server but only locally,
DCOM ... (I don't wanna go there ).

What would be the best implemetation for this scenario? Any suggestions appretiated.

ab · 2012-09-10 05:25:04

From my experiment, DCOM is very "heavy" for callbacks - you are right not going into this direction.

I suspect in your case a good caching on the server side (using e.g. a dedicated TSQLRestStaticInMemory or an external SQLite3 ':memory:' kind of table) + periodic polling against the TimeStamp won't put the mORMot server down.

We are working on implementing a lightweight, but generic, Publish/Subscribe pattern for mORMot.
It will be integrated to the framework core, so will be optimized for any communication protocol used, and will use interface definitions for publish/subscribe mechanism.
See http://blog.synopse.info/post/2012/09/0 … laboration

VojkoCendak · 2012-09-10 13:16:41

=> DCOM - maintainance nightmare and what happens when network disrrupts (wait > 3minutes )

=> cache definetely

=> With publish subscribe, what will be impact on server and transport if
server would invoke each item separately, say 2000 times instead of invoking whole group :
1. server.Clients.InvokeClientCallback(MyTag1) 2000 times in a row
This is easier to implement, only I don't know what impact will be on the server and CPU.

2. server.Clients.InvokeClientCallback(Client[].GroupObjectWithTags[]) once a second or so ...

=> With in memory DataSet, how fast is Query on tags with 5000 rows for several hundred items
with index on TagName (string) ?
Or is better just to retreive whole table ?

Another question: What is the fastest way to implement TCollection or TList and to update it and then serialise it?

ab · 2012-09-10 13:30:28

To reduce communication bandwidth, you'll have to group all pending messages within one response.

With a TSQLRestServerStaticInMemory internal instance, lookup by unique name property will be very fast, thanks to the newly introduced hash-based search.
Around 70,000 requests per second for 5000 rows - see TObjectList / By Name results in http://blog.synopse.info/post/2012/08/0 … properties

A TCollection can be serialized with the ObjectToJson() function.

If you want CRUD, cache and SQL process, you should better use the ORM model, and a TSQLRestServerStaticInMemory instance - it will use internally a TObjectList for storage, then serialize it as JSON or binary.

mORMot Open Source

#1 2012-09-09 21:13:25

Help with implementation Publish/Subscribe with lots of objects

#2 2012-09-10 05:25:04

Re: Help with implementation Publish/Subscribe with lots of objects

#3 2012-09-10 13:16:41

Re: Help with implementation Publish/Subscribe with lots of objects

#4 2012-09-10 13:30:28

Re: Help with implementation Publish/Subscribe with lots of objects

Board footer