mORMot Open Source

Needed changes to run PerfTests at my environment are:

SynDBFireDAC:

498:              P.AsWideStrings[i] := UTF8ToString(tmp);
515:            P.AsWideString := UTF8ToString(VData);

P.S. It is working in PostgreSQL, but not in FireDAC's SQLite3 (maybe some firedac sqlite driver settings needed?). Same Name value error with sqlite3.

Building Win32

same problem:

Executing PerfTest.exe tuned for PostgresSQL gives:
'FireDAC PostgreSQL read failure: Value.FirstName=ValueFirstName[ i ] "Name 2"<>"Namé 2"'

In fact, same error with FireDAC's SQLite3:
'FireDAC SQlite3 read failure: Value.FirstName=ValueFirstName[ i ] "Name 2"<>"Namé 2"'

Delphi XE4, FireDAC 8.0.3.

Just changed some TADParam assignments in SynDBFireDAC.pas (in case of UNICODE defined)

P.AsString => P.AsWideString
P.AsStrings => P.AsWideStrings;

In case of .AsString assignment, FireDAC at some point issue VarAsType(<UTF8 encoded value>, varString) which seem converts the string in default code page, AFAIK.
After the changes UTF8 insertions seem correct, but other assert occur at PerfTest ValueCheck, but this may be due to postgresql specific field type/content handling.

I do not think this change is right, but just a quick fix in order to go forward.

Thanks a lot

Just starting to migrate a real-life server-side application using mORMot with FireDAC/PostgreSQL, as SQLite is becoming a bottleneck even in test environment.
This should be some kind of test.
Will let you know when problems faced.

About UTF8 initialization of new databases, it make sense, since it is highly recommended by postgresql community.
UTF8 client connection should be suitable for most of the cases.

Thanks again.

Is it possible, to use PostgreSql as database backend for mORMot REST server using FireDac?

It will be good to add RegExp functionality into SynDBExplorer in case of SQLite3 connection

Probably you wish to create and destroy TIniData at inside your REST server, which will be done once.
And use threadvar ServiceContext inside your interface service.

Or define any method to be called inside TServiceRemoteLiveServerStandard, do all "create try finaly destroy" operations.

Sample 14 - "14 - Interface based services" may help you.

That is great, ab.
Thanks a lot, definitely will help.

Thanks ab.

It worth everytime a text column is part of an index.
Maybe passing @Value[1], length(Value) is more correct.

If this column is involved within index or is part of not null constraint, the logic brokes.
Also, if you try to .Retrieve such a record passing "" (not null) property you will miss previously inserted same record, due to sqlite3 affinity.

Here is the scenario.

I have two threads.
1 thread: doing RestClient.Retrieve and if not found RestClient.Add for StaticInMemory
2 thread. doing RestClient.Delete

Retrieve() is done by searching for int64 stored false hashed property.
Delete() is done by sqlrecord id

when 2nd thread delete a record and 1st thread try to retrieve and than add same record values following stack trace happens after many iterations:

20130313 20083522 EXCOS 	EAccessViolation (C0000005) at 00174C79 mORMot.GetInt64Prop (10718)  stack trace 00009A3C System.@NewAnsiString 000058D6 System.@FreeMem 00184CF2 mORMot.TSQLRestClientURI.ExecuteList (20431) 0000A2F6 System.@LStrCat 0018524B mORMot.TSQLRestClientURI.InternalListJSON (20567) 001834BF mORMot.TSQLRest.Retrieve (19526) 00183687 mORMot.TSQLRest.Retrieve (19589)
...

20130313 20083736 EXC   	ESynException ("HashInit found dup") at 0015F278 SynCommons.TObjectHash.HashInit (21350)  stack trace 0000A448 System.@LStrCatN 00186D8A mORMot.TSQLRestClientURI.EngineAdd (21115) 0019105E mORMot.TSQLRestClient.Add (25307)
...

code line numbers may differ from official source codes in mormot units.

Tried to Lock/Unlock EngineDeleteWhere, but no success.

Yes, latest sources I am using.

Ok, it seems not well formatted JSON for JSONToNewObject, as it does not start with '{'.

Tried with JSONToObject, same valid False result.
TMyClass instance created well and added to MyClassList. But new instance created contains nil for MyContainedClassList field and processing json fails.

Workaround is to manually create MyContainedClassList field on AfterConstruction, so nasted call to JSONToObject will have valid ObjectInstance as parameter.

Thanks ab,

Like a lot slim read/write locks. From years I use them, introduced in Windows Vista and Server 2008.
Very fast multi read exclusive write. Only one drawback - one thread should not try to reacquire it.
Matter of code design, but think in web server worker threads it is not a big issue. Critical Sections are good, but read-locking a peace of data for all threads, like session data or user context is waste of resources, I think.

You mean usage of abstract class for WebSocket common API or transport abstraction?

I agree interfaces will be more suitable in both cases.
Tried to abstract the usage, further step can be to interface it.

ab, why you think "Socket" server will be more suitable? compatibility?

Do not know if here is the place to report, but know Eric will see it.

Eric, in order to compile the server (Delphi XE3) I have to correct following:

according MSDN,
at DSimpleDWScript.pas: procedure TSimpleDWScript.SetCPUAffinity(), variables procMask, systemMask used in calling GetProcessAffinityMask() should be PDWORD_PTR (in Delphi DWORD_PTR=>ULONG_PTR=>NativeUInt)
at dwsDirectoryNotifier.pas: procedure TdwsFileNotifier.Execute, variable completionKey used in calling GetQueuedCompletionStatus() should be PULONG_PTR (ULONG_PTR=>NativeUInt)

Great work!

One world: awesome!

Feel like new era of server side programming is coming ...
I think it is time SMS to make effort for client side mORMot compatibility.

It was not compatible with http.sys. Starting Win8+ it is.
http://nbevans.wordpress.com/2011/12/20 … ws-server/

Introducing HTTP_SEND_RESPONSE_FLAG_OPAQUE.

In Synopse.inc you have following:

{$RTTI EXPLICIT METHODS([]) PROPERTIES([]) FIELDS([])}

But for my purposes I need following directive:

{$RTTI INHERIT METHODS([vcPublic]) PROPERTIES([vcPublic]) FIELDS([vcPublic])}

If I define it in my class declaration unit seems to work so far.

It may worth to know, that your code change the default behavior of compilator regarding RTTI generation and if anyone starts using mORMot units in existing project, default RTTI behavior may be broken. Like my case

Anyway, sorry for bothering you, hope it may help others too.

Surely it will be shared, I would love to, just give me a few weeks to tune up my existing code within mORMot framework.
I am driven from my use cases, but I am sure they can suit most of developer needs.

There are two use cases I need to cover.
First is handling big amount of data, second is decoupling user interface from the data, even of very intensive insert/update/delete operation over the data. User interface may not react to every single data change. But finally, they will be in sync.

1.
Lets imagine we have a big amount of business data. "Big amount" is abstract for every user. Lets say we have a table, containing a few millions of records.
We expose this table to our customers. It can be read only to them, but not to the editors of the data. But every reader will need to see the changes made by the "editors" on demand.

First problem is, how to show millions of records to every reader?
Google approach is to show nothing at front page, than put a search, will see paginated match of query.
But in custom application, human mind will feel more comfortable if it see a virtual table (list) containing all the records, than filter the records by its criteria, than filter again, result may contain also a millions of records, but he will keep filtering till find what is searching for. And most important, the user will see all the result in one table, being able to scroll and preview all the result data, just scrolling the table, not paginating. This will not help much finding exact match, but it is an advantage from human philosophical point of view.
TVirtualTree is good example how we can show initially big amount of rows (records) and initialize the nodes on demand. Nodes initialized by request from user interface.
This data is most common to persist.

2.
Viewing "real-time" statistic data, which is changed very intensive. Very few rows, let say 50, fitting on one screen. Statistic information which may change hundreds of time per seconds. So much that it is not suitable for user-interface to react to every change.
This data is most common to not persist.

When showing data in virtual table view, we will need to couple the data "row" id and table row index. Just like you do with IDToIndex in TSQLRestServerStaticInMemory.
This is where I define three Virtual "Proxy" Tables. I use proxy word in context of proxy to memory, proxy to db, and proxy to cached db, just like TSQLRestServer do using TSQLRestCache.

1. Proxy to memory:
Little count of rows, which will not need to persists, can be updated very intensive.
2.Proxy to db:
Big amount of on-disk persistent data, which will be not so fast, update more rarely but will be safe.
3.
Proxy to cached db, mix of 1 and 2:
Big amount of on-disk persistent data, little set of which will be updated very intensive. Will persist, but in case of failure last updates may not have a chance to persist.

All three cases are implemented using VirtualTables.
You may say all of this is currently implemented in mORMot framework.

What I will add over current implementation is:
1. All virtual tables will expose virtual methods for functions like GetFieldVarData(), SetFieldVarData() or CopyObject(), so if one like to gain extra speed will implement them knowing what kind of TSQLRecord descendant is dealing with.
2. Will let client to subscribe to listen/notify channel over tables/records of interest. Currently and unfortunately not transaction aware. Will track all changes, consolidate them, stream to the client on request.

In a simple example it may look like this:
1. Client request to listen to TSQLBigData table of records.
2. Client receive thread safe records count.
3. Client request TSQLBigData records from index 0 to 20, for the needs of user-interface.
4. As client scroll, it will need other TSQLBigData records, located at view index position. Returned rows will be according current client view state. Ex. If client request row idx 15 for view, have not still received changes from the server, but record 10 is already deleted and logged as delete event, the server will know that it should return record idx 14 because when the client process deletion of record 10, the requested row 15 will be at position 14.
5. Client will be notified for table count change, inserts, deletes, updates, only for the data, which is already requested in view. It will be up to the client to define the UI refresh interval.
6. Client may change (inset/delete/update) the set. Every other listening client, which have focus on changing record will be notified for the change.

It will be transparent to client how big the data is, how often it is updated, does it persists, etc.
Client will be able to show both big data sources and high intensive ones.

All low-level functionality already done and in production. I can not write here about all implementation details, aspects and use-cases. This post is already too long. But I would like to see them in mORMot framework, hoping they will help other users too.