mORMot Open Source

And one remark: if we deal with JavaScript client (I mean browsers) it's good to start property name in JSON with a small letter. For example {"fieldCount":9,"rowCount":50,"values": [..]}. Name conversation in JS world: property start from small letter, constructor start from Upper case someObject.Create() but someObject.height. When I evaluate JSON in JS I got object and naming conversion is not mandatory, but is desirable. (There is no different in JS between constructor property or method - all is property, but method property if function type. So only one way to good reading code is naming conversation. JS is case sensitive).

AB, I test my system with large query (20000 - 500000 rows)  and find some way to increase performance:
in TSQLTableJSON.ParseAndConvert if we parse "not expanded" format we call GetRowCountNotExpanded function to calculate rowCount - it call take about 5% of all time. My propose is to add property rowCount for "not expanded" format  {"FieldCount":9,"rowCount":50,"Values": [..]}
We always know how many row we fetch from datasource, so it's simple. In my test I have 5% performance increase when remove GetRowCountNotExpanded call. (for 400 000 rows dataset from 4.09 to 3.88 sec on round trip for MS SQL).
Is it possible?

About blog entry http://blog.synopse.info/post/2012/07/1 … erformance
Using BULK operation in oracle (the same is true for MS SQL I think) give huge speed up not only due to the network latency, but in most case because of internal Oracle optimization for such query - in BULK mode all check  (unique index, constraint and so on) disabled for insert time and Oracle do it after inserting all record. Index rebuild and transaction logging made in the same way (after all inserts). All needed extents allocated at once. And so on. It's ease to reproduce that network latency is not a main problem - stored procedure in PL SQL give about 100x speed up too then using BULK.

Hi!
We found some issue using CustomJSONSerializer. I need to define only custom writer for some class but use standard reader  - it not possible in current implementation. We made fix for this problem - I send sources to your mail. Please, include it in main branch. I make corresponding ticket also for tracking.

@ab
For now I don't really need bulk insert - there is some architecture problems with bulk in my system..... (I need per record logic).
About ODBC - it become really need to mORMot when we migrate to x64 - there is no x64 OleDB drivers for latest MS SQL. But for now OleDB is enough.

But BULK operation in mORMot is cool! When I use oracle loader (it's a tools for import data using bulk insert) - the best result I reached is about 100000 rec/sec but in production server (UNIX, 12 Gb, 8 processor, very fast storage, Oracle optimization), so 50000 in simple computer is VERY GOOD result! Congratulations!

About Oracle:
1) look on your Oracle performance. Try this example on your Oracle: http://www.sqlines.com/sql-server/inser … erformance. Example running time is the best result you can gain (in client-server life you code will be 2-5 times slower because of remote execution)
2) in your example you must commit transaction every 1000 execution for best performance (call Client.Commit; and when start new transaction) - Oracle don't like long transaction
3) to load big amount of data to Oracle the only way is to use Oracle bulk operation (supported by ODAC as I know) - 10-100 times faster compared to step by step insert. Not supported by mORMot yet.

About attributes - this is a good idea, but there is 2 problems:
1) mORMot work with Delphi versions (Delphi 5 and up) which dose not support attributes
2) new RTTI in Delphi (which support attribute) is VERY-VERY slow as for me.

If you do not use transaction and batch 600 insert per sec. is a good result. ( try same code with TQuery, for example )
If you want speed up your test on Oracle
1) use transaction and commit it for example every 1000 record inserted  (number of record inserted in one transaction depend of Oracle configuration)
2) use batch. (BatchAdd instead of Add in mORMot).

this is not mORMot limitation - such things must be done in any architecture.

To @ab

As I remember when I use TClientDataSet (long time ago in Delphi 3) in "inmemory" way, setting TClientDataSet.LogChanges := False give GREAT performance increase. I think it's true in other Delphi versions too and must be used in TSQLTableToDataSet.

2 ab.
I use overridden version of TSQLTable.GetJSONValues (when expant=false I write resp in {"fields":["fld1", "fld2"], "data": [[val1, val2,..valN],[val1, val2,..valN],...[]]} - it's more native for JavaScript client, but found a bug in your and my code

function TSQLTable.GetJSONValues(
       line:    if IsString(U^) then begin
must be:    if IsStringJSON(U^) then begin

it's important for string fields started from ZERO must me "code": "01", then IsStrint(U^) I got "code": 01 and JavaScript think code is integer..

may be exist other place where is changes necessary.....
I add ticket.

For example I'm implement binary data transfer by coding method-based service which take a multipart/form-data request(transfer file from client to server) as described in http://www.ietf.org/rfc/rfc1867.txt but with some limitation.
I use browser as client and this is only way to transfer binary data without base64 encoding (base64 not supported by IE<10).
In case of delphi client everything will be much easier, I think..

@ab Hi!
I'm implement HumanReadable writing and reading of set and enumeration property types.

I'm use existed HumanReadable param of  ObjectToJSON function - it should not break other projects that use mORMot if they use JSONToObject for reading.....

With my modification if HumanReadable=True then function ObjectToJSON write (for example)
property logLevel: TSynLogInfos read FlogLevel write SetlogLevel
as RawUTF8StringArray like
"logLevel":["sllDebug","sllTrace","sllWarning","sllError","sllCustom1","sllCustom2","sllCustom3"],

and
property logLevel:  TSynLogInfo  read FlogLevel write SetlogLevel
as
"logLevel": "sllDebug",

function JSONToObject read set and enum writed as integer or in new notation

I'm sent SQLite3Commons to your mail - please, review and if it's possible, include it in main branch.

I'm use this functionality to store configuration files of my application and to pass JSON to JavaScript client in debug mode.

It's a solution, but it not good - setting async:false will block current browser page. The best(and right in javaScritp world) way is async call and making next request in success callback function. Something like jquery.ajax({....., success: function(){.... jquery.ajax(....);}).  There are many solutions how to get round the problem of nested function calls - events for example. (I don't know how do in in jquery - I'm not use jquery).

If you need a thousands of concurrent users ISAPI is not a best choice. The best performance is to use http.sys + IOCP (via mORMot THttpApiServer) and SynDBOracle for database access. In this case you:
1) do not need IIS and oracle client - only WinXP SP2+ and oci.dll
2) can easy implement in-memory server cache, session information storage and other feature
3) get VERY good performance

It's appear in PDF 1.5

I use this documentation when working with pdf:  www.adobe.com/devnet/acrobat/pdfs/PDF32000_2008.pdf
Quote:

PageLayout
(Optional) A name object specifying the page layout shall be used when the document is opened:
SinglePageDisplay one page at a time
OneColumnDisplay the pages in one column
TwoColumnLeftDisplay the pages in two columns, with odd-numbered pages on the left
TwoColumnRightDisplay the pages in two columns, with odd-numbered pages on the right
TwoPageLeft(PDF 1.5) Display the pages two at a time, with odd-numbered pages on the left
TwoPageRight(PDF 1.5) Display the pages two at a time, with odd-numbered pages on the right
Default value: SinglePage.

I create 2 thread per CPU core. On 24 (4 CPU x 6 core) server is 48 thread. So it 85Mb per thread. I use SpiderMonkey for Server-side business logic. SpiderMonkey Garbage Collection is hard operation, and I try to run it as late as possible for every thread, so lot of memory is good . (For example try to open 48 different tabs in FireFox and type about:memory in address bar  - it's a 10-20Mb per tab - in my case - per connected client). Some client may in this time upload big(I support up to 10Mb) files, I use transformation for different type of file to HTML via user business logic for use it later as file preview - it's requires in-memory operation. Other client may create big (100 or more page) reports. For report generating and file upload is possible to use temporary files of course, but for SpiderMonkey and file transformation more memory=best performance.
Thanks for reply!

If you don't mind I can add it and post source to you..

"Smart" project for delphi programmers is like the GWT for Java programmers. But I think that's a bad idea to write on one high level language and translate code to other high level language (with performance and functionality lost). JavaScript is a very good language when use it smartly I develop server part of my project usin Synopse for transport and database layer and SpiderMonkey emadding for business logic. SpiderMonkey internal (C and C++ code) is really beautiful! And VERY fast - compartible and even better than .NET. Client part is writn on Sencha (ExtJS) and it very usefull too. But there are many other JS framework for build UI, like  JQueryUI or Yahoo, qooxdoo.org (it's a guys from ExtJS team).

Cool!!! Thanks a lot!

It's much better!
And adding
case URIMethod of
is a good idea – code becomes much more clear.
There is also a problem with SQL statement execute – I created ticket.

General remark - your framework is the best Delphy code I have seen for 15 years of developing. Very cool!