Benchmarking JsonDataObjects JSON parser

ab · 2015-02-16 11:31:46

There is a new player in town.
Since it has been written by Andreas Hausladen, the maintainer of the great Delphi IDE fix packs, this new JSON library is very promising.

And in fact, it is fast, and sounds pretty great!
Here are some numbers, compared with SuperObject, standard DBXJson, dwsJSON, QDAC and mORMot.

Comments for http://blog.synopse.info/post/2015/02/1 … SON-parser blog article.

See http://synopse.info/fossil/info/efcb1f0b854dd2b7ec54 for the corresponding commit.

ab · 2015-02-16 12:10:17

Here are some updated numbers:

   JSON benchmarking
  -------------------


1. Small content

 1.1. Synopse record:
  - Read: 50,000 assertions passed  157.71ms  317,031/s
  - Access: 100,000 assertions passed  986us  50,709,939/s
  - Write: 50,000 assertions passed  107.84ms  463,636/s
  Total failed: 0 / 200,000  - Synopse record PASSED  267.13ms

 1.2. Synopse variant:
  - Read: 50,000 assertions passed  365.76ms  136,700/s
  - Access direct: 100,000 assertions passed  64.99ms  769,242/s
  - Access late binding: 100,000 assertions passed  701.11ms  71,314/s
  - Write: 50,000 assertions passed  210.13ms  237,943/s
  Total failed: 0 / 300,000  - Synopse variant PASSED  1.34s

 1.3. Synopse cross platform variant:
  - Read: 50,000 assertions passed  651.62ms  76,730/s
  - Access direct: 100,000 assertions passed  42.30ms  1,181,921/s
  - Access late binding: 100,000 assertions passed  615.85ms  81,187/s
  - Write: 50,000 assertions passed  626.12ms  79,856/s
  Total failed: 0 / 300,000  - Synopse cross platform variant PASSED  1.93s

 1.4. Super object record:
  - Read: 50,000 assertions passed  4.33s  11,536/s
  - Access: 100,000 assertions passed  1.15ms  43,327,556/s
  - Write: 50,000 assertions passed  3.75s  13,301/s
  Total failed: 0 / 200,000  - Super object record PASSED  8.09s

 1.5. Super object properties:
  - Read: 50,000 assertions passed  4.74s  10,548/s
  - Access: 100,000 assertions passed  4.19s  11,911/s
  - Write: 50,000 assertions passed  368.69ms  135,614/s
  Total failed: 0 / 200,000  - Super object properties PASSED  9.30s

 1.6. dws JSON:
  - Read: 50,000 assertions passed  281.96ms  177,328/s
  - Access: 100,000 assertions passed  85.91ms  581,970/s
  - Write: 50,000 assertions passed  216.47ms  230,976/s
  Total failed: 0 / 200,000  - dws JSON PASSED  586.38ms

 1.7. DBXJSON:
  - Read: 50,000 assertions passed  4.81s  10,376/s
  - Access: 100,000 assertions passed  48.64ms  1,027,812/s
  - Write: 50,000 assertions passed  620.93ms  80,524/s
  Total failed: 0 / 200,000  - DBXJSON PASSED  5.48s

 1.8. QDAC:
  - Read: 50,000 assertions passed  632.46ms  79,056/s
  - Access: 100,000 assertions passed  99.81ms  500,941/s
  - Write: 50,000 assertions passed  395.22ms  126,511/s
  Total failed: 0 / 200,000  - QDAC PASSED  1.13s

 1.9. Json data objects:
  - Read: 50,000 assertions passed  249.29ms  200,567/s
  - Access: 100,000 assertions passed  8.00ms  6,246,096/s
  - Write: 50,000 assertions passed  112.64ms  443,892/s
  Total failed: 0 / 200,000  - Json data objects PASSED  372.80ms


2. Big content

 2.1. Depth content:
  - Download files if necessary: no assertion  520us
  - Synopse read variant: 1 assertion passed  162.52ms  307,637/s  227 KB
  - Synopse read to BSON: 2 assertions passed  2.72ms  18,375,597/s  159 KB
  - Synopse cross platform: 1 assertion passed  4.01ms  12,465,719/s  300 KB
  - Super object read: 2 assertions passed  9.04ms  5,526,083/s  529 KB
  - dws JSON read: 1 assertion passed  4.88ms  10,231,225/s  307 KB
  - DBXJSON read: 1 assertion passed  93.11ms  536,970/s  679 KB
  - Json data objects read: 1 assertion passed  2.74ms  18,248,175/s  272 KB
  - QDAC read: 1 assertion passed  9.16ms  5,454,942/s  389 KB
  Total failed: 0 / 10  - Depth content PASSED  299.09ms

 2.2. Table content:
  - Download files if necessary: no assertion  242us  34,000,000/s
  - Synopse parse: 1 assertion passed  2.96ms  2,776,577/s  1.1 MB
  - Synopse table index: 41,135 assertions passed  5.16ms  1,594,379/s  1.1 MB
  - Synopse table cached: 41,135 assertions passed  3.24ms  2,534,504/s  1.1 MB
  - Synopse table loop: 41,135 assertions passed  5.05ms  1,626,210/s  1.1 MB
  - Synopse table variant: 41,135 assertions passed  98.75ms  83,307/s  1.1 MB
  - Synopse ORM loop: 41,135 assertions passed  6.78ms  1,212,527/s  1.2 MB
  - Synopse ORM list: 41,135 assertions passed  6.91ms  1,189,560/s  1020 KB
  - Synopse doc variant: 41,137 assertions passed  30.72ms  267,771/s  3.0 MB
  - Synopse late binding: 41,137 assertions passed  107.95ms  76,211/s  3.0 MB
  - Synopse cross ORM: 41,135 assertions passed  18.41ms  446,731/s  1.9 MB
  - Synopse cross direct: 41,135 assertions passed  19.77ms  416,135/s  1.9 MB
  - Synopse cross variant: 41,135 assertions passed  98.71ms  83,344/s  1.9 MB
  - Synopse to BSON: 2 assertions passed  10.38ms  791,895/s  1.0 MB
  - Super object props: 41,136 assertions passed  2.36s  3,475/s  6.3 MB
  - Super object record: 41,136 assertions passed  149.53ms  55,016/s  6.3 MB
  - dws JSON: 41,136 assertions passed  30.68ms  268,093/s  4.7 MB
  - DBXJSON: 41,136 assertions passed  267.06ms  30,805/s  9.10 MB
  - JsonDataObjects: 41,136 assertions passed  17.00ms  483,912/s  2.6 MB
  - QDAC: 41,136 assertions passed  41.45ms  198,446/s  5.9 MB
  Total failed: 0 / 699,308  - Table content PASSED  3.31s

 2.3. Huge content:
  - Download files if necessary: no assertion  504us
  - Synopse beautifier: 1 assertion passed  1.00s  49,969/s  104 B
  - Synopse read record: 4 assertions passed  1.46s  140,598/s  113.5 MB
  - Synopse read variant: 2 assertions passed  3.28s  62,833/s  377.4 MB
  - Synopse cross platform: 2 assertions passed  5.44s  37,938/s  424.8 MB
  - Synopse read to BSON: 3 assertions passed  2.03s  101,744/s  168.1 MB
  - Super object read: 2 assertions passed  9.13s  22,615/s  1.1 GB
  - dws JSON read: 2 assertions passed  2.97s  69,398/s  672.7 MB
  - DBXJSON read: no assertion  253us  197,628,458/s
     DBXJSON will raise EOutOfMemory for 185 MB JSON in Win32 -> skip
  - Json data objects read: 2 assertions passed  1.70s  121,294/s  329.6 MB
  - Json data objects beautifier: no assertion  7.66s  6,526/s  510.6 MB
  - QDAC read: 2 assertions passed  8.24s  25,046/s  1.1 GB
  Total failed: 0 / 20  - Huge content PASSED  43.92s


Generated with: Delphi XE7 compiler

Time elapsed for all tests: 76.08s
Tests performed at 16/02/2015 13:06:15

Total assertions failed for all test suits:  0 / 2,699,338
! All tests passed successfully.

In short:
- We did include the latest QDAC revision, which fixed the fieldname = "" bug;
- We fixed SuperObject to let the huge content test pass;
- We presented several ways of TSQLTableJSON access (as "Synopse Table" numbers).

The advantage of our SAX approach is clear with TTestTableContent.SynopseTableCached, in terms of performance.
More than 5 times faster than JsonDataObjects.

Wodzu · 2015-02-17 06:59:00

Thank you for making the tests:) Unfortunately the results are barely readable in this format. Couldn't you put it in some excel charts? We would immediately see the differences between parsers.

bookbarsa · 2015-02-17 18:59:59

Unfortunately it is not compatible with Lazarus.
Ab, which JSON Parser you recommend for Lazarus?

ab · 2015-02-17 19:46:19

Both Synopse parsers are compatible with Lazarus!
You can use the one in SynCommons.pas + mORMot.pas under Windows or Linux, or SynCrossPlatformREST.pas on most FPC platforms.

If you use TDocVariant or TJSONVariant and latebinding, ensure you got the latest version from SVN, including the needed http://mantis.freepascal.org/view.php?id=26773 fix.
See http://synopse.info/files/html/Synopse% … l#TITL_125

z. · 2015-02-25 06:36:57

Your tests do not include any real write test - create JSON structure programmatically and change its values (S['title']:='data') - if I am not mistaken.
Is there any reason why it is not included? Because that would probably influence results a lot.

ab · 2015-02-25 12:29:46

The first tests also benchmark writes.

Then others are more read centered, with write testing for the refactoring.

z. · 2015-02-25 13:39:23

I have this test code:

      for j := 0 to 10000 - 1 do
        begin
          json.S[IntToStr(j)+'S']:='abcdefg';
          json.I[IntToStr(j)+'I']:=1234;
        end;

and JsonDataObjects is 5 times slower than SuperObject (which I currently use).

So I am not sure if I do anything wrong or you may consider something like that in your tests

ab · 2015-02-25 19:26:59

In fact, you are not writing anything here.
You are setting property values, without using JSON serialization itself.

But it may be a good idea to add a WriteAccess method benchmark in addition to the existing Access method, which may be renamed ReadAccess.

edwinsn · 2019-03-14 07:19:43

FYI:
A comprehensive test suite for RFC 8259 compliant JSON parsers: https://github.com/nst/JSONTestSuite

I came across this and thought I'd post it here.

ab · 2019-03-14 08:35:33

Our JSON parser is more relaxed than most JSON parsers: it supports the "MongoDB extended syntax" for instance.
See https://docs.mongodb.com/manual/referen … nded-json/

So I guess some tests may fail.

mORMot Open Source

#1 2015-02-16 11:31:46

Benchmarking JsonDataObjects JSON parser

#2 2015-02-16 12:10:17

Re: Benchmarking JsonDataObjects JSON parser

#3 2015-02-17 06:59:00

Re: Benchmarking JsonDataObjects JSON parser

#4 2015-02-17 18:59:59

Re: Benchmarking JsonDataObjects JSON parser

#5 2015-02-17 19:46:19

Re: Benchmarking JsonDataObjects JSON parser

#6 2015-02-25 06:36:57

Re: Benchmarking JsonDataObjects JSON parser

#7 2015-02-25 12:29:46

Re: Benchmarking JsonDataObjects JSON parser

#8 2015-02-25 13:39:23

Re: Benchmarking JsonDataObjects JSON parser

#9 2015-02-25 19:26:59

Re: Benchmarking JsonDataObjects JSON parser

#10 2019-03-14 07:19:43

Re: Benchmarking JsonDataObjects JSON parser

#11 2019-03-14 08:35:33

Re: Benchmarking JsonDataObjects JSON parser

Board footer