Adding JavaScript support for mORMot framework

edwinsn · 2014-04-10 05:38:05

Great job, you guys!

Can't wait to try this! Is the mozjs.dll file for SM 24 bigger?

Please don't forget to notify us here when it's done and the link to the new mozjs.dll

mpv wrote:

AB, the next day (april 8), we plan to merge current version SynSM & SynSMAPI with our new SM24 implementation, so please - do not modify sources.
Full list of changes can to JavaScript be found here:
Changes in SM 18
Changes in SM 19
Changes in SM 20
Changes in SM 21
Changes in SM 22
Changes in SM 23
Changes in SM 24
After this we start to:
redesign our debugger implementation to be contributed to mORMot ( Yes, we have debugger )
redesign out TObject descendant import mechanism to be contributed to mORMot
Last allow write in delphi (for "old" RTTI):
TMyClass = class
published
  function write(cx: PJSContext; argc: uintN; vp: pjsval): JSBool; // for OLD RTTI - only this signature of function
  property myProp: integer; // but all kind of property 
end;
TSMEngine.registerClass(TMyClass); 
And when in SM
var inst = new TMyClass();
inst.write('some arg');
inst.myProp;
Also we have the same but for new RTTI - it allow to publish to SM almost all methods of TObject

ab · 2014-04-10 09:04:28

Hello Pavel?

Any news?

:-)

edwinsn · 2014-04-10 11:40:34

Hi Arnaud,

With respect, I assume currently others sending you source code using email, wouldn't it much convenient for communication if the source code is hosted on github.com? Other can contribute to the source too.

ab wrote:

Hello Pavel?
Any news?
:-)

ab · 2014-04-10 13:51:16

You can contribute directly on our fossil server http://synopse.info/fossil
GitHub would not offer anything more.
See http://www.fossil-scm.org/index.html/do … v-git.wiki

Contributions are always welcome (we never refused to give a developer access account to our fossil repository), and any external commit is always validated and (most of the time) refactored before integration.
For instance, it should meet our KISS code style and high quality documentation criteria.
When we did input some code of external contribution directly into the trunk, with very small code review and validation, it was much difficult to integrate it later.

Most of the time, we prefer to merge the code here at Synopse.
It allows peer review, avoid most unexpected regressions, and sometimes add a more open way of implementing things.

mpv · 2014-04-10 14:17:24

edwinsn wrote:

Is the mozjs.dll file for SM 24 bigger?

mozjs-24.dll = 3 030 016 bytes
mozjs-17.dll = 1 729 536 bytes
Size increased mainly because of many C++ functions added.

ab wrote:

Any news?

almost finished. I think tomorrow or in Saturday I commit.

While merge we switch to last mORMot version and found very interesting bug (we think it is in out SM module or inside Mozilla, spend many time and found it inside last TTextWriter.AddNoJSONEscapeW version).
See http://synopse.info/fossil/tktview?name=a75c0c6759

ab · 2014-04-10 14:59:43

Thanks for the status update!

We are waiting for your input.

Ticket is already fixed.

Sorry for it!

edwinsn · 2014-04-10 16:33:25

@mpv, thanks for the update. The scale of file size is not a big deal.

@ab, I think it's not about the version control features, but the popularity of the system - how are the client programs? How many good tutorials? I assume many developers have used svn or git already. But I understand the reasons not to switch on your side.

ab wrote:

You can contribute directly on our fossil server http://synopse.info/fossil
GitHub would not offer anything more.
See http://www.fossil-scm.org/index.html/do … v-git.wiki
Contributions are always welcome (we never refused to give a developer access account to our fossil repository), and any external commit is always validated and (most of the time) refactored before integration.
For instance, it should meet our KISS code style and high quality documentation criteria.
When we did input some code of external contribution directly into the trunk, with very small code review and validation, it was much difficult to integrate it later.
Most of the time, we prefer to merge the code here at Synopse.
It allows peer review, avoid most unexpected regressions, and sometimes add a more open way of implementing things.

edwinsn · 2014-04-11 14:14:21

@mpv, at the beginning of the development, you avoided v8 because it provides c++ interfaces which is more complex than C API, I think you did not expect SpiderMonkey would drop C API, right?

mpv wrote:

starting from SM24 mozilla drop support for C api (only c++).

edwinsn · 2014-04-13 03:33:54

Hi ab and mpv,

A new finding: Browserify.org (http://browserify.org/) contains the browser versions of many node modules, and I did not dig deeply, but I guess those modules should work fine with spidermonkey without or with minimal changes.

mpv · 2014-04-13 10:49:45

We finish modification of SynSM* to migrate to SM24.
Bad news: on "22 - JavaScript HTTPApi web server" we see 20% performance lost on hi load

 >wrk -c1000 -t10 http://host:888/root/readme.md

on SM17 give 16000 RPS, on SM24 - 13000.
We continue investigation - may be we need to change compiler (currently we compile SM using Visual Studio 2010). But IMHO this is because internally many features migrate form C to C++.
Good news:

- we also include dubugger low-level API in this commit. (GUI part we commit later)
- now we know how to add two way ArrayBuffer support - this give us possibility to exchange binary data with SM. We commit this feature in few days.

edwinsn wrote:

I think you did not expect SpiderMonkey would drop C API, right?

Yes, this is unexpected Mozilla design. But internally all function we need is steel have C-style, so we only need to modify some C++ headers.

edwinsn wrote:

Browserify.org

Yes, we know this lib. Some part work, some - not. The main problem is setTimeout/setInterval.

@AB - I will definitely find the time to learn Fossil - I use tortioseSVN for last 10 year and very used to the GUI and IDE integration. IMHO this is the main things fossil is mis.
While sending a letter..

ab · 2014-04-13 14:38:55

Thanks for sharing.

This performance regression is indeed disappointing...
I found out that VS 2010 has a good compiler under Windows.
Perhaps gcc may work better, but you are right: it is probably not the compiler, but the new C++ based architecture - perhaps the gcc RTL is more optimized.

edwinsn · 2014-04-13 15:04:50

mpv wrote:

 >wrk -c1000 -t10 http://host:888/root/readme.md

This is interesting, I want to do a load test here too, I did a google search and it led me to this github repository (https://github.com/wg/wrk), but I'm not sure how to install and use it on **Windows**, any hints?

mpv wrote:

- we also include dubugger low-level API in this commit. (GUI part we commit later)

Congratulations on the great work! A debugger API is really a nice addition!

mpv wrote:

@AB - I will definitely find the time to learn Fossil - I use tortioseSVN for last 10 year and very used to the GUI and IDE integration. IMHO this is the main things fossil is missing.
While sending a letter..

Same here, that's why I suggest to move the source code to google code or github, but keep everything here. Just my suggestions.

edwinsn · 2014-04-13 16:06:54

mpv wrote:

Some part work, some - not. The main problem is setTimeout/setInterval.

Forget to ask: I assume we can implement those two functions as native functions in the Delphi host?

edwinsn · 2014-04-13 18:08:27

ok, on my i5/6GB local PC, I just did a load test using a command line tool called openload (http://openwebload.sourceforge.net/) and got the following result, using the nightly build ( I think it's SpiderMonkey 1.8.5):

D:\>openload localhost:888/root/readme.md 1000
URL: http://localhost:888/root/readme.md
Clients: 1000
MaTps 814.00, Tps 814.00, Resp Time  0.138, Err   0%, Count   814
MaTps 907.20, Tps 1746.00, Resp Time  0.074, Err   0%, Count  2560
MaTps 985.53, Tps 1690.55, Resp Time  0.072, Err   0%, Count  4259
MaTps 1056.18, Tps 1692.00, Resp Time  0.079, Err   0%, Count  5951
MaTps 1120.82, Tps 1702.59, Resp Time  0.074, Err   0%, Count  7657
MaTps 1167.62, Tps 1588.82, Resp Time  0.081, Err   0%, Count  9249
MaTps 1208.67, Tps 1578.06, Resp Time  0.080, Err   0%, Count 10846
MaTps 1259.41, Tps 1716.12, Resp Time  0.075, Err   0%, Count 12581

The second column, Tps, is the number of completed requests during a second.

I will do the same test once I got the SM24 version and we'll see how it'll go.

mpv · 2014-04-13 20:22:40

I start new topic about load testing - http://synopse.info/forum/viewtopic.php?id=1700

About native implementation of setTimeout/setInterval - this is not easy. To implement it we need to implement something like Coroutine

mpv · 2014-04-14 13:50:49

It seems we solve performance problem. This is because Garbage collection model is changed in SM24. We need small additional investigation how to tune it.... Commit coming soon..

edwinsn · 2014-04-14 14:17:57

mpv wrote:

It seems we solve performance problem. This is because Garbage collection model is changed in SM24. We need small additional investigation how to tune it.... Commit coming soon..

That's really excellent job mpv!

At a later time I **maybe** (just maybe) benchmark SM with the Besen pure Pascal implementation JS engine

ab · 2014-04-21 07:51:25

Here we are back!

I've integrated SM 24, and updated the binaries.
See http://synopse.info/fossil/info/687f12d548

What about the small GC-related performance issue?
What about providing the C wrapper source code?

Feedback is welcome!

edwinsn · 2014-04-21 15:30:23

ab wrote:

Here we are back!
I've integrated SM 24, and updated the binaries.
See http://synopse.info/fossil/info/687f12d548
What about the small GC-related performance issue?
What about providing the C wrapper source code?
Feedback is welcome!

I suggest mpv to host the C wrapper source code on github or google code, since it's standalone. And this is really an excellent idea to get around the difficulty of wrapping c++ library using Delphi!

I think I'll be able to test it in the near future

mpv · 2014-04-22 06:49:55

We are still looking for the performance problem. SM24 itself is very fast, but if we use it via our wrapper something going wrong (seems IonMonkey not work). So we need more time to solve problem.
Simple test:

function factorial(val){
  return val === 1 ? 1 : val * factorial(val-1);
}
var d = new Date();
var res = 0;
for (var i = 0; i < 500000; i++){
  res += factorial(100);
}
d = new Date().getTime() - d.getTime();

sm24.exe - 983ms
sm17.exe - 2600ms
our wrapper for sm42 - 7800ms (the same result with jsoIon and without jsoIon, so it not work)
native Delphi implementation - 1100ms

So sm24 itself is faster when native Delphi... Very unexpected for me.

Native code:

procedure TForm1.btn1Click(Sender: TObject);
var
  n, x: Integer;
  res: Double;
  T: TPrecisionTimer;
  function factorial(x: Integer): Double;
  begin
     if x = 1 then
        result :=  x
     else
        result := x * factorial(x-1);
  end;
begin
  res := 0; x := 100;
  T.Start;
  for n := 0 to 500000 do
    res := res + factorial(x);
  mmoRes.Lines.Add(T.Stop);
  mmoRes.Lines.Add(FloatToStr(res));
end;

ab · 2014-04-22 07:21:06

Delphi generates very unoptimized x87 FPU code, which is much slower than the SSE code JITted by SM.
See http://www.delphitools.info/2011/09/02/ … erformance

Yes, IonMonkey is a need...
Is there not any flag or API to explicitly enable it when initializing the runtime?

edwinsn · 2014-04-23 03:36:48

is it possible that the problem is coming from the c wrapper?

edwinsn · 2014-04-23 04:58:40

Hi mpv, does the spiderMonkey dll expose the JS parser API that can generate AST (abstract syntax tree), as described here:
https://developer.mozilla.org/en-US/doc … Parser_API

mpv · 2014-04-24 18:37:48

We finish performance problem solving. Regressiin test AB provide also fixed. Tonight we prettify code and describe the root of problem we solv. Also we add ArayBuffer API.
About AST - this is actually a part of Firefox, not SM itself. But on the low level it is posible to do the same operation as in reflect.jsm...

edwinsn · 2014-04-25 05:12:23

Great! Great! You guys!
I'll check the low level API someday.

mpv wrote:

We finish performance problem solving. Regressiin test AB provide also fixed. Tonight we prettify code and describe the root of problem we solv. Also we add ArayBuffer API.
About AST - this is actually a part of Firefox, not SM itself. But on the low level it is posible to do the same operation as in reflect.jsm...

ab · 2014-04-25 16:58:38

I received the files so I will update the source code repository soon....

edwinsn · 2014-04-28 13:33:54

ab wrote:

I received the files so I will update the source code repository soon....

Great job ab and mpv! Yester day I downloaded and it complied and the jshttpserver demo worked.

I've got a question for mpv though, shortly I'll need to make a decision on using the spidermonkey wrapper, one key factor would be, are you planning to release the source code of the 'c wrapper over the c++' ab mentioned earlier? Thanks.

mpv · 2014-04-28 15:50:53

Yes, we will create patch for Mozilla mercurial repository ASAP. But I'm little bit busy now

edwinsn · 2014-04-28 16:18:39

mpv wrote:

Yes, we will create patch for Mozilla mercurial repository ASAP. But I'm little bit busy now

Great! And you must know that I'm not pushing you, just need to make sure you'll eventually release it, time frame is not a concern

edwinsn · 2014-06-12 05:50:21

Hi Folks, any news on this

edwinsn · 2014-06-29 03:12:39

@ab, would you contact mpv via e-mail ask for the source code of the DLL which he promised to publish it? I think it's a key aspect to keep the JavaScript part of the mORMot framework live and going?

mpv · 2014-07-03 10:15:16

Hello. Sorry that disappeared....
We sand patch (and instruction) for C++ SpiderMonkey dll compilation to AB's e-mail.
@edwinsn - about your question about AST (abstract syntax tree) generation - take a look at http://ternjs.net - this is JS parser written on JS

edwinsn · 2014-07-03 10:48:58

@mpv, welcome back! And great to hear that. And yes I already know Ternjs, which is written by an amazing developer!

ab · 2014-07-03 14:49:15

We have updated the SynSM/SynSMAPI units:
http://synopse.info/fossil/info/5ca6fab … 75cd1e25f0

And added the mozjs patches to our source code repository:
http://synopse.info/fossil/info/f10b5ce … 0f8152e2eb

See http://synopse.info/fossil/dir?ci=tip&n … &type=tree for a tree view of the new folder.
There are low-level compilation instruction available, too.

Thanks mpv for sharing!

edwinsn · 2014-07-04 07:12:30

Thank you guys!

I checked the BuildInstruction.md file, there is only one thing not clear to me - what program to use to apply those .patch files? Thanks.

mpv · 2014-07-04 09:34:04

*.patch is unified diff format. We generate out patch (Delphi patch.patch) using mercurial - the same repository Mozilla use. But you can apply this patch by any software, support unified diff (svn e.t.c.). In case of mercurial - more information here: http://mercurial.selenic.com/wiki/PatchFile. In short:

>hg import my-patch-file.patch

or visually using tortoiseHg - http://tortoisehg.bitbucket.org/manual/1.0/patches.html

Last edited by mpv (2014-07-04 09:35:05)

edwinsn · 2014-07-04 12:48:16

Hi MPV,

It's great! Thanks for the help. I'm not trying to build the dll in the near future, just wanted to get things clarified

win2014 · 2014-08-05 00:42:56

@mpv What things to do in JS_AbortIfWrongThread in sm, in delphi debug evironment, I found it raise zero point write av exception.
Hope reply.

mpv · 2014-08-06 09:45:53

You try to do something with javaScript runtime not from thread this runtime created. If this happens in delphi debugger in multithread application this is "normal" situation, because debugger thread is not a thread where runtime initially created. In our implementation we create one runtime and one context per thread. See Mozilla sources:

JSRuntime::abortIfWrongThread() const
{
    if (ownerThread_ != PR_GetCurrentThread())
        MOZ_CRASH();
    if (!js::TlsPerThreadData.get()->associatedWith(this))
        MOZ_CRASH();
}

win2014 · 2014-08-07 04:02:13

mpv wrote:

You try to do something with javaScript runtime not from thread this runtime created. If this happens in delphi debugger in multithread application this is "normal" situation, because debugger thread is not a thread where runtime initially created. In our implementation we create one runtime and one context per thread. See Mozilla sources:
JSRuntime::abortIfWrongThread() const
{
    if (ownerThread_ != PR_GetCurrentThread())
        MOZ_CRASH();
    if (!js::TlsPerThreadData.get()->associatedWith(this))
        MOZ_CRASH();
}

Thanks mpv very much. I have read these statement above from sm source and I have trace into cpu viewer for asm, but my asm knowledge as my english, haha...
Thanks excellent sm wrapper again!

ab · 2014-08-20 11:11:45

Note to self: since JavaScript support is no official in the framework, I "unstick" this topic from our "mORMot framework" forum thread.

win2014 · 2014-08-21 08:31:46

Agree with ab. Now SynSM hides so deep.

win2014 · 2014-09-03 09:19:42

Hi @mpv,
Where is the debugger?

mpv · 2014-09-04 08:52:09

In our product we already integrate debugger. This feature consist of 3 part:
GUI (very specific to my application, so can not be shared) <-> Controller <-> TDebugger ( Object wrapper around SpiderMonkey debugger functions )
Controller and Debugger code still depend from my application so not ready yet for sharing (but we work on remove this dependency).
If you need it now, I can send you Controller and Debugger code to use as a sample for your implementation.....
Please, write me a mail ( my e-mail inside SynSMAPI header).

edwinsn · 2014-09-04 10:36:51

mpv wrote:

Controller and Debugger code still depend from my application so not ready yet for sharing (but we work on remove this dependency).

@mpv, that is a great news!

proto · 2014-10-30 10:19:19

Spidermonkey has passed V8 on Octane performance on arewefastyet, and is now leading V8 and JSC on Octane, Sunspider and Kraken.
http://robert.ocallahan.org/2014/10/are … e-are.html

mpv · 2014-10-30 20:38:12

And this is synthetic test. In real server side app Spidermonkey is faster compared to v8 up to 4 times.

ab · 2014-10-30 20:42:05

Really?
Is it the new jit?

mpv · 2014-11-02 10:22:10

By "real server app" I mean compare nodeJS and SynSM based application. So in my test I test not only JavaScript engine, but native implementation also..

ab · 2014-11-02 14:08:19

Nice feedback!_
:-)

mORMot Open Source

#51 2014-04-10 05:38:05

Re: Adding JavaScript support for mORMot framework

#52 2014-04-10 09:04:28

Re: Adding JavaScript support for mORMot framework

#53 2014-04-10 11:40:34

Re: Adding JavaScript support for mORMot framework

#54 2014-04-10 13:51:16

Re: Adding JavaScript support for mORMot framework

#55 2014-04-10 14:17:24

Re: Adding JavaScript support for mORMot framework

#56 2014-04-10 14:59:43

Re: Adding JavaScript support for mORMot framework

#57 2014-04-10 16:33:25

Re: Adding JavaScript support for mORMot framework

#58 2014-04-11 14:14:21

Re: Adding JavaScript support for mORMot framework

#59 2014-04-13 03:33:54

Re: Adding JavaScript support for mORMot framework

#60 2014-04-13 10:49:45

Re: Adding JavaScript support for mORMot framework

#61 2014-04-13 14:38:55

Re: Adding JavaScript support for mORMot framework

#62 2014-04-13 15:04:50

Re: Adding JavaScript support for mORMot framework

#63 2014-04-13 16:06:54

Re: Adding JavaScript support for mORMot framework

#64 2014-04-13 18:08:27

Re: Adding JavaScript support for mORMot framework

#65 2014-04-13 20:22:40

Re: Adding JavaScript support for mORMot framework

#66 2014-04-14 13:50:49

Re: Adding JavaScript support for mORMot framework

#67 2014-04-14 14:17:57

Re: Adding JavaScript support for mORMot framework

#68 2014-04-21 07:51:25

Re: Adding JavaScript support for mORMot framework

#69 2014-04-21 15:30:23

Re: Adding JavaScript support for mORMot framework

#70 2014-04-22 06:49:55

Re: Adding JavaScript support for mORMot framework

#71 2014-04-22 07:21:06

Re: Adding JavaScript support for mORMot framework

#72 2014-04-23 03:36:48

Re: Adding JavaScript support for mORMot framework

#73 2014-04-23 04:58:40

Re: Adding JavaScript support for mORMot framework

#74 2014-04-24 18:37:48

Re: Adding JavaScript support for mORMot framework

#75 2014-04-25 05:12:23

Re: Adding JavaScript support for mORMot framework

#76 2014-04-25 16:58:38

Re: Adding JavaScript support for mORMot framework

#77 2014-04-28 13:33:54

Re: Adding JavaScript support for mORMot framework

#78 2014-04-28 15:50:53

Re: Adding JavaScript support for mORMot framework

#79 2014-04-28 16:18:39

Re: Adding JavaScript support for mORMot framework

#80 2014-06-12 05:50:21

Re: Adding JavaScript support for mORMot framework

#81 2014-06-29 03:12:39

Re: Adding JavaScript support for mORMot framework

#82 2014-07-03 10:15:16

Re: Adding JavaScript support for mORMot framework

#83 2014-07-03 10:48:58

Re: Adding JavaScript support for mORMot framework

#84 2014-07-03 14:49:15

Re: Adding JavaScript support for mORMot framework

#85 2014-07-04 07:12:30

Re: Adding JavaScript support for mORMot framework

#86 2014-07-04 09:34:04

Re: Adding JavaScript support for mORMot framework

#87 2014-07-04 12:48:16

Re: Adding JavaScript support for mORMot framework

#88 2014-08-05 00:42:56

Re: Adding JavaScript support for mORMot framework

#89 2014-08-06 09:45:53

Re: Adding JavaScript support for mORMot framework

#90 2014-08-07 04:02:13