#1 Re: mORMot 2 » Create SynCrossPlatform Wrapper with m2 broken » 2024-03-20 12:51:38

Comparing M1/M2 context, main deafferents is M2 have arrays, M1 don't have (in FPC).
And nestedRecord/Array is array of fields in M1, in M2 we have "fields" property as array of fields.
context.json M1 and M2 https://mega.nz/file/SbBUGaYQ#fwFb07BBX … qM4J3hyeU0
for sample https://synopse.info/forum/viewtopic.ph … 163#p41163

#2 Re: mORMot 2 » Create SynCrossPlatform Wrapper with m2 broken » 2024-03-20 11:26:01

Hi @itSDS,
0. Can you check/compare M1/M2 context.json, http://localhost:888/root/wrapper/context
1. Yes, depend of mustache template. I'm not sure that is possible correct order of types. I only look at SMS template to convert to pas2js template and in this template first is created all enums, then all record, then all arrays. Manuel modification is needed (reorder types) in created client.
2. Check 0. In M2 context we have all nested records/arrays as temporary type. This is needed for client that don't support nested types. Pas2js support nested records and arrays from some version, but don't support nested array of nested record and we need this temp types

// New Delphi/FPC version support
TPeople = packed record
  MyArr: array of record
                 name: RawUtf8;
               end;
end;
// in Pas2JS, must be transformed to
TTmp1 = record
  name: RawUtf8;
end;
TMyArr2 = array of TTmp1;
TPeople: TMyArr2;

This temp types also simplify code to all helper variant2* *2variant functions, if you use this types.
Some changes must be done in template if you don't need this temp types. Looking at context.json

					"nestedRecord": {
						"nestedRecord": null,
						"fields": [
							{
								"typeWrapper": "wRawUtf8",
								"typeSource": "RawUtf8",
								"typeDelphi": "RawUtf8",
								"typePascal": "String",
								"typeCS": "string",
								"typeJava": "String",
								"typeTS": "string",
								"typeSwagger": {
									"type": "string"
								},
								"propName": "Name",                 <---
								"fullPropName": "CatNested.Name",   <---
								"nestedIdentation": "  ",          
								"isSimple": true
							},

You can change in template fullPropName to propName, depend what you need to be generated.
I tested only on FPC and Pas2JS template. Delphi testing is needed and all clients templates.
It is good to add field isNested: true to all anonymous temp types to be able to create or not code for this types in template!

#3 Re: mORMot 2 » Create SynCrossPlatform Wrapper with m2 broken » 2024-03-19 20:56:32

Please try/test my fork mormot.soa.codegen.pas.  Pull request is 2 days ago, testing is welcome before merge.
https://github.com/TTomas/mORMot2/blob/ … odegen.pas
Discussed in topic https://synopse.info/forum/viewtopic.php?id=6375

#4 Re: mORMot 1 » TMS WebCore » 2024-03-17 14:42:14

@ab, I create pull request
@rdevine, I update pull request
This type of interface is tested

  TSex = (cMale, cFemale);

  TCat = packed record
    Name: RawUtf8;
    Sex: TSex;
    Birthday: TDateTime;
  end;

  TCatDynArray = array of TCat;
  TCat3Array = array[0..2] of TCat;

  TPeople = packed record
    FirstName: RawUtf8;
    LastName: RawUtf8;
    Sex: TSex;
    Birthday: TDateTime;
    Cat: TCat;
    CatNested: packed record
        Name: RawUtf8;
        Sex: TSex;
        Birthday: TDateTime;
      end;
    Cat3: TCat3Array;
    Cats: TCatDynArray;
    CatsNested: array of TCat;
  end;

  ICalculator = interface(IInvokable)
    ['{9A60C8ED-CEB2-4E09-87D4-4A16F496E5FE}']
    function Add(n1, n2: integer): integer;
    function ArrayValue(const arrJSON: RawUtf8; ix: integer): variant;
    function CountArray(const jsn: RawUtf8): integer;
    function SumArray(const jsn: RawUtf8): double;
    procedure FullName(const aFirstName, aLastName: RawUtf8;
      var aFullName: RawUtf8; var aSize: integer);
    function CatIsMale(const aCat: TCat): Boolean;
    function GetCat: TCat;
    function GetPeople(aId: integer; var aPeople: TPeople): Boolean;
    function AddPeople(const aPeople: TPeople): integer;
    function AddCat2People(const aCat: TCat; var aPeople: TPeople): boolean;
  end; 

With this changes created client need only 2 manual changes, move array types before TPeople record and Replace All nested type names with correct type name.
In https://github.com/TTomas/mORMot-Web/tr … Pas2JSTest you can view context.json and exported client mORMotPas2JsClientExport.pas and edited mORMotPas2JsClient.pas.
I try version without helper record/array function, In GetPeople onSuccess to use

      onSuccess(TPeople(res[0]),Boolean(res[1]));

For simple types we can use this casting, but Birthday TDateTime field will be corrupt in pas2js. After this type of casting this field is no longer TDateTime (double/Number) but String (mormot serialize as iso8601) and using any datetime function is broken. This is JS world!
Using helper functions everythig is OK. Template is modified SMS and still there is a lot of SMS code in template. Need more testing on different structures.
@ab we need correct type naming for temporary nested fields (CatNested/CatsNested).

#5 Re: mORMot 1 » TMS WebCore » 2024-03-11 15:06:32

Hi @rdevine
I create Pull request
-Some small changes to be Pas2JS compatible without TMS Web Core
-Calculator service add code export wrappers and add procedure with 2 var params for testing
-Pas2JS Template in Template folder. Generated code mORMotPas2JsClient.pas is used in Pas2JSTest without any change
http://localhost:888/root/wrapper
-Pas2JSTest Demo
-Async and Sync calls works
I tested only on FPC and Pas2JS, not in Delphi or TMS Web Core
My fork: https://github.com/TTomas/mORMot-Web

#6 Re: mORMot 1 » Firebird transactions » 2024-02-02 10:30:40

Thanks @vs for report of exception with LEGACYFIREBIRDAPIONLY. I found problem with definition of isc_start_transaction dll function in fbintf.
More info in this topic: https://forum.lazarus.freepascal.org/in … opic=66087

#7 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2024-01-25 12:31:06

Thanks @ab, new router feature is great and very powerful addon.
Just one note, if I create SOA method <func>, without using any router.rewrite/setup, OnBeforeUri is called with all URI:
http://localhost/root/func/*
http://localhost/root/func/Variable/P/a/t/h/...
Is this as designed or working by chance (by mistake)?

#8 Re: mORMot 2 » Communication between microservices » 2024-01-22 21:45:21

Microservices with several servers usually use reverse proxy server.

                         -> Myservice1
client N -> proxy(nginx) -> Myservice2
                         -> Myservice3
                         -> Myservice1

#9 Re: mORMot 1 » Firebird transactions » 2024-01-20 20:01:35

Do you change line 616 cMaxStm = 50 in mormot.db.sql.ibx. Previous test on 2.5 pass without problem. Fb3 have limit of 255 statements, Fb2.5???
Edited: Or execute statement size exceeded  64K  limit on Fb2.5, Fb3 and above have limit of 10MB.
Batch insert create execute block statement with 50 insert statements inside.
Do you try to add define LEGACYFIREBIRDAPIONLY to force using of legacy api with Fb4/5?

#10 Re: mORMot 1 » Firebird transactions » 2024-01-20 00:28:11

@vs, do you success build with delphi, if yes delphi version please. @ab alearedy commit changes in trunk.

#11 Re: mORMot 1 » Firebird transactions » 2024-01-19 23:45:18

@ab, I create push request.
2. Correct, this is the reason I create ibx mormot driver. Main goal is to have 0 active transaction if nobody is connected to app/server.
{$define ZEOSTRANS} is only for testing, not recomended for production.

#12 Re: mORMot 1 » Firebird transactions » 2024-01-19 16:52:28

This results is expected with default config. 3 reasons:
1. Zeos have better parameters and results binding, low level access to Fb Api records, without copy strings/varchars.
2. Zeos only use CommitRetaining after any statement and never close/Commit transaction. Ibx use StartTransaction, execute statement and CommitTransaction. This is by design. You can simulate zeos transaction handling if you enable {$define ZEOSTRANS} at line 48 in mormot.db.sql.ibx.
3. fbintf for Fb3 and above use new Firebird3 interface, by default, which is slower then old legacy ICS/Interbase Api. zeos use only legacy Api. Look in fbintf, IB.pas lines 76-81, you can force to use legacy Api interface only for Fb3/4/5 by defining LEGACYFIREBIRDAPIONLY define to your project. You will see Direct/Trans insertion speed increase for Fb4/5 in range of Fb25 (Fb2.5 use only legacy interface)

#13 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2024-01-19 13:50:29

ab wrote:

I am a bit confused about how you make the routing.

By parsing URI in OnBeforeURI event.

ab wrote:

Try to find out what is registered in TRestRouter.Setup for you application, and why your requests are not routed.
If you don't use the REST routing, you could use the HTTP server routing, or HTTP interception.

As I said before this RestServer doesn't have any ORM or SOA interface or method, just OnBeforeURI event and nothing is registered in Router. Adding:

  aWebhookRestServer.Router.Setup(mGET, 'test', rnMethod);
  aWebhookRestServer.Router.Setup(mPOST, '', rnMethod);

I solve problem for this project and now OnBeforeUri is called. I use MyRestServer just to separate code(baseness logic) and define newroot needed for reverse proxy. I will create two published methods and use router.rewrite instead of OnBeforeURI.

As this topic title said I have 2-3 years old mormot1 project broken to migrate to m2. This project replace some Magento(PHP)  eCommerce image processing URI, where I start using OnBeforeURI. RestServer root is 'Assets' and need to process:
http://localhost/Assets/image/Variable/P/a/t/h/image1.jpg
as static server serve original image and
http://localhost/Assets/img/<func>/<x>/<y>/Variable/P/a/t/h/image1.jpg
http://localhost/Assets/img/resize/200/200/Variable/P/a/t/h/image1.jpg
On first call resize original image to 200x200 and keep/cache on disk, next call serve image as static server.

I know this is not standard use of OnBeforeURI in RestServer and don't insist any change. This URI serving can be moved to HTTP server interception.
One possible change will be line 7546 if (node=nil) to try to call OnBeforeURI before ctx.Error('Invalid URI'...

#14 Re: mORMot 1 » Firebird transactions » 2024-01-18 20:57:47

vs wrote:

MVCServerFirebirdIbx.dpr(54,17) Error: identifier idents no member "CreateDescendingPK"

property CreateDescendingPK was changed to CreateDescendingOnlyPK, ex\mvc-blog need to change

vs wrote:

MVCServerFirebirdIbx.dpr(95,1) Error: Can't open resource file "D:\Projects\Delphi\Components\Synopse\mORMot2-2.0.stable\ex\mvc-blog\mormot.win.default.manifest.res"

I usually copy this file to project folder from mormot2/src. Line 10 can be changed to:

  {$R ..\..\src\mormot.win.default.manifest.res}

mormot.db.sql.ibx.pas(119):
property FirebirdLibraryPathName: RawUtf8
but
mormot.db.sql.ibx.pas(87):
fFirebirdLibraryPathName: string;

Yes need to be changed to string. I will try to create push request.

#15 Re: mORMot 1 » Firebird transactions » 2024-01-18 10:27:37

I see dcc32 Error, you use Delphi!
mormot.db.sql.ibx.pas is Fpc/Lazarus only, based on IBX for Lazarus interface: https://www.mwasoftware.co.uk/fb-pascal-api
Never tested with Delphi. Try to install  fbintf1.4-0 on Delphi. Any help is welcome.

#16 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2024-01-18 09:38:04

Sorry for bad line numbers not from trunk. I don't use router. TMyRestServer receive different aRoot on creation 'newroot/webhook', BeforeUri response to POST request on 'newroot/webhook' and GET request to 'newroot/webhook/test' for testing that service is alive. In mormot1 and mormot2 before router, OnBeforeUri is called for all 'newroot/webhook*' requests.

  TMyRestServer = class(TRestServerFullMemory)
  private
    function BeforeUri(Ctxt: TRestServerUriContext): boolean;
  public
    constructor CreateWithOwnModel(aModelServer: ThkOrmRestServerDB;
      aProps: TSqlDBConnectionPropertiesThreadSafe;
      aShApiConfig: ThkConfigSh;
      const aRoot: RawUtf8;
      aKorBr, aMagBr, aDrzBr, aLoyArtPay: integer;
      aMapPay, aMapShip: ThkMapDynArray); reintroduce; overload;
  end;

implementation

constructor TMyRestServer.CreateWithOwnModel(
  aModelServer: ThkOrmRestServerDB;
  aProps: TSqlDBConnectionPropertiesThreadSafe;
  aShApiConfig: ThkConfigSh; const aRoot: RawUtf8;
  aKorBr, aMagBr, aDrzBr, aLoyArtPay: integer;
  aMapPay, aMapShip: ThkMapDynArray);
begin
  inherited CreateWithOwnModel([], false, aRoot);
  ...
  OnBeforeURI:=BeforeURI;
end;

Correct trunk line numbers in mormot.rest.server :-), line 7544 node := fRouter.Lookup(ctxt);, node is nil and never reach line 7584 OnBeforeUri(ctxt).
Lookup line 5978 result := pointer(TRestTreeNode(fTree[Ctxt.Method].Root).Lookup(p, Ctxt));, result is nil
I workaround comment lines 7543..7548, just to build new version of my service and everithing is OK now. Just workaround.

#17 Re: mORMot 2 » Difference in URL parsing between mORMot1 and 2 » 2024-01-18 00:56:56

Migrate mormot2 to trunk and my production service fail.
I have TMyRestServer = class(TRestServerFullMemory) , without interface or published method, only OnBeforeUri.
OnBeforeUri never call!
procedure TRestServer.Uri(var Call: TRestUriParams) return Error in line 7553  // 4. decode..
never reach //6.  line 7589 OnBeforeUri
Older version of mormot2 from beginning of 2023 work OK.

#18 Re: mORMot 2 » High-Performance Frameworks » 2023-08-25 11:11:05

Not expected. What about "-s 8 -t 1" vs "-s 8 -t 2", both use same number of dbpull connections 8, -t1 (10K req/sec, 25 threads) , -t2 (7K req/sec, 33 threads). For -t 1 look like R0 make db connections?

#19 Re: mORMot 2 » High-Performance Frameworks » 2023-08-25 08:41:25

@ab, something is wrong with asyncserver or dbpull!
Running /rawdb wrk test (/db,/asyncdb also) ./raw -s 8 -t 8, total threads 64, I expect 64 connection to postgresql, but only 56 found.
DBPull conections are servers*(threads-1), not servers*threads!
Running ./raw -s 8 -t 4, only 24 connections, not 32
You can check active statements/connection with:
SELECT * from pg_stat_activity WHERE query='select id,randomNumber from World where id=$1'
When raw start we have 8 (servers) connections from main thread to load cache. Running wrk create servers*(threads-1) connections.
Main thread connections + threads connection = servers*threads, but worker threads use only s*(t-1).
Source /ex/techempower-bench, pull 2 days ago, 23.08.2024
raw -s 8 -t 8, htop show 81 threads.

#20 Re: mORMot 2 » mORMot2 + JavaScript client » 2023-08-24 08:19:05

> I think so, it has to include offline mode (I'll try that via service workers) and some processing on client side.
In that case you must build JS application using JS,TS,pas2js or DWScript and use mormot as API rest server.

> Thanks, I've heard about htmx before but haven't tried it yet.
> Do you create a method based functions on mORMot side that you call from htmx?
No I don't use method based functions, only mormot MVC app. htmx expect HTML response, not json and replace only part of HTML in your browser. Browser don't need to parse json data and create/modify DOM, just replace/add/delete some part of HTML in DOM. Easy way to transform SSR app to SPA app.

#21 Re: mORMot 2 » mORMot2 + JavaScript client » 2023-08-23 14:22:17

Yes @ab, almost no change in mormot mvc application.
Some hint, look for HX-Request in header, it is needed if user refresh page.
Just add this code in the end of procedure GetViewInfo, in mvc example.

  if context.Request.InHeader['HX-Request']<>'' then
  begin
    _ObjAddProps(['Htmx', _Obj([
        'HX-Request', context.Request.InHeader['HX-Request'],
        'HX-Trigger', context.Request.InHeader['HX-Trigger'],
        'HX-Trigger-Name', context.Request.InHeader['HX-Trigger-Name'],
        'HX-Target', context.Request.InHeader['HX-Target'],
        'HX-Prompt', context.Request.InHeader['HX-Prompt']
       ])], info);
  end;

Then in template you can use

{{^main.Htmx}} {{>header}} {{>masthead}} {{/main.Htmx}}
<!-- your page -->
{{^main.Htmx}} {{>footer}} {{/main.Htmx}}

#22 Re: mORMot 2 » mORMot2 + JavaScript client » 2023-08-23 07:36:24

Depend of type of your web app. Do you really need JS?
I build some internal projects with mormot mvc + mustache + htmx + Bootstrap5 and I will recommend to try htmx ( https://htmx.org/ ).

#24 Re: mORMot 2 » High-Performance Frameworks » 2023-04-14 12:19:10

@mvp, nice gist. Just some comment about concurrency, maybe add as param to perform, for all db test to use 512  (will have impact on active connections to postgers), for plaintext to use 1k or 4k, the best concurrency for each test.

#25 Re: mORMot 2 » High-Performance Frameworks » 2023-03-14 15:49:09

ab wrote:

@ab
Do you mean than 228 threads is slower than less threads?

Yes looking in the pgbench test table for 96Cpu server, best result when connections=cpu, for read only. We are testing postgres server and depend on I/O shared resource speed (hdd), the bigger bottleneck, too many threads, too many connections drop performance. Write/update connections have more stress to IO.
@mvp, different HW, hdd/ssd/raidX, maybe tfb HW. I know we miss async client. On slow hdd threads count is more important as in my old server.

#26 Re: mORMot 2 » High-Performance Frameworks » 2023-03-14 14:16:18

Great results for /json /plaintext.
Looking again on read only connection table https://techcommunity.microsoft.com/t5/ … 06462#fn:1
for 96vCpu compare 100 and 800 connections.
I don't find similar table for write/update connections, but for sure impact will be much more worse for shared I/O (hdd/ssd).
Maybe you can create one more test (docker) with same code but reduced threads/connections (2*CPU) and in one round see differences.

#27 Re: mORMot 2 » High-Performance Frameworks » 2023-03-01 23:54:49

Playing with mormot2 master from today. Best results on 24 vCpu are 2 servers with 12 threads and include flags hsoThreadSmooting and hsoReusePort. Some strange results when run docker -p 8080:8080 vs --net=host, compared with ntex. ntex run faster with -p, mormot with --net=host. Also result running raw native on host without docker. mormot is faster then ntex if run with --net!!!

                        T1       T2       T3       AVG
Native raw 12 24 2      196.817  196.177  196.558  196.517
docker run -p 8080:8080
raw 12 24 2             132.468  123.207  126.877  127.517
ntex                    144.824  148.111  146.022  146.319
docker run --net=host
raw 12 24 2             186.369  188.466  186.378  187.071
ntex                    137.510  136.848  136.738  137.032

#28 Re: mORMot 2 » High-Performance Frameworks » 2023-02-28 10:09:06

Just checked /json on Ubuntu 18.04, Kernel 4.15 as host for docker, previous tests are on U22.04 VM, to be sure thаt different kernel, glibc/MM version have any impact.
Small improvements on all tests, also for ntex. The Ubuntu version does not affect the test!

        Workers  Ser. Thr. T1       T2       T3       AVG
libc MM   48      2   24   121.680  126.686  124.550  124.305
mimalloc  48      2   24   125.354  120.953  127.821  124.709
ntex [tokio,platform]      144.824  148.111  146.022  146.319

#29 Re: mORMot 2 » High-Performance Frameworks » 2023-02-27 10:42:34

Just tried Microsoft mimalloc MM. https://github.com/microsoft/mimalloc
No noticeable improvements, same results as libc MM. ldd /usr/local/bin/raw confirm libmimalloc.so is used.
I used alpine docker https://github.com/emerzon/alpine-mimalloc.
Dockerfile

# ... Same builder

FROM emerzon/alpine-mimalloc
COPY --from=builder /build/bin/fpc-x86_64-linux/raw /usr/local/bin/raw

RUN apk --update add postgresql-client && \
    # Workaround musl vs glibc entrypoint
    mkdir /lib64 && \
    ln -s /lib/ld-musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

EXPOSE 8080
CMD ["raw"]

#30 Re: mORMot 2 » Serializing an array of records How To? » 2023-02-23 11:23:54

If you use SQLite and store as RawJSON you can use JSON functions and operators.
https://www.sqlite.org/json1.html

#31 Re: mORMot 2 » Serializing an array of records How To? » 2023-02-22 14:10:01

This json is not array! You need record

TMyData = packed record
  currentYear: integer;
  ActualWeek: integer;
  weeks: TWeekArray;
end;

Then you can use RecordLoadJson/RecordSaveJson from mormot.core.json.pas

#32 Re: mORMot 2 » Serializing an array of records How To? » 2023-02-22 09:47:08

If TWeek = class, not record, then name array TWeekObjArray = array of TWeek. Then you can use
Rtti.RegisterObjArray(T,Tweek);
In your case TWeek is packed record, just use:
Rtti.RegisterType(TypeInfo(TWeek));
Rtti.RegisterType(TypeInfo(TWeekArray));
If you need compatibility with older Delphi versions and Fpc then you can use
Rtti.RegisterFromText(TypeInfo(TWeek), __TWeek);
where __TWeek is const string with definition of record, read documentation for details.

Edited :-), Sorry typo TRtti->Rtti

#33 Re: mORMot 2 » High-Performance Frameworks » 2023-02-21 10:01:04

Wow, /plaintext results are impressive. Idea of 3 docker/test for json/plaintext was to tune workers to 2*CPU. Differences are small, in range of statistical errors.

wrk -d 15 -c 256 --timeout 8 -t 7 "http://localhost:8080/json"                         
Workers  Servers  Threads  T1       T2       T3       AVG
24       4         6       119.802  123.277  122.998  122.026
48       4        12       117.842  125.779  126.378  123.333
72       4        18       121.465  121.547  119.914  120.975
96       4        24       116.398  116.500  120.684  117.860
ntex [tokio,platform]      139.886  138.929  141.283  140.033

#34 Re: mORMot 2 » High-Performance Frameworks » 2023-02-21 01:10:32

I expect more difference playing with servers and threads for pipeline /plaintext.
Tested on old Dell R710, 2xE5645 (6 cores, 12 threads), XCP-NG(XenServer), VM Ubuntu 22.04 with 24 vCPU.
@mvp nice catch about hsoThreadSmooting, I delete   
if servers = 1 then
    include(flags, hsoThreadSmooting); // 30% better /plaintext e.g. on i5 7300U
raw running in docker, also ntex [tokio,platform] docker (in last tfb round ~7M req/s)
mormot raw have better results then ntex on this platform!!!
wrk is run on same VM with -t 6, letter found -t 7 have better results. 3 test without warming.
About docker, we use ubuntu 20.04 focal for builder and run on ubuntu 22.04. If TFB run docker on Ubuntu 18, maybe is better to use ubuntu 18. I will try same docker test on U18VM.

wrk -d 15 -c 256 --timeout 8 -t 6 "http://localhost:8080/plaintext" -s pipeline.lua -- 16

Workers Servers Threads T1         T2         T3         AVG
24      1       24      1,342,597  1,431,568  1,408,269  1,394,145
24      2       12      1,372,775  1,395,293  1,399,634  1,389,234
24      3       8       1,402,108  1,399,117  1,387,009  1,396,078
24      4       6       1,424,154  1,419,472  1,389,615  1,411,080
48      1       48      1,292,962  1,341,772  1,278,475  1,304,403
48      2       24      1,391,753  1,380,388  1,384,175  1,385,439
48      3       16      1,404,478  1,417,240  1,403,237  1,408,318
48      4       12      1,375,386  1,384,120  1,383,750  1,381,085
48      6       8       1,374,656  1,410,551  1,378,311  1,387,839
72      1       72      1,268,166  1,018,549    966,550  1,084,422
72      2       36      1,372,683  1,338,490  1,357,713  1,356,295
72      3       24      1,384,253  1,376,085  1,362,731  1,374,356
72      4       18      1,363,745  1,400,351  1,360,997  1,375,031
72      6       12      1,379,854  1,401,052  1,376,467  1,385,791
96      1       96        918,802    972,689    946,801    946,098
96      2       48      1,282,146  1,329,924  1,341,937  1,318,003
96      3       32      1,356,407  1,358,175  1,379,885  1,364,822
96      4       24      1,355,498  1,387,323  1,361,684  1,368,168
96      6       16      1,379,136  1,367,597  1,393,248  1,379,994
ntex [tokio,platform]   1,042,539  1,014,515  1,042,050  1,033,035

#35 Re: mORMot 2 » High-Performance Frameworks » 2023-02-18 12:29:04

Correct me if I'm wrong, looking at python source look like benchmark start postgres docker, start app docker, then run all tests define in benchmark_config.json. If this is true then any prev test have impact to next test, mostly leaving MM garbage/fragmentation behind, also posgres is not restarted for orm/raw tests.
Looking at benchmark_config.json from ntex (No1 in compose score from prev round), also asp.net.core, instead of using one docker/monolith app, I propose to create 3 docker/app, 1-json/plaintext, 2-orm, 3-raw. Compose score will aggregate best results from all 3 app. Also for json/plaintext you can use different No of threads then db tests, what is scale better.

#36 Re: mORMot 2 » mvc-blog using Postgres - external content FTS4/5 table exception » 2023-02-16 12:01:54

Look at MVCModel.pas for {$ifdef/ifndef  USEFIREBIRD
External database can't use FTS feature of SQLite, TOrmArticleSearch is not defined for USEFIREBIRD.

#37 Re: mORMot 2 » High-Performance Frameworks » 2023-02-15 10:25:55

Great results! Thanks a lot @ab and @mvp for your hard work.
When you finish testing all new improvements, push one round with 28 threads. I expect best performance for /json and /plaintext.
Nice blog post for libreactor https://talawah.io/blog/extreme-http-pe … o-million/

#38 Re: mORMot 1 » High-performance frameworks » 2022-12-07 16:44:23

ab wrote:

It is better with the thread affinity

Just note, prev test use threads=CPU*5,last *4. We cant compare results

#39 Re: mORMot 1 » High-performance frameworks » 2022-11-29 11:20:26

@mvp, can you push to TFB version with threads=CPU cores, to see performance for json and plaintext (to test http server performance without db). I expect better performance with db also. Looking at https://techcommunity.microsoft.com/t5/ … 06462#fn:1
test on 96 CPU core system, postgres have best perf at 100 clients 1,538,186 tps (100%), x4 400 clients 1,305,039 tps (85%), x5 500 clients 1,390,359 tps (90%).

#40 Re: mORMot 1 » High-performance frameworks » 2022-11-01 09:09:30

Testing on same HW will produce false thread/workers optimization. We need 2 servers (app and db) with same cpu/cores.
With this bench we also test posgresql. Looking at
https://pganalyze.com/blog/postgres-14- … monitoring
we can see that best TPS is when # connection=cpu cores (96 vCPU)
Looking at drogon source config.json https://github.com/TechEmpower/Framewor … onfig.json
app.threads_num: 0, mean worker threads=cpu cores.
Looking at actix-http source, no change of number of worker threads, actix default # is cpu cores.
For production server worker threads is Ok to be (1-4)*cpu cores, for this type of benchmark I will try with # cpu cores.

#41 Re: mORMot 1 » High-performance frameworks » 2022-10-30 10:06:20

Doing the same before @mvp post :-), rank/concurrency Fortunes test data.
No improvements between 256 and 512

Framework	16	32	64	128	256	512
Mormot-raw	10	9	13	20	35	38
Mormot-orm	33	32	30	47	54	51

#42 Re: mORMot 1 » High-performance frameworks » 2022-10-27 10:17:40

Test is finally running. TechEmpower have a lot of problems and stop tests, change HW and move office.
Thanks in advance to Pavel and Arnaud. Waiting for results...
Pavel I hope you are OK, I'm really admire to you and your work in this terrible time for you and your family.

#43 Re: mORMot 1 » TDocVariant: text encoding » 2022-08-31 18:14:15

@MC, Delphi 7, can you try to change StringToUTF8 with UTF8Encode and UTF8ToString with UTF8Decode, don't have D7 to test. Also check your .pas file encoding, is it Utf8 or Ansi format.

#44 Re: mORMot 1 » Enhanced URI routing for method-based services » 2022-05-14 12:29:01

edwinsn wrote:

If what you are proposing is possible with mORMot, when we can write a 'load balancer' with mORMot? big_smile

Yes why not  big_smile
THttpGatewayServer as descended, with methods or interface for remote AddRoute, DeleteRoute
In case of

HttpServer.AddRoute('$/$/overview', 'http://dynamicserver1.net/restroot/overview?lng=$&product=$');
HttpServer.AddRoute('$/$/overview', 'http://dynamicserver2.net/restroot/overview?lng=$&product=$');
HttpServer.AddRoute('$/$/overview', 'http://dynamicserver3.net/restroot/overview?lng=$&product=$');

you will have load balancer big_smile
Very useful in case of bugfix update/release, Just run new bugfix server and shutdown buggy server
Or dynamically update version, if you running AddRoute('/api/v1/func1...')
Just run new release with AddRoute('/api/v2/func1...'), update all clients to v2 and shutdown v1 server
Nice feature will be integrated MVC admin panel where you can monitor and dynamically add or delete working servers

THttpGatewayClientServer(THttpServer)
Just configure GW URI and server will register/AddRoute  all routes on start. On shutdown DeleteRoutes.
Shutdown can be invoked remotely by GW Server

Only sky is the limit big_smile

#45 Re: mORMot 1 » Enhanced URI routing for method-based services » 2022-05-13 22:11:11

@ab,
Excellent idea, it make sense to me for MVC application where user see URL.

Another idea

HttpServer.Route('$/$/overview', 'http://dynamicserver1.net/restroot/overview?lng=$&product=$');

Converting httpserver to proxy/gateway where dynamicserver1.net can dynamically add route to httpserver on start.
I know this is more complex change. Just idea.

#46 Re: mORMot 1 » MVC URI routing » 2022-05-11 09:32:17

Not possible in interface based service.
Workaround will be to create method based service, manually parse URI after root and call interface in method implementation.

#47 Re: mORMot 1 » Benchmark PostgreSQL, SQLite and mORMot » 2022-05-07 18:28:32

@tihory, I play with your gist bench, add SQLite3 and FirebirdSQL server and BatchAdd in client.

SQLite3 DB ORM :	Records Per Second : 17623
SQLite3 DB ORM :	Batch Records Per Second : 509113
PostgreSQL DB ORM :	Records Per Second : 1295
PostgreSQL DB ORM :	Batch Records Per Second : 108610
FirebirdSQL DB ORM :	Records Per Second : 2605
FirebirdSQL DB ORM :	Batch Records Per Second : 37846

This is localhost test on my 3-4 year old laptop with i3-8145U with 2 cores 4 HT.
On server you can play with PQ and FB defines
On client you have BatchAdd define to switch client mode
With your hardware you can expect much better req/sec
Batch Inserts is not a silver bullet for every case/project.
Without Batch all clients sends 1000 http req. and most of the time is spent in http parsing, object serialization/deserialization, ORM mapping and small time in real DB insert.
http and ORM have some overhead in this test, but this is much closer to real server app.
My changes in gist https://gist.github.com/TTomas/619173ca … 3c9c290cd7

#49 Re: mORMot 1 » Benchmark PostgreSQL, SQLite and mORMot » 2022-05-04 19:07:16

@tihory you don't publish results row/sec
With small changes you can change PQ to SQLite3 ORM to compare results.
You don't stress database enough with this micro bench. You see only 1-2 connections with insert in a time.
Try to increase client threads to some real numbers ex.100. Try with 10 worker threads and 100 clients threads.
It will be nice if you have 1 client server, 1 app server and 1 PQ server in LAN, real HW servers.
You can add in Client

    writeln('Start Client: ', Index); //Add
    for I := Index * PerConnection to (Index + 1) * PerConnection - 1 do
    begin
      Http.Add(Texts[I], True);
    end;
    writeln('End Client: ', Index);  //Add

Just to check that all clients run in parallel.

Edited
Try to create separate client and server bench application. Too many threads in 1 process will stress memory manager.

#50 Re: mORMot 1 » Benchmark PostgreSQL, SQLite and mORMot » 2022-05-03 13:11:13

@tihory, your test benchmark is not complete. You don't test PQ ORM multithreaded inserts. Try to add http server in test TRestHttpServer. TRestHttpServer.Create have aThreadPoolCount param default to 32 worker threads with own PQ connection pool.
After PQProps := TSqlDBPostgresConnectionProperties.Create('127.0.0.1', 'mormot_postgres_test', 'root', '1');
Set
PQProps.ThreadingMode := tmThreadPool;
If you use
PQProps.ThreadingMode := tmMainConnection;
you will have only 1 db connection and 32 working threads will use this one connection with Lock/UnLock.
You will need multiple TRestHttpClient's in multiple threads for test.

Board footer

Powered by FluxBB