fpc memorymanager > cmem seems to be faster

danielkuettner · 2020-03-29 10:01:57

I've looked for FastMM and fpc an comes to using UBLOCKS and cmem.

Compiling with USEBLOCKS had no effect, but using of cmem works noticeable better (faster).

My requests get better responds but the most interesting result cames from a larger statistic call, where I get a huge response from MongoDB. Without cmem I got an timeout from the MongoDB call. With cmem it works as under Windows/Delphi 64bit.

Last edited by danielkuettner (2020-03-29 10:43:54)

ab · 2020-03-29 10:56:29

We tried most alternatives, but went back to FPC heap manager.
They all consumed much more memory, and were not typically faster for our root services: mORMot tends to avoid memory allocation.

But we reached instability issues with other MM.
If your code has the slightest memory problem, most C heaps will kill the process! Whereas FPC would let an Access Violation be raised.
Even with valgrid and FullDebugMode validation, it is hard to have 100% memory safe server code. And killing a server process is definitively not a good idea.
So I would stay with

I tried to adapt FastMM4 with Linux.
Check the version in https://synopse.info/fossil/finfo?name=RTL7/FastMM4.pas
But we detected issues when sizing-up a memory block - so it was not usable.

Last but not least, never get a huge response from any DB.
The faster memory allocation is the one which is not allocated.
Use proper paging in your requests, or reduce/filter the content, or duplicate your data into another table/collection, with only the information needed. Data sharding is a good rule for scaling. Don't hesitate to duplicate data, and have prepared information standing by. For instance, don't use your main DB to make reports, but create report databases (and associated microservices) with their own tuned data. Storage is cheaper than CPU and bandwidth.

danielkuettner · 2020-03-29 11:17:18

Thanks for your information and results. I will try your FastMM4 and give you feedback.

Just a few remarks to your statements:

In my use cases there is more than enough server memory. The cpu cores are rather scarce. Therefore I consider the client CPU's like a shared system. The server delivers the data without special preparation, the clients prepare the delivered data as needed (sorting, aggregating ...).

In statistical pov the user gets only one time all needed data of the period of analysis and all the other work is done at client side (js). This gives the user a impression of faster responds, because data will only prepared at client side without request.

The second use case of getting large data from DB is for exporting data into second DB's (as you suggested) e.g. MS PowerBI.

danielkuettner · 2020-03-29 12:18:29

After starting with FastMM4:

Runtime error 216 at $0000000000428845
$0000000000428845
$000000000042484C

danielkuettner · 2020-03-30 20:49:00

@ab
Did you know this:

http://free-pascal-general.1045716.n5.n … 09801.html

? From 2005?!

ab · 2020-03-30 22:01:05

The FPC MM has been rewritten since, IIRC.

danielkuettner · 2020-03-31 16:00:02

I would test FastMM4 and give you feedback. Compilation works, but after starting comes:

Runtime error 216 at $000000000042E335
$000000000042E335 GETCPUID
$0000000000438109 RUNINITIALIZATIONCODE, line 17342 of ../../mORMot/trunk/RTL7/FastMM4.pas
$0000000000438135 FASTMM4_$$_init$, line 17362 of ../../mORMot/trunk/RTL7/FastMM4.pas
$0000000000418FA9
$000000000042496C

mORMot Open Source

#1 2020-03-29 10:01:57

fpc memorymanager > cmem seems to be faster

#2 2020-03-29 10:56:29

Re: fpc memorymanager > cmem seems to be faster

#3 2020-03-29 11:17:18

Re: fpc memorymanager > cmem seems to be faster

#4 2020-03-29 12:18:29

Re: fpc memorymanager > cmem seems to be faster

#5 2020-03-30 20:49:00

Re: fpc memorymanager > cmem seems to be faster

#6 2020-03-30 22:01:05

Re: fpc memorymanager > cmem seems to be faster

#7 2020-03-31 16:00:02

Re: fpc memorymanager > cmem seems to be faster

Board footer