High-Performance Frameworks

ttomas · 2025-01-04 12:31:47

ab wrote:

The TFB program is compiled with FPC 3.2.3 so is not affected by those FPC trunk (temporary) issues - at least on my machine.

Inline function bug is affected in FPC 3.2
https://gitlab.com/freepascal.org/fpc/s … sues/41031

ab · 2025-01-04 17:55:38

Using absolute with result is very risky and never used in mormot.
I suspect this is not the cause of our stability issue..

pvn0 · 2025-01-04 19:04:01

Hi ab, I'm the author of that inline function bug report, the use of absolute keyword has no relevance, you can replace it with type casting if you want (which is functionally the same), the issue remains.
I do agree that it is probably not the source of your plaintext issue but it's worth doing a default (sane) options test to avoid chasing dead ends.
Anyway I know it's a bit late but I wish everyone a good and productive year of 2025.

mpv · 2025-01-14 10:25:23

PR 9480 with mORMot@2.3.9262 2024-12-23 fail again. Round 23 starts in few days, PR accepted until Jan 17. @ab - do we have some changes what may fix our problem (I saw changes about closing connections) - should I do a new PR?

ab · 2025-01-14 14:34:57

The whole connection locking has been rewritten and proven.
We could do a PR with the current state.
But I am not sure it is the culprit.

OR we could make a PR with an old version, to bisect the regression.
OR we could check for the FPC revision, or try with -O1, or try to remove -dFPC_LIBCMM_NOMSIZE.
OR we could remove {$define HASINLINE} in mormot.defines.inc to ensure it is not an inlining bug.

The problem is that I am not able to reproduce the issue outside of the TFB hardware.
Edit: I just spent again 2 hours trying to reproduce it using current trunk, without success.

mpv · 2025-01-15 11:27:54

We have already tried without FPC_LIBCMM_NOMSIZE in one of the previous run. Since there is no time to experiment until the 23rd round, I will do PR with the current state.
After the 23rd round, we will continue to test other ideas ( -O1, no inline)
IMHO the only way to reproduce is to run in debug mode on powerful server... I also tried to reproduce on my dev. PC and on old Xeon MANY times without success

mpv · 2025-01-15 12:35:13

9536 mORMot@2.3.9395 `net: async code cleaning`. We can't do any PR until the 23rd round is over (I think about a month), so I'm taking a break for now

danielkuettner · 2025-01-17 11:30:18

@ab
today I had a crash of my service. My log shows following entry:
20250117 09214012 C trace mormot.net.async.THttpAsyncConnections(7f48913675e0) DoGC #1=0 #2=8 free=0 client=3

that could be related to
the first exception
20250117 09214037 # EXCOS EAccessViolation (8ff6e188) [R1:root] at 41c680

which comes from an harmless mongodb call.

Is it possible that DoGC (I guess it means "Do garbage collector") is the culprit?

Last edited by danielkuettner (2025-01-17 11:36:47)

ab · 2025-01-17 11:31:29

@daniel
Which MM are you using?

" DoGC #1=0 #2=8 free=0 client=3 " indicates that 8 instances are in the GC#2 generation, no instance was freed, and there are current 3 connections.
So I don't think DoGC did anything here.

danielkuettner · 2025-01-17 11:37:12

cmem

danielkuettner · 2025-01-17 11:45:35

After running for days without errors, today there are ones again.
Although my service is still running, I have got an av in the log:

20250117 11262413 % EXC EOutOfMemory {Message:"Out of memory"} [R3:root] at 41c307

which comes from a call that will be called always and without issues at all. There is enough memory (45GB free of 100GB), so I guess cmem should find a free block.

I couldn't find anything interesting in log before that EXC. There is only this:

20250117 11262407 C trace mormot.net.async.THttpAsyncConnections(7f1eaaf795e0) DoGC #1=0 #2=13 free=0 client=0

But as you wrote before, nothing interesting.

Last edited by danielkuettner (2025-01-17 11:46:18)

ab · 2025-01-17 11:46:51

What is the harmless mongodb call in question?
Do you have the stack traces?

danielkuettner · 2025-01-17 11:59:20

I don't have a stack trace on my production system, just a log file (in verbose logging).

The mongdb call is a simple aggregate with a small result, called over a sicSingle service method. Not complex, just a very simple method.

I believe the root of the issue comes from our side, not from mORMot2. But I don't find any entries of interest in my logs.
The mongodb call methods are so simple, that there couldn't be any issue come from. I have 32 workers and every worker has its own mongodb connection in his own threadvar. No race condition, no blocking issue.
MongoDB calls are the main work in our service. If any worker connection would have an issue I would have a crash or errors instantly.

Atm I don't use your new rewrite version (just ef8764e2c).

To find out if async server is involved, should I switch to useHttpSocket and watch?

Last edited by danielkuettner (2025-01-17 12:18:34)

ab · 2025-01-17 12:43:09

Try to either enable the debug information within the executable, or compile it as external, then generate the .mab file with our tool, and distribute the .map file with the executable.
You would have detailed stack trace during all exceptions.
It may help the debugging.

Switching to useHttpSocket may be a good idea, if you don't have too much connected clients at the same time (there will be one thread per client).

danielkuettner · 2025-01-17 16:59:18

I had added a new error in my service and that's the reason of the error accumulation from today. Sorry for that.
But it seems to exists an issue before my update, so I have posted here.
The conversation was nevertheless useless, becaus my service runs now in debug mode so I'm able to give you a map file next time the issue will occur.
Thanks for your help!

ab · 2025-01-18 09:01:50

@Daniel
It is perhaps worth a new forum thread when you have some feedback.

mORMot Open Source

#351 2025-01-04 12:31:47

Re: High-Performance Frameworks

#352 2025-01-04 17:55:38

Re: High-Performance Frameworks

#353 2025-01-04 19:04:01

Re: High-Performance Frameworks

#354 2025-01-14 10:25:23

Re: High-Performance Frameworks

#355 2025-01-14 14:34:57

Re: High-Performance Frameworks

#356 2025-01-15 11:27:54

Re: High-Performance Frameworks

#357 2025-01-15 12:35:13

Re: High-Performance Frameworks

#358 2025-01-17 11:30:18

Re: High-Performance Frameworks

#359 2025-01-17 11:31:29

Re: High-Performance Frameworks

#360 2025-01-17 11:37:12

Re: High-Performance Frameworks

#361 2025-01-17 11:45:35

Re: High-Performance Frameworks

#362 2025-01-17 11:46:51

Re: High-Performance Frameworks

#363 2025-01-17 11:59:20

Re: High-Performance Frameworks

#364 2025-01-17 12:43:09

Re: High-Performance Frameworks

#365 2025-01-17 16:59:18

Re: High-Performance Frameworks

#366 2025-01-18 09:01:50

Re: High-Performance Frameworks

Board footer