You are not logged in.
The TFB program is compiled with FPC 3.2.3 so is not affected by those FPC trunk (temporary) issues - at least on my machine.
Inline function bug is affected in FPC 3.2
https://gitlab.com/freepascal.org/fpc/s … sues/41031
Offline
Hi ab, I'm the author of that inline function bug report, the use of absolute keyword has no relevance, you can replace it with type casting if you want (which is functionally the same), the issue remains.
I do agree that it is probably not the source of your plaintext issue but it's worth doing a default (sane) options test to avoid chasing dead ends.
Anyway I know it's a bit late but I wish everyone a good and productive year of 2025.
Offline
PR 9480 with mORMot@2.3.9262 2024-12-23 fail again. Round 23 starts in few days, PR accepted until Jan 17. @ab - do we have some changes what may fix our problem (I saw changes about closing connections) - should I do a new PR?
Offline
The whole connection locking has been rewritten and proven.
We could do a PR with the current state.
But I am not sure it is the culprit.
OR we could make a PR with an old version, to bisect the regression.
OR we could check for the FPC revision, or try with -O1, or try to remove -dFPC_LIBCMM_NOMSIZE.
OR we could remove {$define HASINLINE} in mormot.defines.inc to ensure it is not an inlining bug.
The problem is that I am not able to reproduce the issue outside of the TFB hardware.
Edit: I just spent again 2 hours trying to reproduce it using current trunk, without success.
Online
We have already tried without FPC_LIBCMM_NOMSIZE in one of the previous run. Since there is no time to experiment until the 23rd round, I will do PR with the current state.
After the 23rd round, we will continue to test other ideas ( -O1, no inline)
IMHO the only way to reproduce is to run in debug mode on powerful server... I also tried to reproduce on my dev. PC and on old Xeon MANY times without success
Offline
@ab
today I had a crash of my service. My log shows following entry:
20250117 09214012 C trace mormot.net.async.THttpAsyncConnections(7f48913675e0) DoGC #1=0 #2=8 free=0 client=3
that could be related to
the first exception
20250117 09214037 # EXCOS EAccessViolation (8ff6e188) [R1:root] at 41c680
which comes from an harmless mongodb call.
Is it possible that DoGC (I guess it means "Do garbage collector") is the culprit?
Last edited by danielkuettner (2025-01-17 11:36:47)
Offline
@daniel
Which MM are you using?
" DoGC #1=0 #2=8 free=0 client=3 " indicates that 8 instances are in the GC#2 generation, no instance was freed, and there are current 3 connections.
So I don't think DoGC did anything here.
Online
cmem
Offline
After running for days without errors, today there are ones again.
Although my service is still running, I have got an av in the log:
20250117 11262413 % EXC EOutOfMemory {Message:"Out of memory"} [R3:root] at 41c307
which comes from a call that will be called always and without issues at all. There is enough memory (45GB free of 100GB), so I guess cmem should find a free block.
I couldn't find anything interesting in log before that EXC. There is only this:
20250117 11262407 C trace mormot.net.async.THttpAsyncConnections(7f1eaaf795e0) DoGC #1=0 #2=13 free=0 client=0
But as you wrote before, nothing interesting.
Last edited by danielkuettner (2025-01-17 11:46:18)
Offline
I don't have a stack trace on my production system, just a log file (in verbose logging).
The mongdb call is a simple aggregate with a small result, called over a sicSingle service method. Not complex, just a very simple method.
I believe the root of the issue comes from our side, not from mORMot2. But I don't find any entries of interest in my logs.
The mongodb call methods are so simple, that there couldn't be any issue come from. I have 32 workers and every worker has its own mongodb connection in his own threadvar. No race condition, no blocking issue.
MongoDB calls are the main work in our service. If any worker connection would have an issue I would have a crash or errors instantly.
Atm I don't use your new rewrite version (just ef8764e2c).
To find out if async server is involved, should I switch to useHttpSocket and watch?
Last edited by danielkuettner (2025-01-17 12:18:34)
Offline
Try to either enable the debug information within the executable, or compile it as external, then generate the .mab file with our tool, and distribute the .map file with the executable.
You would have detailed stack trace during all exceptions.
It may help the debugging.
Switching to useHttpSocket may be a good idea, if you don't have too much connected clients at the same time (there will be one thread per client).
Online
I had added a new error in my service and that's the reason of the error accumulation from today. Sorry for that.
But it seems to exists an issue before my update, so I have posted here.
The conversation was nevertheless useless, becaus my service runs now in debug mode so I'm able to give you a map file next time the issue will occur.
Thanks for your help!
Offline