You are not logged in.
- Topics: Active | Unanswered
#301 2024-05-29 14:17:32
Re: High-Performance Frameworks
The ByteScanIndex() is using Agner Fog's assembly code.
It reads at aligned 16 bytes, and ignore the out-of-range bytes.
This is a pretty valid scheme in practice, but I understand it does not please ValGrid.
Please try with
https://github.com/synopse/mORMot2/commit/ba84f2d0
But I don't know with TDYNARRAYHASHER_
Could you try without the hsoHeadersInterning option in the THttpAsyncServer?
This is the only place I could see use of TDynArrayHasher.
Or please try with https://github.com/synopse/mORMot2/commit/d6b92b1f
During my tests, I also discovered that shutdown may block if several servers are listening (noport).
I will investigate.
Offline
#302 2024-05-30 09:06:26
Re: High-Performance Frameworks
On May 29 commit
- with hsoHeadersInterning warnings remains the same
- without hsoHeadersInterning only one remain
==1532807== Conditional jump or move depends on uninitialised value(s)
==1532807== at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1532807== by 0x6BA141: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)About "shutdown may block" - from my experience it not block, but sometimes wait too long
Last edited by mpv (2024-05-30 09:07:19)
Offline
#303 2024-05-30 11:49:10
#304 2024-05-30 12:10:37
Re: High-Performance Frameworks
It is weird, because _BYTESCANINDEX should be fixed by allocating the memory buffer size + APPEND_OVERLOAD = 24 in TRawByteStringBuffer.RawAppend.
And our ByteScanIndex() has in fact the same logic as in FPC RTL IndexByte(). Just a bit more optimized, e.g. with proper opcode fusing. But it is the very same idea about aligned 16-bytes reads.
About hsoHeadersInterning, I will try to investigate a bit more but I did not see anything wrong with how TDynArrayHasher works.
Edit: I have added some hardcoded checks, and was NOT able to find anything wrong about indexes, as reported by valgrid:
function TDynArrayHasher.HashTableIndexToIndex(aHashTableIndex: PtrInt): PtrInt;
begin
if ((hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore)) * 2)) or
((not (hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore))))) then
RaiseFatalCollision('HashTableIndexToIndex', aHashTableIndex);
result := PtrUInt(fHashTableStore);
...Offline
#305 2024-06-01 13:34:49
Re: High-Performance Frameworks
I also do not see any problem there - may be this is false-positive warning. And I can`t reproduce crush on my environment 
On laet run (with mimalloc) failure message is
mimalloc: assertion failed: at "./src/alloc.c":231, mi_page_usable_size_of
assertion: "ok"assertion source - https://github.com/microsoft/mimalloc/b … loc.c#L231
P.S. Real assertion error is here - https://github.com/microsoft/mimalloc/b … loc.c#L223
P.P.S.
a test case where we fails (may be it helps to reproduce crash on your side):
wrk -H 'Host: 10.0.1.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 56 http://10.0.1.1:8080/plaintext -s pipeline.lua -- 16A pipeline.lua is here - https://github.com/TechEmpower/Framewor … peline.lua
Last edited by mpv (2024-06-01 13:47:03)
Offline
#306 2024-06-01 19:43:54
Re: High-Performance Frameworks
Sounds like a canary overwrite. That is, some buffer overflow.
Pretty weird. Because in FPC heaptrc does not trigger it.
Edit 1:
I was not able to reproduce the problem either here.
Perhaps we may disable the msize() function in mormot.core.fcplibcmm, and let _MemSize() always return 0.
I have added the new FPC_LIBCMM_NOMSIZE conditional, to be used together with FPC_LIBCMM.
https://github.com/synopse/mORMot2/commit/360d7835
Edit 2:
We use -O4 to compile the executable and I doubt it is worth it.
I only always use and validate -O3 level, and optimize the pascal code to have the best generated asm at -O3.
From -O4 may come some instability, with potentially no performance benefit.
Edit 3:
But I may have an hint about what happens. My current guess is that the issue could be about interning the "Host" and "Accept" header values.
I will investigate.
Edit 3-followup: Found nothing after investigation. There is no specific memory allocation due to those additional headers.
Offline
#307 2024-06-03 14:56:33
Re: High-Performance Frameworks
Updated PR 9072:
- migrate to Ubuntu24.04;
- mORMot@2.2.7579;
- return to glibc MM;
- plaintext is run after cached-queries
- define FPC_LIBCMM_NOMSIZE to disable the malloc_usable_size() call on Linux
- use O3 compiler optimization level
Hope that FPC_LIBCMM_NOMSIZE or -O3 will help
Last edited by mpv (2024-06-03 14:57:08)
Offline
#308 2024-06-04 09:45:11
Re: High-Performance Frameworks
With the current round, we got very good results, but... /plaintext failed
https://www.techempower.com/benchmarks/ … =plaintext
We will see with your PR.
Offline
#309 2024-06-08 10:54:10
Re: High-Performance Frameworks
In previous run (2d run with mimalloc) where is new warnings and error
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd9040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8be1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mimalloc: error: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb689a040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8be1040
mormot: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd9040
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd1040
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68b2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68a2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68aa040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"Offline
#310 2024-06-17 16:35:30
Re: High-Performance Frameworks
@mpv
I guess I found the cause of our problem.
I was able to reproduce it with a Windows VM hosting a mORMot web server, and a Linux client running wrk requests.
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
https://github.com/synopse/mORMot2/commit/a0dda418
Offline
#311 2024-06-17 16:41:13
#312 2024-06-17 17:20:11
#313 2024-06-18 08:14:21
Offline
#314 2024-06-18 09:55:31
Re: High-Performance Frameworks
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
I have seen that sometimes, I believe. Will check it out and watch it. Could never reproduce it in any sensible way. Let you know, if it seems better.
Thanks!
Offline
#315 2024-06-19 04:09:53
Re: High-Performance Frameworks
Nope, I still sometimes get the following error:
0000000000B012CF " EXC EInvalidPointer {Message:"Invalid pointer operation"} [TRestHttpSrv 443psyprax THttp] at 7c0784 System.pas @DynArrayClear (37156)- usually when the browser tries to prefetch a petch, while the server is just starting. I'll keep trying to localize the problem for more information, but for now, I'm at a lost as to exactly why/when that happens.
Offline
#316 2024-06-19 07:38:51
Re: High-Performance Frameworks
DynArrayClear() is called by the system e.g. when a class containing a dynamic array is released, or when a dynamic array on stack is released.
If it happens at startup, it may be due to some initialization code, which is not thread-safe.
You may enable the stack frames in your project option, to have a larger stack trace.
Offline
#317 2024-07-02 13:56:13
#318 2024-07-04 15:00:01
#319 2024-07-13 14:59:32
#320 2024-07-17 07:00:32
#321 2024-07-17 07:23:31
#322 2024-09-16 07:24:58
- TPrami
- Member
- Registered: 2010-07-06
- Posts: 119
Re: High-Performance Frameworks
With no stack trace it is very difficult to investigate...
Any luck in finding the bug?
-Tee-
Offline
#323 2024-09-16 07:29:48
Re: High-Performance Frameworks
Since it was only in pipelined mode, which is never used on production, I honestly did not investigate much.
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
Offline
#324 2024-09-16 09:58:08
- TPrami
- Member
- Registered: 2010-07-06
- Posts: 119
Re: High-Performance Frameworks
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
That is worrying indeed, hope he is OK.
-Tee-
Offline
#325 2024-09-16 14:05:18
- dcoun
- Member
- From: Crete, Greece
- Registered: 2020-02-18
- Posts: 430
Re: High-Performance Frameworks
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
I believe he is ok. Probably busy with UnityBase
Offline
#326 2024-09-18 14:51:01
Re: High-Performance Frameworks
Guys, thanks, I'm fine. It's just that time flies very strangely at war...
I will try to find the cause of GPF by switching FTB to mormot MM - PR 9283
If this does not help, we will disable HTTP pipelining in the next round
Last edited by mpv (2024-09-18 14:51:41)
Offline
#327 2024-09-27 11:49:24
Re: High-Performance Frameworks
We have the result for TFB with FPC_X64MM - it still fails for /plaintext, it also fails for /cachedqueries (ORM) and very low results for /fortunes. We'll have the log in a few days - maybe we'll see something interesting there...
Offline
#328 2024-09-30 10:17:44
Re: High-Performance Frameworks
Got a log: with FPC_X64MM /plaintext endpoint crashes with malloc(): unaligned tcache chunk detected - so, IMHO, we have corrupted or overwritten the memory allocator's internal metadata - may be some code in mORMot with direct memory access write outside a buffer..
Offline
#329 2024-09-30 10:38:12
Re: High-Performance Frameworks
I don't understand because malloc(): unaligned tcache chunk detected is a malloc error, not a mormot.core.fpcx64mm error.
It sounds like if malloc is still used somewhere by the program.
The only place where I could see malloc still being used is in the PostgreSQL client library.
But it should not trash the /plaintext test, where PostgreSQL is not involved.
Offline
#330 2024-10-23 09:25:02
Re: High-Performance Frameworks
New PR 9350 to TFB with Mormot2@2.3.stable + CMem. Hope fixed TPollAsyncConnection locks structure commit solves mem problems
Offline
#331 2024-11-18 14:25:59
Re: High-Performance Frameworks
Because memory problems is not solved I tries *without* `hsoEnablePipelining` option, and got a strange result: when client query /plaintext in pipelining mode (see https://synopse.info/forum/viewtopic.ph … 769#p38769 for how to call) - server answer only to firs request in each thread. @ab - can you, please, look at this problem? Correct server behavior is "to respond to pipelined requests correctly, with non-pipelined but valid responses even if server does not support HTTP pipelining."
Offline
#332 2024-11-18 15:10:02
Re: High-Performance Frameworks
I suspect this behavior has always been the case with the async server: pipelined requests were never taken into account, but in explicit hsoEnablePipelining.
And it seems that the mem problem occurred also outside of the pipeline /plaintext request, right?
Offline
#333 2024-11-18 21:12:30
- danielkuettner
- Member
- From: Germany
- Registered: 2014-08-06
- Posts: 357
Re: High-Performance Frameworks
@ab Yes mem issue is outside of pipeline.
Today we had such an issue in our program (of course without pipeline):
SOneSrv2: malloc.c:4302: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed
We had no error in Postgres log at all. But we are using cmem.
Last edited by danielkuettner (2024-11-19 07:27:34)
Offline