You are not logged in.
- Topics: Active | Unanswered
#301 2024-05-29 14:17:32
Re: High-Performance Frameworks
The ByteScanIndex() is using Agner Fog's assembly code.
It reads at aligned 16 bytes, and ignore the out-of-range bytes.
This is a pretty valid scheme in practice, but I understand it does not please ValGrid.
Please try with
https://github.com/synopse/mORMot2/commit/ba84f2d0
But I don't know with TDYNARRAYHASHER_
Could you try without the hsoHeadersInterning option in the THttpAsyncServer?
This is the only place I could see use of TDynArrayHasher.
Or please try with https://github.com/synopse/mORMot2/commit/d6b92b1f
During my tests, I also discovered that shutdown may block if several servers are listening (noport).
I will investigate.
Offline
#302 2024-05-30 09:06:26
Re: High-Performance Frameworks
On May 29 commit
- with hsoHeadersInterning warnings remains the same
- without hsoHeadersInterning only one remain
==1532807== Conditional jump or move depends on uninitialised value(s)
==1532807== at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1532807== by 0x6BA141: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)About "shutdown may block" - from my experience it not block, but sometimes wait too long
Last edited by mpv (2024-05-30 09:07:19)
Offline
#303 2024-05-30 11:49:10
#304 2024-05-30 12:10:37
Re: High-Performance Frameworks
It is weird, because _BYTESCANINDEX should be fixed by allocating the memory buffer size + APPEND_OVERLOAD = 24 in TRawByteStringBuffer.RawAppend.
And our ByteScanIndex() has in fact the same logic as in FPC RTL IndexByte(). Just a bit more optimized, e.g. with proper opcode fusing. But it is the very same idea about aligned 16-bytes reads.
About hsoHeadersInterning, I will try to investigate a bit more but I did not see anything wrong with how TDynArrayHasher works.
Edit: I have added some hardcoded checks, and was NOT able to find anything wrong about indexes, as reported by valgrid:
function TDynArrayHasher.HashTableIndexToIndex(aHashTableIndex: PtrInt): PtrInt;
begin
if ((hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore)) * 2)) or
((not (hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore))))) then
RaiseFatalCollision('HashTableIndexToIndex', aHashTableIndex);
result := PtrUInt(fHashTableStore);
...Offline
#305 2024-06-01 13:34:49
Re: High-Performance Frameworks
I also do not see any problem there - may be this is false-positive warning. And I can`t reproduce crush on my environment 
On laet run (with mimalloc) failure message is
mimalloc: assertion failed: at "./src/alloc.c":231, mi_page_usable_size_of
assertion: "ok"assertion source - https://github.com/microsoft/mimalloc/b … loc.c#L231
P.S. Real assertion error is here - https://github.com/microsoft/mimalloc/b … loc.c#L223
P.P.S.
a test case where we fails (may be it helps to reproduce crash on your side):
wrk -H 'Host: 10.0.1.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 56 http://10.0.1.1:8080/plaintext -s pipeline.lua -- 16A pipeline.lua is here - https://github.com/TechEmpower/Framewor … peline.lua
Last edited by mpv (2024-06-01 13:47:03)
Offline
#306 2024-06-01 19:43:54
Re: High-Performance Frameworks
Sounds like a canary overwrite. That is, some buffer overflow.
Pretty weird. Because in FPC heaptrc does not trigger it.
Edit 1:
I was not able to reproduce the problem either here.
Perhaps we may disable the msize() function in mormot.core.fcplibcmm, and let _MemSize() always return 0.
I have added the new FPC_LIBCMM_NOMSIZE conditional, to be used together with FPC_LIBCMM.
https://github.com/synopse/mORMot2/commit/360d7835
Edit 2:
We use -O4 to compile the executable and I doubt it is worth it.
I only always use and validate -O3 level, and optimize the pascal code to have the best generated asm at -O3.
From -O4 may come some instability, with potentially no performance benefit.
Edit 3:
But I may have an hint about what happens. My current guess is that the issue could be about interning the "Host" and "Accept" header values.
I will investigate.
Edit 3-followup: Found nothing after investigation. There is no specific memory allocation due to those additional headers.
Offline
#307 2024-06-03 14:56:33
Re: High-Performance Frameworks
Updated PR 9072:
- migrate to Ubuntu24.04;
- mORMot@2.2.7579;
- return to glibc MM;
- plaintext is run after cached-queries
- define FPC_LIBCMM_NOMSIZE to disable the malloc_usable_size() call on Linux
- use O3 compiler optimization level
Hope that FPC_LIBCMM_NOMSIZE or -O3 will help
Last edited by mpv (2024-06-03 14:57:08)
Offline
#308 2024-06-04 09:45:11
Re: High-Performance Frameworks
With the current round, we got very good results, but... /plaintext failed
https://www.techempower.com/benchmarks/ … =plaintext
We will see with your PR.
Offline
#309 2024-06-08 10:54:10
Re: High-Performance Frameworks
In previous run (2d run with mimalloc) where is new warnings and error
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd9040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8be1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mimalloc: error: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb689a040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8be1040
mormot: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd9040
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd1040
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68b2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68a2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68aa040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"Offline
#310 2024-06-17 16:35:30
Re: High-Performance Frameworks
@mpv
I guess I found the cause of our problem.
I was able to reproduce it with a Windows VM hosting a mORMot web server, and a Linux client running wrk requests.
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
https://github.com/synopse/mORMot2/commit/a0dda418
Offline
#311 2024-06-17 16:41:13
#312 2024-06-17 17:20:11
#313 2024-06-18 08:14:21
Offline
#314 2024-06-18 09:55:31
Re: High-Performance Frameworks
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
I have seen that sometimes, I believe. Will check it out and watch it. Could never reproduce it in any sensible way. Let you know, if it seems better.
Thanks!
Offline
#315 2024-06-19 04:09:53
Re: High-Performance Frameworks
Nope, I still sometimes get the following error:
0000000000B012CF " EXC EInvalidPointer {Message:"Invalid pointer operation"} [TRestHttpSrv 443psyprax THttp] at 7c0784 System.pas @DynArrayClear (37156)- usually when the browser tries to prefetch a petch, while the server is just starting. I'll keep trying to localize the problem for more information, but for now, I'm at a lost as to exactly why/when that happens.
Offline
#316 2024-06-19 07:38:51
Re: High-Performance Frameworks
DynArrayClear() is called by the system e.g. when a class containing a dynamic array is released, or when a dynamic array on stack is released.
If it happens at startup, it may be due to some initialization code, which is not thread-safe.
You may enable the stack frames in your project option, to have a larger stack trace.
Offline