You are not logged in.
The ByteScanIndex() is using Agner Fog's assembly code.
It reads at aligned 16 bytes, and ignore the out-of-range bytes.
This is a pretty valid scheme in practice, but I understand it does not please ValGrid.
Please try with
https://github.com/synopse/mORMot2/commit/ba84f2d0
But I don't know with TDYNARRAYHASHER_
Could you try without the hsoHeadersInterning option in the THttpAsyncServer?
This is the only place I could see use of TDynArrayHasher.
Or please try with https://github.com/synopse/mORMot2/commit/d6b92b1f
During my tests, I also discovered that shutdown may block if several servers are listening (noport).
I will investigate.
Offline
On May 29 commit
- with hsoHeadersInterning warnings remains the same
- without hsoHeadersInterning only one remain
==1532807== Conditional jump or move depends on uninitialised value(s)
==1532807== at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1532807== by 0x6BA141: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)
About "shutdown may block" - from my experience it not block, but sometimes wait too long
Last edited by mpv (2024-05-30 09:07:19)
Offline
It is weird, because _BYTESCANINDEX should be fixed by allocating the memory buffer size + APPEND_OVERLOAD = 24 in TRawByteStringBuffer.RawAppend.
And our ByteScanIndex() has in fact the same logic as in FPC RTL IndexByte(). Just a bit more optimized, e.g. with proper opcode fusing. But it is the very same idea about aligned 16-bytes reads.
About hsoHeadersInterning, I will try to investigate a bit more but I did not see anything wrong with how TDynArrayHasher works.
Edit: I have added some hardcoded checks, and was NOT able to find anything wrong about indexes, as reported by valgrid:
function TDynArrayHasher.HashTableIndexToIndex(aHashTableIndex: PtrInt): PtrInt;
begin
if ((hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore)) * 2)) or
((not (hash16bit in fState) and
(PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore))))) then
RaiseFatalCollision('HashTableIndexToIndex', aHashTableIndex);
result := PtrUInt(fHashTableStore);
...
Offline
I also do not see any problem there - may be this is false-positive warning. And I can`t reproduce crush on my environment
On laet run (with mimalloc) failure message is
mimalloc: assertion failed: at "./src/alloc.c":231, mi_page_usable_size_of
assertion: "ok"
assertion source - https://github.com/microsoft/mimalloc/b … loc.c#L231
P.S. Real assertion error is here - https://github.com/microsoft/mimalloc/b … loc.c#L223
P.P.S.
a test case where we fails (may be it helps to reproduce crash on your side):
wrk -H 'Host: 10.0.1.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 56 http://10.0.1.1:8080/plaintext -s pipeline.lua -- 16
A pipeline.lua is here - https://github.com/TechEmpower/Framewor … peline.lua
Last edited by mpv (2024-06-01 13:47:03)
Offline
Sounds like a canary overwrite. That is, some buffer overflow.
Pretty weird. Because in FPC heaptrc does not trigger it.
Edit 1:
I was not able to reproduce the problem either here.
Perhaps we may disable the msize() function in mormot.core.fcplibcmm, and let _MemSize() always return 0.
I have added the new FPC_LIBCMM_NOMSIZE conditional, to be used together with FPC_LIBCMM.
https://github.com/synopse/mORMot2/commit/360d7835
Edit 2:
We use -O4 to compile the executable and I doubt it is worth it.
I only always use and validate -O3 level, and optimize the pascal code to have the best generated asm at -O3.
From -O4 may come some instability, with potentially no performance benefit.
Edit 3:
But I may have an hint about what happens. My current guess is that the issue could be about interning the "Host" and "Accept" header values.
I will investigate.
Edit 3-followup: Found nothing after investigation. There is no specific memory allocation due to those additional headers.
Offline
Updated PR 9072:
- migrate to Ubuntu24.04;
- mORMot@2.2.7579;
- return to glibc MM;
- plaintext is run after cached-queries
- define FPC_LIBCMM_NOMSIZE to disable the malloc_usable_size() call on Linux
- use O3 compiler optimization level
Hope that FPC_LIBCMM_NOMSIZE or -O3 will help
Last edited by mpv (2024-06-03 14:57:08)
Offline
With the current round, we got very good results, but... /plaintext failed
https://www.techempower.com/benchmarks/ … =plaintext
We will see with your PR.
Offline
In previous run (2d run with mimalloc) where is new warnings and error
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd9040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8be1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mimalloc: error: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb689a040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8be1040
mormot: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd9040
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd1040
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68b2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68a2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68aa040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot: assertion: "idx < segment->slice_entries"
Offline
@mpv
I guess I found the cause of our problem.
I was able to reproduce it with a Windows VM hosting a mORMot web server, and a Linux client running wrk requests.
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
https://github.com/synopse/mORMot2/commit/a0dda418
Offline
Offline
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
I have seen that sometimes, I believe. Will check it out and watch it. Could never reproduce it in any sensible way. Let you know, if it seems better.
Thanks!
Offline
Nope, I still sometimes get the following error:
0000000000B012CF " EXC EInvalidPointer {Message:"Invalid pointer operation"} [TRestHttpSrv 443psyprax THttp] at 7c0784 System.pas @DynArrayClear (37156)
- usually when the browser tries to prefetch a petch, while the server is just starting. I'll keep trying to localize the problem for more information, but for now, I'm at a lost as to exactly why/when that happens.
Offline
DynArrayClear() is called by the system e.g. when a class containing a dynamic array is released, or when a dynamic array on stack is released.
If it happens at startup, it may be due to some initialization code, which is not thread-safe.
You may enable the stack frames in your project option, to have a larger stack trace.
Offline
With no stack trace it is very difficult to investigate...
Any luck in finding the bug?
-Tee-
Offline
Since it was only in pipelined mode, which is never used on production, I honestly did not investigate much.
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
Offline
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
That is worrying indeed, hope he is OK.
-Tee-
Offline
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
I believe he is ok. Probably busy with UnityBase
Offline
Guys, thanks, I'm fine. It's just that time flies very strangely at war...
I will try to find the cause of GPF by switching FTB to mormot MM - PR 9283
If this does not help, we will disable HTTP pipelining in the next round
Last edited by mpv (2024-09-18 14:51:41)
Offline
We have the result for TFB with FPC_X64MM - it still fails for /plaintext, it also fails for /cachedqueries (ORM) and very low results for /fortunes. We'll have the log in a few days - maybe we'll see something interesting there...
Offline
Got a log: with FPC_X64MM /plaintext endpoint crashes with malloc(): unaligned tcache chunk detected - so, IMHO, we have corrupted or overwritten the memory allocator's internal metadata - may be some code in mORMot with direct memory access write outside a buffer..
Offline
I don't understand because malloc(): unaligned tcache chunk detected is a malloc error, not a mormot.core.fpcx64mm error.
It sounds like if malloc is still used somewhere by the program.
The only place where I could see malloc still being used is in the PostgreSQL client library.
But it should not trash the /plaintext test, where PostgreSQL is not involved.
Offline