#301 2024-05-29 14:17:32

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

The ByteScanIndex() is using Agner Fog's assembly code.
It reads at aligned 16 bytes, and ignore the out-of-range bytes.
This is a pretty valid scheme in practice, but I understand it does not please ValGrid.

Please try with
https://github.com/synopse/mORMot2/commit/ba84f2d0

But I don't know with TDYNARRAYHASHER_
Could you try without the hsoHeadersInterning option in the THttpAsyncServer?
This is the only place I could see use of TDynArrayHasher.
Or please try with https://github.com/synopse/mORMot2/commit/d6b92b1f

During my tests, I also discovered that shutdown may block if several servers are listening (noport).
I will investigate.

Offline

#302 2024-05-30 09:06:26

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

On May 29 commit
- with hsoHeadersInterning warnings remains the same
- without hsoHeadersInterning only one remain

==1532807== Conditional jump or move depends on uninitialised value(s)
==1532807==    at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1532807==    by 0x6BA141: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)

About "shutdown may block" - from my experience it not block, but sometimes wait too long

Last edited by mpv (2024-05-30 09:07:19)

Offline

#303 2024-05-30 11:49:10

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

Made a PR 9072:
  -  migrate to Ubuntu24.04;
  -  mORMot@2.2.7565;
  -  return to glibc MM;
  -  plaintext is run after cached-queries

Offline

#304 2024-05-30 12:10:37

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

It is weird, because _BYTESCANINDEX should be fixed by allocating the memory buffer size + APPEND_OVERLOAD = 24 in TRawByteStringBuffer.RawAppend.
And our ByteScanIndex() has in fact the same logic as in FPC RTL IndexByte(). Just a bit more optimized, e.g. with proper opcode fusing. But it is the very same idea about aligned 16-bytes reads.

About hsoHeadersInterning, I will try to investigate a bit more but I did not see anything wrong with how TDynArrayHasher works.
Edit: I have added some hardcoded checks, and was NOT able to find anything wrong about indexes, as reported by valgrid:

function TDynArrayHasher.HashTableIndexToIndex(aHashTableIndex: PtrInt): PtrInt;
begin
  if ((hash16bit in fState) and
      (PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore)) * 2)) or
     ((not (hash16bit in fState) and
      (PtrUInt(aHashTableIndex) >= PtrUInt(length(fHashTableStore))))) then
      RaiseFatalCollision('HashTableIndexToIndex', aHashTableIndex);
  result := PtrUInt(fHashTableStore);
  ...

Offline

#305 2024-06-01 13:34:49

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

I also do not see any problem there - may be this is false-positive warning. And I can`t reproduce crush on my environment sad
On laet run (with mimalloc) failure message is

mimalloc: assertion failed: at "./src/alloc.c":231, mi_page_usable_size_of
assertion: "ok"

assertion source - https://github.com/microsoft/mimalloc/b … loc.c#L231

P.S. Real assertion error is here - https://github.com/microsoft/mimalloc/b … loc.c#L223

P.P.S.
a test case where we fails (may be it helps to reproduce crash on your side):

wrk -H 'Host: 10.0.1.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 56 http://10.0.1.1:8080/plaintext -s pipeline.lua -- 16

A pipeline.lua is here - https://github.com/TechEmpower/Framewor … peline.lua

Last edited by mpv (2024-06-01 13:47:03)

Offline

#306 2024-06-01 19:43:54

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

Sounds like a canary overwrite. That is, some buffer overflow.
Pretty weird. Because in FPC heaptrc does not trigger it.

Edit 1:
I was not able to reproduce the problem either here.
Perhaps we may disable the msize() function in mormot.core.fcplibcmm, and let _MemSize() always return 0.
I have added the new FPC_LIBCMM_NOMSIZE conditional, to be used together with FPC_LIBCMM.
https://github.com/synopse/mORMot2/commit/360d7835

Edit 2:
We use -O4 to compile the executable and I doubt it is worth it.
I only always use and validate -O3 level, and optimize the pascal code to have the best generated asm at -O3.
From -O4 may come some instability, with potentially no performance benefit.

Edit 3:
But I may have an hint about what happens. My current guess is that the issue could be about interning the "Host" and "Accept" header values.
I will investigate.
Edit 3-followup: Found nothing after investigation. There is no specific memory allocation due to those additional headers.

Offline

#307 2024-06-03 14:56:33

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

Updated PR 9072:
- migrate to Ubuntu24.04;
- mORMot@2.2.7579;
- return to glibc MM;
- plaintext is run after cached-queries
  - define FPC_LIBCMM_NOMSIZE to disable the malloc_usable_size() call on Linux
- use O3 compiler optimization level

Hope that FPC_LIBCMM_NOMSIZE or -O3 will help

Last edited by mpv (2024-06-03 14:57:08)

Offline

#308 2024-06-04 09:45:11

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

With the current round, we got very good results, but... /plaintext failed sad
https://www.techempower.com/benchmarks/ … =plaintext

We will see with your PR.

Offline

#309 2024-06-08 10:54:10

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

In previous run (2d run with mimalloc) where is new warnings and error

mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd9040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8be1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mimalloc: error: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb689a040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8be1040
mormot: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd9040
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd1040
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68b2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68a2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68aa040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"

Offline

#310 2024-06-17 16:35:30

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

@mpv

I guess I found the cause of our problem.
I was able to reproduce it with a Windows VM hosting a mORMot web server, and a Linux client running wrk requests.
Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.
https://github.com/synopse/mORMot2/commit/a0dda418

Offline

#311 2024-06-17 16:41:13

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

WOW!!! That's just great!
I will immediately do PR for TFB.

Offline

#312 2024-06-17 17:20:11

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

PR 9121 is ready. If everything goes well, I'll bring '-O4' back in next PR - it makes tiny improvements

Offline

#313 2024-06-18 08:14:21

TPrami
Member
Registered: 2010-07-06
Posts: 116

Re: High-Performance Frameworks

mpv wrote:

PR 9121 is ready. If everything goes well, I'll bring '-O4' back in next PR - it makes tiny improvements

Please post link to results when available.

This is more interesting to follow than most movies on Netflix smile

Offline

#314 2024-06-18 09:55:31

sakura
Member
From: Germany
Registered: 2018-02-21
Posts: 239
Website

Re: High-Performance Frameworks

ab wrote:

Randomly, some connections instances may be freed twice, so GPF occurred after a successful run.

I have seen that sometimes, I believe. Will check it out and watch it. Could never reproduce it in any sensible way. Let you know, if it seems better.

Thanks!

Offline

#315 2024-06-19 04:09:53

sakura
Member
From: Germany
Registered: 2018-02-21
Posts: 239
Website

Re: High-Performance Frameworks

Nope, I still sometimes get the following error:

0000000000B012CF  " EXC   EInvalidPointer {Message:"Invalid pointer operation"} [TRestHttpSrv 443psyprax THttp] at 7c0784 System.pas @DynArrayClear (37156)

- usually when the browser tries to prefetch a petch, while the server is just starting. I'll keep trying to localize the problem for more information, but for now, I'm at a lost as to exactly why/when that happens.

Offline

#316 2024-06-19 07:38:51

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

DynArrayClear() is called by the system e.g. when a class containing a dynamic array is released, or when a dynamic array on stack is released.
If it happens at startup, it may be due to some initialization code, which is not thread-safe.

You may enable the stack frames in your project option, to have a larger stack trace.

Offline

#317 2024-07-02 13:56:13

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

The round including PR 9121 started on 2024-07-01, so hopefully ~2024-07-04 we will complete plaintext without GPF

Offline

#318 2024-07-04 15:00:01

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

I don't understand...
This round did not let /plaintext complete either...
We will see if we have additional information in the logs at the end of the round.

Offline

#319 2024-07-13 14:59:32

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

New round, everything fine but pipelined /plaintext which still fails.

Offline

#320 2024-07-17 07:00:32

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

FYI: exception is

free(): invalid size

Offline

#321 2024-07-17 07:23:31

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

With no stack trace it is very difficult to investigate...

sad

Offline

#322 2024-09-16 07:24:58

TPrami
Member
Registered: 2010-07-06
Posts: 116

Re: High-Performance Frameworks

ab wrote:

With no stack trace it is very difficult to investigate...

sad

Any luck in finding the bug?

-Tee-

Offline

#323 2024-09-16 07:29:48

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

Since it was only in pipelined mode, which is never used on production, I honestly did not investigate much.

But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.

Offline

#324 2024-09-16 09:58:08

TPrami
Member
Registered: 2010-07-06
Posts: 116

Re: High-Performance Frameworks

ab wrote:

But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.

That is worrying indeed, hope he is OK.

-Tee-

Offline

#325 2024-09-16 14:05:18

dcoun
Member
From: Crete, Greece
Registered: 2020-02-18
Posts: 427

Re: High-Performance Frameworks

ab wrote:

But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.

I believe he is ok. Probably busy with UnityBase

Offline

#326 2024-09-18 14:51:01

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

Guys, thanks, I'm fine. It's just that time flies very strangely at war...
I will try to find the cause of GPF by switching FTB to mormot MM - PR 9283
If this does not help, we will disable HTTP pipelining in the next round

Last edited by mpv (2024-09-18 14:51:41)

Offline

#327 2024-09-27 11:49:24

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

We have the result for TFB with FPC_X64MM - it still fails for /plaintext, it also fails for /cachedqueries (ORM) and very low results for /fortunes. We'll have the log in a few days - maybe we'll see something interesting there...

Offline

#328 2024-09-30 10:17:44

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

Got a log: with FPC_X64MM /plaintext endpoint crashes with malloc(): unaligned tcache chunk detected - so, IMHO,  we have corrupted or overwritten the memory allocator's internal metadata - may be some code in mORMot with direct memory access write outside a buffer..

Offline

#329 2024-09-30 10:38:12

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,622
Website

Re: High-Performance Frameworks

I don't understand because malloc(): unaligned tcache chunk detected is a malloc error, not a mormot.core.fpcx64mm error.
It sounds like if malloc is still used somewhere by the program.

The only place where I could see malloc still being used is in the PostgreSQL client library.
But it should not trash the /plaintext test, where PostgreSQL is not involved.

Offline

#330 2024-10-23 09:25:02

mpv
Member
From: Ukraine
Registered: 2012-03-24
Posts: 1,570
Website

Re: High-Performance Frameworks

New PR 9350 to TFB with Mormot2@2.3.stable + CMem. Hope fixed TPollAsyncConnection locks structure commit solves mem problems

Offline

Board footer

Powered by FluxBB