mORMot Open Source

PR 9480 with mORMot@2.3.9262 2024-12-23  fail again. Round 23 starts in few days, PR accepted until Jan 17. @ab - do we have some changes what may fix our problem (I saw changes about closing connections) - should I do a new PR?

Are you planning any more improvements in the next few days? Or will I do an MR with current sources?

Yes, there is regression. I will fix the PR after you find the reason.

Because memory problems is not solved I tries *without* `hsoEnablePipelining` option, and got a strange result: when client query /plaintext  in pipelining mode (see https://synopse.info/forum/viewtopic.ph … 769#p38769 for how to call) - server answer only to firs request in each thread. @ab - can you, please, look at this problem? Correct server behavior is "to respond to pipelined requests correctly, with non-pipelined but valid responses even if server does not support HTTP pipelining."

New PR 9350 to TFB with Mormot2@2.3.stable + CMem. Hope fixed TPollAsyncConnection locks structure commit solves mem problems

Got a log: with FPC_X64MM /plaintext endpoint crashes with malloc(): unaligned tcache chunk detected - so, IMHO,  we have corrupted or overwritten the memory allocator's internal metadata - may be some code in mORMot with direct memory access write outside a buffer..

Guys, thanks, I'm fine. It's just that time flies very strangely at war...
I will try to find the cause of GPF by switching FTB to mormot MM - PR 9283
If this does not help, we will disable HTTP pipelining in the next round

The easy way  to use proxy with curl is to set the environment variables http_proxy and https_proxy

PR 9121 is ready. If everything goes well, I'll bring '-O4' back in next PR - it makes tiny improvements

In previous run (2d run with mimalloc) where is new warnings and error

mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd9040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bc8be1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mimalloc: error: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb689a040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8be1040
mormot: mi_free: pointer might not point to a valid heap region: 0x7c5bc8bd1040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd9040
mormot: mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x7c5bc8bd1040
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68b2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68a2040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x7c5bb68aa040
mormot: (this may still be a valid very large allocation (over 64MiB))
mormot: mimalloc: assertion failed: at "./include/mimalloc-internal.h":467, _mi_segment_page_of
mormot:   assertion: "idx < segment->slice_entries"

I also do not see any problem there - may be this is false-positive warning. And I can`t reproduce crush on my environment
On laet run (with mimalloc) failure message is

mimalloc: assertion failed: at "./src/alloc.c":231, mi_page_usable_size_of
assertion: "ok"

assertion source - https://github.com/microsoft/mimalloc/b … loc.c#L231

P.S. Real assertion error is here - https://github.com/microsoft/mimalloc/b … loc.c#L223

P.P.S.
a test case where we fails (may be it helps to reproduce crash on your side):

wrk -H 'Host: 10.0.1.1' -H 'Accept: text/plain,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 56 http://10.0.1.1:8080/plaintext -s pipeline.lua -- 16

A pipeline.lua is here - https://github.com/TechEmpower/Framewor … peline.lua

On May 29 commit
- with hsoHeadersInterning warnings remains the same
- without hsoHeadersInterning only one remain

==1532807== Conditional jump or move depends on uninitialised value(s)
==1532807==    at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1532807==    by 0x6BA141: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)

About "shutdown may block" - from my experience it not block, but sometimes wait too long

This is valgrind warnings I got on /plaintext with pipelining

==1515797== Thread 14 R1:8080:
==1515797== Use of uninitialised value of size 8
==1515797==    at 0x527D43: MORMOT.CORE.DATA$_$TDYNARRAYHASHER_$__$$_FINDORNEWCOMP$LONGWORD$POINTER$TDYNARRAYSORTCOMPARE$$INT64 (mormot.core.data.pas:9774)
==1515797==    by 0x9CC166DE: ???
==1515797==    by 0x6D437AF: ???
==1515797==    by 0x492B7F: ??? (in /home/pavelmash/dev/mORMot2/ex/techempower-bench/exe/raw)
==1515797== 
==1515797== Use of uninitialised value of size 8
==1515797==    at 0x527BCF: MORMOT.CORE.DATA$_$TDYNARRAYHASHER_$__$$_FINDORNEW$LONGWORD$POINTER$PPTRINT$$INT64 (mormot.core.data.pas:9718)
==1515797==    by 0xDE: ???
==1515797==    by 0x9CC166DE: ???
==1515797== 
==1515797== Conditional jump or move depends on uninitialised value(s)
==1515797==    at 0x5281D5: MORMOT.CORE.DATA$_$TDYNARRAYHASHER_$__$$_FINDBEFOREADD$POINTER$BOOLEAN$LONGWORD$$INT64 (mormot.core.data.pas:9891)
==1515797==    by 0xFFFFFFFFFFFFFF1F: ???
==1515797== 
==1515797== Use of uninitialised value of size 8
==1515797==    at 0x527E88: MORMOT.CORE.DATA$_$TDYNARRAYHASHER_$__$$_HASHADD$LONGWORD$INT64 (mormot.core.data.pas:9815)
==1515797==    by 0xAF5034F: ???
==1515797== 
==1515797== Conditional jump or move depends on uninitialised value(s)
==1515797==    at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1515797==    by 0x6BA1E1: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3693)
==1515797== 
==1515797== Thread 104 R1:8080:
==1515797== Use of uninitialised value of size 8
==1515797==    at 0x527E88: MORMOT.CORE.DATA$_$TDYNARRAYHASHER_$__$$_HASHADD$LONGWORD$INT64 (mormot.core.data.pas:9815)
==1515797==    by 0xAF5198F: ???
.....
Please wait: Shutdown 12 servers and 96 threads
==1515797== Thread 64 R1:8080:
==1515797== Conditional jump or move depends on uninitialised value(s)
==1515797==    at 0x48881F: MORMOT.CORE.BASE_$$_BYTESCANINDEX$PBYTEARRAY$INT64$BYTE$$INT64 (mormot.core.base.asmx64.inc:2091)
==1515797==    by 0x6BA11C: MORMOT.NET.HTTP$_$THTTPREQUESTCONTEXT_$__$$_PROCESSREAD$TPROCESSPARSELINE$BOOLEAN$$BOOLEAN (mormot.net.http.pas:3672)
==1515797==    by 0xAF52E8F: ???
==1515797==    by 0x6DEA18: MORMOT.NET.ASYNC$_$THTTPASYNCSERVERCONNECTION_$__$$_FLUSHPIPELINEDWRITE$$TPOLLASYNCSOCKETONREADWRITE (mormot.net.async.pas:4283)
==1515797==    by 0x1FFEFFF06F: ???
==1515797==    by 0x49247CF: ??? (pthread_create.c:321)
==1515797==    by 0x76B61FF: ???
==1515797==    by 0xA2D9900: ???
==1515797==    by 0x76B61EF: ???
==1515797==    by 0x6DED35: MORMOT.NET.ASYNC$_$THTTPASYNCSERVERCONNECTION_$__$$_ONREAD$$TPOLLASYNCSOCKETONREADWRITE (mormot.net.async.pas:4334)
==1515797== 

I think current run fails again because of /plaintext (therefore cached_query is not started at all).
BTW, previous run crushes on /plaintext with message munmap_chunk(): invalid pointer (pre-previous with malloc(): invalid next size (unsorted) )
PR with mimalloc just merged, let's see how we run with it in next round.

Ahh.. years ago I also tries to catch this bug without success.. I even tries with valgrind in memcheck mode, but there is so many warnings (mostly in mormot crypto) what the root of evils hidden.
For a while I made a MR 9036 what replace glibc MM with mimalloc.

I suggest we start looking into the valgrind warnings (see https://valgrind.org/docs/manual/mc-manual.html). Maybe if we remove them one by one, we will find the problem. By the way, the heisenbug also exists in mORMot1 - very rarely, once a week with an average load of 1000 RPS, I get an AV in my mORMot1 application in production.

This time we crashes on plaintext (not on cached queryes) on 256 concurrecy. See https://tfb-status.techempower.com/unzi … xt/raw.txt

Let's wait for a next round to detect is this a heisenbug or not

Unfortunately, with MR 8949, it seems that the server has crashed on /cached-queries (orm), so plaintext is not executed and we do not appear in the composite score this time. We will see what happens after round ends in text logs.
Increasing the number of threads for an asynchronous server cpuCount * 2 -> cpuCount * 4 does not increase the speed of async(nopin) (becomes slower)
But /cached_query implementation for raw server is now #1

Sorry for delay. Now crashes on TRawAsyncServer.rawqueries because uninitialized PLecuer is passed to GetRawRandomWorlds.

@ab -  maybe we should go back to https://github.com/synopse/mORMot2/commit/d312c00d ? because these lecturers who are now everywhere have made the code unreadable (and not necessarily faster)

@ttomas - I'll increase threads 2 -> 4 for [async,nopin]
@ab - Please, fix raw.pas, because it do not compiles (TRawAsyncServer.ComputeRandomWorld is not accessible from inside TAsyncWorld.DoUpdates / Queries) - I'll follow your fix and made an MR

Verified with 2.2.7351 - now escaping is OK. TFB MR #8883 is ready.
I don't bother with mimalloc either - we'll see how it goes with the current improvements.

Yes, disabling sleep should help for the asynchronous server, because according to dstat (now it's running) the processor is idle ~66% (only 40% is used). By the way, for a raw, idle is ~44%, which is also too high, for example, in h2o idle is ~20%.
In today's result based on mORMot2.2.stable, our score increased from 18,281 to 19,303, I hope we will be #5.

I'll test fresh sources on my computer (my test server is in a region with power outages and is unavailable) and do a MR  (hope today)

Currently, with 56 CPUs, mormot [async,nopin] (1 process, 112 threads, no pinning) is good for updates, but bad for other scenarios (compared to async with pinning and direct pinning).
Let's wait for the next launch with the latest PR (the current launch is based on the same sources as Round22).

The bad thing is that we don't have a test server that matches the TFB configuration now, so we can't test different thread/server/pin combinations to find the best one.

To solve a problem, described in #211 ( `GLIBC_2.34' not found ) I compile a production version of my product using docker with fpc what based on Ubuntu 20.04 (GLIBC 2.31)

I'm confused a little, because the same sqlite3.o is used in mormot2.2_stable and TFB example compiles well. Any help is wellcome