mORMot Open Source

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,542

Website

EThreadError "Thread error" on Linux (most likely with SynLog)

@ab - need tour help. I got a "Thread error" on Linux 1 week ago on one of my production (mORMot from 2020 january) , but can't reproduce it on dev environment.
Today I reproduce "Thread error" while testing a SynDBPostgres (I develop a new unit to access Postgres directly (w/o SynDBZeos overhead) - not ready yet, but tests promises it will be very fast)

I think this may happens in case several new threads simultaneously spawns and are trying to create a TSynLogFamily simultaneously. I'm right?
The call stack is (sorry for long snipped - I try to provide as much info as possible):

SYSTEM_$$_WAITFREE_FIXED$PMEMCHUNK_FIXED$POSCHUNK 
000000000041A140 53                       push   %rbx
000000000041A141 4154                     push   %r12
000000000041A143 488d6424f8               lea    -0x8(%rsp),%rsp
000000000041A148 4889fb                   mov    %rdi,%rbx
000000000041A14B 4989f4                   mov    %rsi,%r12
000000000041A14E 488d3d5bd95a00           lea    0x5ad95b(%rip),%rdi        # 0x9c7ab0 <U_$SYSTEM_$$_HEAP_LOCK>
000000000041A155 e8160d0000               callq  0x41ae70 <SYSTEM_$$_ENTERCRITICALSECTION$TRTLCRITICALSECTION>
000000000041A15A 498b542428               mov    0x28(%r12),%rdx
000000000041A15F 4889d8                   mov    %rbx,%rax
000000000041A162 488b92b0000000           mov    0xb0(%rdx),%rdx  <------- here
000000000041A169 48895008                 mov    %rdx,0x8(%rax)
000000000041A16D 498b542428               mov    0x28(%r12),%rdx
 
mORMot/SynCommons.pas:61650               GarbageCollectorFreeAndNilList.Add(@InstanceVariable);
00000000004A702C 488b75f8                 mov    -0x8(%rbp),%rsi
00000000004A7030 488b3db98a5200           mov    0x528ab9(%rip),%rdi        # 0x9cfaf0 <U_$SYNCOMMONS_$$_GARBAGECOLLECTORFREEANDNILLIST>
00000000004A7037 488b05b28a5200           mov    0x528ab2(%rip),%rax        # 0x9cfaf0 <U_$SYNCOMMONS_$$_GARBAGECOLLECTORFREEANDNILLIST>
00000000004A703E 488b00                   mov    (%rax),%rax    <--------


mORMot/SynLog.pas:3152                    fIdent := SynLogFamily.Add(self);
0000000000585F2B 488b75e8                 mov    -0x18(%rbp),%rsi   <------------

mORMot/SynLog.pas:4001                    result := TSynLogFamily.Create(self);
00000000005888A9 488b55f8                 mov    -0x8(%rbp),%rdx
00000000005888AD be01000000               mov    $0x1,%esi
00000000005888B2 488d3d87ae3d00           lea    0x3dae87(%rip),%rdi        # 0x963740 <VMT_$SYNLOG_$$_TSYNLOGFAMILY>
00000000005888B9 e8c2d5ffff               callq  0x585e80 <CREATE>
00000000005888BE 488945f0                 mov    %rax,-0x10(%rbp)   <------------


syndbpostgres.pas:466                     with SynDBLog.Add do
00000000004D5662 488d05e7184400           lea    0x4418e7(%rip),%rax        # 0x916f50 <TC_$SYNDB_$$_SYNDBLOG>
00000000004D5669 488b18                   mov    (%rax),%rbx
00000000004D566C 48899d70ffffff           mov    %rbx,-0x90(%rbp)
00000000004D5673 4885db                   test   %rbx,%rbx
00000000004D5676 7423                     je     0x4d569b <EXECUTEPREPARED+283>
00000000004D5678 488b5b48                 mov    0x48(%rbx),%rbx
00000000004D567C 4885db                   test   %rbx,%rbx
00000000004D567F 750f                     jne    0x4d5690 <EXECUTEPREPARED+272>
00000000004D5681 488bbd70ffffff           mov    -0x90(%rbp),%rdi
00000000004D5688 e8a3310b00               callq  0x588830 <FAMILYCREATE>  <-------------------
00000000004D568D 4889c3                   mov    %rax,%rbx

The worth (as always with multi-thread) what it not reproduced while debuging in IDE and reproduced once per ~10 test program  execution

Last edited by mpv (2020-03-16 15:46:23)

danielkuettner

Member

From: Germany

Registered: 2014-08-06

Posts: 330

Re: EThreadError "Thread error" on Linux (most likely with SynLog)

We are using TSynlog very often (but only under Win) in our multi-threaded environment with no error/AVs.

What sometimes happens is TSQLLog.Add without call of TSynLog.Enter before. If this was too stupid please apologize...

I develop a new unit to access Postgres directly (w/o SynDBZeos overhead)

Thats very interesting. We also plan this but our intension is not less overhead or more performance. Instead we are trying to be more flexible with bypassing libpq.
I've found a swift project (https://github.com/codewinsdotcom/PostgresClientKit). But I'm just starting this project.

It were great if you would share your Postgres driver!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,231

Website

Re: EThreadError "Thread error" on Linux (most likely with SynLog)

Perhaps https://synopse.info/fossil/info/ee2b5cceb5 may help...

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,542

Website

Re: EThreadError "Thread error" on Linux (most likely with SynLog)

Yesterday I test latest SynLog changes day long - at last on my test case AV is disppeared - @ab - thanks!

What sometimes happens is TSQLLog.Add without call of TSynLog.Enter before

We also use SynLog very hardly (in full log mode some of our prod. can produce ~200Mb of logs per minute). Problem (hope it fixed now) happens only on hi-concurrency.
BTW we never see error if adding to log without call to Enter (using calls like SynDBLog.Add.Log(sllInfo, ...) for example)

It were great if you would share your Postgres driver!

I will contribte it it to mORMot repo (almost ready - some small fixes required).  But currently it's FPC only

Last edited by mpv (2020-03-18 07:33:33)