mORMot Open Source

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

SynCrtSock - unexpected behavior in case port already in use

Commit for FreeBSD [2958b7f] (17 March) introduce an unexpected behavior for TCrtSocket.OpenBind

Before [2958b7f] in case, for example, port is already used we got a valid Exception during Bind call  - ECrtSocket ("OpenBind(0.0.0.0:8881,bind) failed: Another process may be currently listening to this port! [98  Address already in use]")

After Alfred's patch we do not got an exception (typos for sure), but when I try to return exception back  - see #pull requiest #309 I got VERY unexpected behavior.

My server (and mORMot auto-test)  either hangs, or (in case I set fExecuteFinished) I got EThreadError while server stops if port is already in use. In TSQl3 error looks like

! Exception EThreadError raised with messsage:
!  Thread error
! Service oriented architecture - Client side REST main thread
! Exception EThreadError raised with messsage:

Need help to solve a situation...

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,673

Website

Re: SynCrtSock - unexpected behavior in case port already in use

Please try https://synopse.info/fossil/info/6934f79766

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

Now Exception text is as expected. But the main problem still exists: in case port is busy ./TestSQL3  throws EThreadError.

To verify just run something what listen on 8888 ( python -m SimpleHTTPServer 8888 should work on ubuntu) and when ./TestSQL3

I'm care about it because I got a EThreadError on real life scenarios in my app what I can't solve even after aggressive long-term debugging  (not related to port, but to exceptions inside threadPool thread execution). Now it reproduced with mORMot test (I think the root of the problem is the same).

@ab - may be you can look on this scenario?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,673

Website

Re: SynCrtSock - unexpected behavior in case port already in use

My guess is that it is a FPC TThread limitation/bug: any exception in a TThread.Create constructor is fatal - at least under Linux.

I tried to put the Bind() call into the THttpServer.Execute method.
You should now call THttpServer.WaitStarted after THttpServer.Create if you want to have an exception raised in the main thread. If you don't call it, the server will fail silently to start, but at least with no EThreadError.
Please check https://synopse.info/fossil/info/434798ffa8

I don't have the EThreadError any more in TestSQL3.

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

Looks like I found the reason of EThreadError. At last for THttpApiServer. In my case I randomly got it because of long operation inside HttpThreadTerminate handler, but problem can  occurs everywhere. The flow:

FreeAndNill(fHttpApiServer) -> inside THttpApiServer.Destroy main thread terminates and call -> THttpApiServer.DestroyMainThread -> and inside DestroyMainThread

procedure THttpApiServer.DestroyMainThread;
begin
    for i := 0 to length(fClones)-1 do
      fClones[i].Terminate; // for CloseHandle() below to finish Execute // <-------- trigger TSynThread.DoTerminate (1)
    ......

    for i := 0 to length(fClones)-1 do
      fClones[i].Free;    // <------------------- here HttpThreadTerminate for some of clones is not finished yet, but fClones is already starting to Free (2)
    fClones := nil;
end;

and finally EThreadError in

procedure TSynThread.DoTerminate;
begin
  if Assigned(fStartNotified) and Assigned(fOnThreadTerminate) then begin
    fOnThreadTerminate(self);    /// <----------- this function take some time
    fStartNotified := nil;            //// <----------- here Self is already destroyed by fClones[i].Free from DestroyMainThread
  end;
  inherited DoTerminate;
end;

I think we need some waiting loop between (1) and (2). @ab - can You fix it or I can try?

Last edited by mpv (2020-04-20 14:40:38)

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

About my prev. post - sorry for wrong topic. For a while I add a sleep(1000) inside THttpApiServer.DestroyMainThread between (1) and (2)
It temporary solve my problems. Since no one voiced a similar problem looks like most of Windows builds are done with Delphi + FastMM. In case of FastMM problem exists for sure, but AV may not occurs because memory not actually returned to OS.

We can keep all as is in mORMot1, because a good solution require a deep refactoring from my POW. Today we have a wait-for-thread-finish loops in several places of the framework. The ideal is to have such loop in one place (inside ThreadPool)

@ab - please, take this problem into account in mORMot2

Last edited by mpv (2020-04-21 08:43:49)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,673

Website

Re: SynCrtSock - unexpected behavior in case port already in use

Please check https://synopse.info/fossil/info/210562c189

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

This not solve the problem (verified - got AV on destroy on FPC + Windows). We need to wait all clones terminated inside  THttpApiServer.DestroyMainThread before doing a clones(i).free;

Something as in TSQLRestServer.EndCurrentThread did

fStats.NotifyThreadCount(-1);  

and wait in DestroyMainThread until fCurrentThreadCount <> 0

Last edited by mpv (2020-04-21 12:05:11)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,673

Website

Re: SynCrtSock - unexpected behavior in case port already in use

Clones[].Free calls Destroy which will wait until actually finalizing the array.

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

@ab - after a long debugging I don't understand WHY tour implementation not works as expected. So I create a minimal reproducible example see this gist what shows an HTTP API server termination problem for the current trunk.

In case OnHTTPThreadTerminate handler takes some time (REAL_WORK_DURATION is 100 for example) this program shows what OnHTTPThreadTerminate does not have time to work out completely.
In real app I got AV randomly. In this example - assertion

Please, look on it - I do not understand what to do...

mpv

Member

From: Ukraine

Registered: 2012-03-24

Posts: 1,571

Website

Re: SynCrtSock - unexpected behavior in case port already in use

I even recompile FPC in debug mode (in fpcupdeluxe Setup+ add -gl to "FPC Options"). Av occurs inside threads.inc procedure TThread.DoTerminate; - Self is points to unallocated memory block there (because thread is destroyed by THttpApiServer.DestroyMainThread fClones[].free while we sleep in OnHTTPThreadTerminate ?)
Since I have this problem may be a year or longer this not depends on FPC version i think.

Last edited by mpv (2020-04-24 07:43:46)