Websockets client - how to detect the closing of the socket?

jaclas · 2019-05-26 10:45:46

Based on example 31, I wrote a simple service for uploading logs using websockets.
However, I would like the client side (which implements the callback class) to receive an event in the event of a lost connection so that it can re-connect to the server and pass the callback interface.

I made this code for client:

procedure TLogClientWebsockets.OnClientDisconnected(sender: TObject);
begin
  //   disconnect event handling is never called!!
end;

procedure TLogClientWebsockets.OnWebSocketsClosed(sender: TObject);
begin
  //   disconnect event handling is never called!!
end;

procedure TLogClientWebsockets.Start();
begin
  fModel := TSQLModel.Create([], cRemoteLogRootName);

  fRestClient := TSQLHttpClientWebsockets.Create('127.0.0.1', cRemoteLogHTTPPort.ToString, fModel);
  fRestClient.Model.Owner := fRestClient;

  fRestClient.OnWebSocketsClosed := OnWebSocketsClosed;
  fRestClient.WebSockets.OnWebSocketsClosed := OnWebSocketsClosed;
  fRestClient.WebSockets.Settings.OnClientDisconnected := OnClientDisconnected;
  fRestClient.WebSockets.Settings.HeartbeatDelay := 3000;
  fRestClient.WebSockets.Settings.DisconnectAfterInvalidHeartbeatCount := 3;
  fRestClient.WebSocketLoopDelay := 100;
  fRestClient.WebSocketsUpgrade(cWebSocketEncryptionKey);
  fRestClient.ServiceDefine([ILogService], sicShared);

  if not fRestClient.Services.Resolve(ILogService, fService) then
    raise EServiceException.Create('Service ILogService unavailable');

  fLogManager := TLogManager.Create(TPanel(fLogPanel), 0, nil);
  fCallback := TRemoteLogCallback.Create(fLogManager, fRestClient, IRemoteLogCallback, fLogPanel.Handle);
  fService.StartLogService(fCallback);
end;

Then I run the application and during transmission using TCPView (from Sysinternals) I close the socket that supports the websockets connection. On the server side, the onClientDisconnect event fires. But nothing is happening on the client's side!
How to detect connection loss from the client side?

ab · 2019-05-26 21:26:59

A raw TCP Connection closed is not noticeable until a send/receive operation fails, by definition of how TCP works.

A WebSockets connection gracefully closed with its "Connection Close" message from the other end will be notified.

jaclas · 2019-05-27 06:30:20

So there is no way to know from the client side (callback implementation side) that the connection has died?

My problem is protection against network errors and the possibility of resuming the connection. How to do it?

pvn0 · 2019-05-27 09:09:18

jaclas wrote:

So there is no way to know from the client side (callback implementation side) that the connection has died?

But you will know, the connection will timeout and it will throw an exception (or trigger some onFail event handle) that you are supposed to handle.

jaclas · 2019-05-27 09:39:28

In the code that I showed in the example, there are two events:

procedure TLogClientWebsockets.OnClientDisconnected(sender: TObject);
begin
  //   disconnect event handling is never called!!
end;

procedure TLogClientWebsockets.OnWebSocketsClosed(sender: TObject);
begin
  //   disconnect event handling is never called!!
end;

None of these events is called, so I ask what I am doing wrong or how to do it correctly?

pvn0 · 2019-05-27 11:58:59

The client websocket connection does not send ping frames to the server by default so you will never get a timeout exception if you don't actually make a request to the server.

Regardless, there seems to be a bug anyway, since setting HeartbeatDelay does absolutely nothing. I've made a pull request with a fix that I tested against sample 31 chat application, https://github.com/synopse/mORMot/pull/204.

With this fix, WebSockets.Settings.OnClientDisconnected will be called once DisconnectAfterInvalidHeartbeatCount has been reached.

Last edited by pvn0 (2019-05-27 12:01:03)

ab · 2019-05-27 12:58:41

I have merged your pull request.

Thanks!

jaclas · 2019-05-27 14:17:34

Thanks pvn0 and Arnaud! I will test it today...

Daniel · 2019-06-10 08:52:35

Has anyone tested and confirmed that this works now ?

jaclas · 2019-06-10 11:52:11

It partially works, but there are further problems.
On the server side, I assign the OnClientDisconnected event:

procedure TWSLogServer.StartRemoteLogService();
begin
  [...]
  fHTTPServer := TSQLHttpServer.Create(IntToStr(fHTTPPort), [fRestServer], '+', useBidirSocket);
  fWSServer := fHTTPServer.WebSocketsEnable(fRestServer, cWebSocketEncryptionKey);
  fWSServer.Settings.OnClientDisconnected := OnClientDisconnected;   <========== event assigned
end;

When I release the server, I want to release the event, so I execute the code:

procedure TWSLogServer.StopRemoteLogService();
begin
  if Assigned(fHTTPServer) then
  begin
    fWSServer.Settings.OnClientDisconnected := nil;  <======== event released, but only on TWebSocketServer class side!
    FreeAndNil(fHTTPServer);
    FreeAndNil(fRestServer);
  end;
end;

However, the event is still called!

Code review and debugging led me to the conclusion that there is an error in the settings, because they are saved in the record:

  /// parameters to be used for WebSockets process
  TWebSocketProcessSettings = record
 [...]

And in working classes there are separate instances of those records that have copies of settings data, so that changing Settings in the TWebSocketServer class instance does not affect the TWebSocketProcess class instance.

  TWebSocketServer = class(THttpServer)
  protected
    fSettings: TWebSocketProcessSettings;  <==== first settings
    
  TWebSocketProcess = class(TSynPersistent)
  protected
    fSettings: TWebSocketProcessSettings;  <===== next settings

How do I release a OnClientDisconnected event from a TWebSocketProcess?

Maybe worth change TWebSocketProcessSettings from record to class?

Last edited by jaclas (2019-06-10 11:54:24)

pvn0 · 2019-06-10 14:38:24

This is unrelated to the original question and it would be better to open it's own forum thread, regardless, yes, if you change TWebSocketServer.Settings it won't propagate to existing connections, this is by design :

// the settings to be used for WebSockets process
// - note that those parameters won't be propagated to existing connections
function Settings: PWebSocketProcessSettings; {$ifdef HASINLINE}inline;{$endif}

jaclas wrote:

How do I release a OnClientDisconnected event from a TWebSocketProcess?

Currently, you can't, at least not for existing connections. You have control over code that is executed in the event handler so you can chose to ignore the notification. (that code better be thread safe either way).

I agree that the current implementation is not ideal, I will fix this when I have some free time.

jaclas · 2019-06-10 15:50:36

I can't ignore events, because the problem is that the event method is part of the freed server object and at the moment of calling the object is already released which ends with a series of AV...
For now, I did a workaround, I additionally support the OnClientConnected() event and in it I remember the "self" object, which is TWebSocketProcess, thanks to which it later has access to its settings. But it is ugly :-|

But I will wait for a better solution and thanks for the comments :-)

Daniel · 2019-06-10 16:01:35

Frankly, problems with such basic functionality as disconnect are discouraging.
I introduced Mormot into my company for it's WebSocket implementation but my colleagues now doubt it it was a good choice.

jaclas · 2019-06-14 16:40:02

I got the next problem.

If after unexpected closing the websocket connection I will resume this connection again (start websocket from client side) then next connection break (simulated by TCPView -> Close connection) no longer triggers the OnWebSocketsClosed event.

pvn0 · 2019-06-14 20:59:58

jaclas wrote:

I got the next problem.
If after unexpected closing the websocket connection I will resume this connection again (start websocket from client side) then next connection break (simulated by TCPView -> Close connection) no longer triggers the OnWebSocketsClosed event.

I was unable to reproduce this , can you give a more detailed explanation?

Chaa · 2019-06-15 04:22:11

jaclas wrote:

If after unexpected closing the websocket connection I will resume this connection again (start websocket from client side) then next connection break (simulated by TCPView -> Close connection) no longer triggers the OnWebSocketsClosed event.

I have a similar problem:
https://synopse.info/forum/viewtopic.ph … 392#p29392

jaclas · 2019-06-15 09:21:43

My event OnWebSocketsClosed looks like this:

procedure TRemoteLogClientWebsockets.OnWebSocketsClosed(sender: TObject);
begin
  TThread.Queue(nil, procedure
                           begin
                             fService.StartLogSession(fCallback);
                           end);
end;

After brute-force close connection (by using this tool: https://docs.microsoft.com/en-us/sysint … s/tcpview) first time this event is called and websocket is connected again, but after second disconnection event OnWebSocketsClosed event is never called again.

jaclas · 2019-06-17 09:19:58

I create simple example apps based od Example 31 from mORMot demos.
See screencast how the second close the connection does not result in an OnWebSocketsClosed event on client side:

https://youtu.be/ugrVsqb-998

client source: https://pastebin.com/4teDqTJG
server source: https://pastebin.com/QKLFUKZF

Last edited by jaclas (2019-06-17 09:22:40)

pvn0 · 2019-06-18 07:27:14

jaclas wrote:

I create simple example apps based od Example 31 from mORMot demos.
See screencast how the second close the connection does not result in an OnWebSocketsClosed event on client side:
https://youtu.be/ugrVsqb-998

client source: https://pastebin.com/4teDqTJG
server source: https://pastebin.com/QKLFUKZF

Thanks, very nice example.

Fixed in https://github.com/synopse/mORMot/pull/213

the problem was the very unfortunate placement of the TWebSocketProcessSettings record in the THttpClientWebSockets class, when the socket closes, class is destroyed and the settings lost with it. That is why you don't get the second event triggered, the settings are reset to default. I've now introduced DefaultWebSocketProcessSettings in the TSQLHttpClientWebsockets class, every new client websocket class instance will inherit this default settings (it won't break any existing implementation) so you better do this:

  Client.DefaultWebSocketProcessSettings.HeartbeatDelay := 3000;
  Client.DefaultWebSocketProcessSettings.DisconnectAfterInvalidHeartbeatCount := 2;
  Client.OnWebSocketsClosed := OnWebSocketsClosed;

instead of

  Client.WebSockets.Settings.HeartbeatDelay := 3000;
  Client.WebSockets.Settings.DisconnectAfterInvalidHeartbeatCount := 2;
  Client.WebSockets.OnWebSocketsClosed := OnWebSocketsClosed;

notice the change in assigning the OnWebSocketsClosed event, it's important.

Solving this problem revealed a much bigger issue, there's an internal connection state that is tracked at the TSQLRestClientURI level, it was not being properly updated when the websocket connection re-establishes, this was also fixed.

Last edited by pvn0 (2019-06-18 08:04:11)

jaclas · 2019-06-18 11:58:55

Very, very big thanks for your effort and time pvn0!

Now all is right! :-D

Last edited by jaclas (2019-06-18 11:59:25)

Daniel · 2019-06-21 06:49:40

This is not yet merged into master ?

jaclas · 2019-06-21 09:59:19

Yes, I'm also waiting for the official repo.
Arnaud, please review and merge these changes.

ab · 2019-06-21 12:23:35

Has been merged - with some minor blank fixes - as https://github.com/synopse/mORMot/pull/213

Thanks for the feedback!

mORMot Open Source

#1 2019-05-26 10:45:46

Websockets client - how to detect the closing of the socket?

#2 2019-05-26 21:26:59

Re: Websockets client - how to detect the closing of the socket?

#3 2019-05-27 06:30:20

Re: Websockets client - how to detect the closing of the socket?

#4 2019-05-27 09:09:18

Re: Websockets client - how to detect the closing of the socket?

#5 2019-05-27 09:39:28

Re: Websockets client - how to detect the closing of the socket?

#6 2019-05-27 11:58:59

Re: Websockets client - how to detect the closing of the socket?

#7 2019-05-27 12:58:41

Re: Websockets client - how to detect the closing of the socket?

#8 2019-05-27 14:17:34

Re: Websockets client - how to detect the closing of the socket?

#9 2019-06-10 08:52:35

Re: Websockets client - how to detect the closing of the socket?

#10 2019-06-10 11:52:11

Re: Websockets client - how to detect the closing of the socket?

#11 2019-06-10 14:38:24

Re: Websockets client - how to detect the closing of the socket?

#12 2019-06-10 15:50:36

Re: Websockets client - how to detect the closing of the socket?

#13 2019-06-10 16:01:35

Re: Websockets client - how to detect the closing of the socket?

#14 2019-06-14 16:40:02

Re: Websockets client - how to detect the closing of the socket?

#15 2019-06-14 20:59:58

Re: Websockets client - how to detect the closing of the socket?

#16 2019-06-15 04:22:11

Re: Websockets client - how to detect the closing of the socket?

#17 2019-06-15 09:21:43

Re: Websockets client - how to detect the closing of the socket?

#18 2019-06-17 09:19:58

Re: Websockets client - how to detect the closing of the socket?

#19 2019-06-18 07:27:14

Re: Websockets client - how to detect the closing of the socket?

#20 2019-06-18 11:58:55

Re: Websockets client - how to detect the closing of the socket?

#21 2019-06-21 06:49:40

Re: Websockets client - how to detect the closing of the socket?

#22 2019-06-21 09:59:19

Re: Websockets client - how to detect the closing of the socket?

#23 2019-06-21 12:23:35

Re: Websockets client - how to detect the closing of the socket?

Board footer