You are not logged in.
I am using last version from GitHub,
The following can exist also:
Exception: 20241108 20071551 ! EXC EInterfaceFactory {Message:"TInterfacedObjectFakeClient.FakeCall(ImwbService.GetDBormvers) failed: 'Invalid returned JSON content: expects {result:...}, got "} [Main] at 45df37 mormot.core.text.pas ESynException.RaiseUtf8 (9507) fresultcomp.pas Tresultcomp.FormCreate (150) Vcl.Forms.pas TCustomForm.DoCreate (5516) Vcl.Forms.pas TCustomForm.AfterConstruction (5384) System.pas @AfterConstruction (19735) Vcl.Forms.pas TCustomForm.Create (5373) feditdoc.pas Teditdocument.b_protoClick (713) Vcl.Controls.pas TControl.Click (8037) AdvGlowButton.pas TAdvCustomGlowButton.Click (3798) Vcl.Controls.pas TControl.WMLButtonUp (8177) AdvGlowButton.pas TAdvCustomGlowButton.WMLButtonUp (4342) Vcl.Controls.pas TControl.WndProc (7921) Vcl.Controls.pas TWinControl.IsControlMouseMsg (10886) Vcl.Controls.pas TWinControl.WndProc (11156) Vcl.Controls.pas TWinControl.MainWndProc (10823) System.Classes.pas StdWndProc (19092) Vcl.AppEvnts.pas TMultiCaster.GetAppEvents (687) Vcl.Forms.pas TApplication.ProcessMessage (13376) Vcl.Forms.pas TApplication.HandleMessage (13406) Vcl.Forms.pas TApplication.Run (13545)
GetDBormvers is declared as:
function GetDBormvers:rawjson;
An other error I usually find:
Exception: 20241112 17391118 # EXC ENetSock {LastError:"nrClosed",Message:"THttpClientSocket.SockRecv(283) read=0 [Closed - #5]"} [Main] at b2bf7e mormot.net.sock.pas TCrtSocket.SockRecv (5862) mormot.net.client.pas THttpClientSocket.RequestInternal (2408) fastmm5.pas FastMM_GetMem_GetSmallBlock (7186) System.pas @GetMem (4969) System.pas @NewAnsiString (26157) System.pas @LStrSetLength (30045) mormot.net.client.pas THttpClientSocket.Request (2516) mormot.rest.client.pas TRestClientUri.IsOpen (2155) mormot.rest.client.pas TRestClientUri.IsOpen (2227) mormot.rest.http.client.pas TRestHttpClientSocket.InternalRequest (780) mormot.rest.http.client.pas TRestHttpClientGeneric.InternalUri (580) mormot.rest.http.client.pas TRestHttpClientGeneric.InternalUri (582) rest.wsclient.pas TPCHRclient.setJWTheader (278) rest.wsclient.pas TPCHRclient.setJWTheader (279) rest.wsclient.pas TPCHRclient.setJWTheader (280) mormot.rest.client.pas TRestClientUri.OnBackgroundProcess (1980) mormot.rest.client.pas CallInternalUri (2524) mormot.rest.client.pas TRestClientUri.Uri (2556) System.pas TMonitor.TryEnter (20364) System.pas TMonitor.TryEnter (20364) System.Generics.Collections.pas TListHelper.InternalDeleteRange4 (3873) System.Generics.Collections.pas TListHelper.InternalDeleteRange4 (3879) fastmm5.pas FastMM_GetMem_GetSmallBlock (7186) System.pas @GetMem (4969) System.pas @NewAnsiString (26157) mormot.soa.client.pas DoClientCall (643) mormot.soa.client.pas TServiceFactoryClient.InternalInvoke (688) mormot.soa.client.pas TServiceFactoryClient.Invoke (583) mormot.core.interfaces.pas TInterfacedObjectFake.FakeCallGetJsonFromStack (3448) mormot.core.interfaces.pas TInterfacedObjectFake.FakeCallInternalProcess (3544) mormot.core.interfaces.pas TInterfacedObjectFakeRaw.FakeCall (3307)
Is this expected due to network conditions?
I was a bit late to answer because OnClientDisconnected caused the service to hang but with the https://github.com/synopse/mORMot2/issues/312 it works now a week without problems
In the client, I have assigned to TRestHttpClientWebsockets instance the OnWebSocketsUpgraded event before calling WebSocketsUpgrade
in the OnWebSocketsUpgraded event procedure I am getting the server's connection ID for that client: TRestHttpClientWebsockets(sender).WebSockets.WebSockets.ConnectionID and I pass with a call to the server at once together with a unique client ID that my client app has per client.
The server keeps a list of clients and connectionIDs in a IKeyValue directory that it gets from the above pathway and when OnClientDisconnected in the server is fired, it searchs the IKeyValue to find if this connection ID exists.
if so, it pushs a "request" in a TSynThreadPool instance (mormot.core.threads) and exits. I could just remove it from a list but just it takes some time to do some more cleaning work, I do not want the server to wait. I keep also a timestamp to check using the unique client ID, if my client has already reconnected in the meantime, in order not to clear it. Probably the tdatetime should be converted to TTimeLog also.
This thread pool is created for asynchronous task by inheriting from it:
TworkSoul=class
public
work:rawutf8; id:int64; dat:tdatetime;
end;
TmySlavePool=class(TSynThreadPool)
protected
procedure Task(aCaller: TSynThreadPoolWorkThread; aContext: Pointer); override;
procedure TaskAbort(aContext: Pointer); override;
end;
var soulsbox:TmySlavePool;
procedure Tmyserver.WSclientDisconnects(Protocol: TWebSocketProtocol);
var s:tworksoul;
begin
s:=tworksoul.Create;
s.work:='removeclient'; s.id:=protocol.ConnectionID; s.dat:=now;
soulsbox.push(s);
end;
Thank you @ab.
Is it possible to check if a string is utf8 or has invalid characters?
I am getting the following exception in the logs
31/10/2024 10:57:40.832 Exception 8 ESqlDBPostgres {Message:"TSqlDBPostgresLib Exec failed: 22021 [ERROR: invalid byte sequence for encoding \"UTF8\": 0xa2\nCONTEXT: unnamed portal parameter $14\n]",Statement:null} [R1:testdv1testdv0] at 8ccdc6 ../../../../../DG/mORMot2/src/db/mormot.db.raw.postgres.pas (437) ../../../../../DG/mORMot2/src/db/mormot.db.sql.postgres.pas (1015)
I think I managed to arrive to a working solution. Tcpview from microsoft was also real helpful to "close" connections
I have found also this thread: https://synopse.info/forum/viewtopic.php?id=5018
Useful will be to know how ClientRestoreCallbacks work and the automatic re-connection with full callbacks support mentioned here: https://synopse.info/forum/viewtopic.php?id=6066
Thanks you a lot, this really answers to my question.
So probably I should not try to deal with ClientRestoreCallbacks option
I have a lot of clients behind a firewall and I can see one IP for multiple clients. It does not helps
I was thinking to add a custom header for each client, to keep with the callback interface, but I was wondering if there is an other way through mormot framework
Yes, exactly, this the process for a client closing normaly. I am using it. I was wondering if there is a way to identify clients closed abormally
Documentation is more detailed than the example and I have followed the direction to keep an array of the callback interfaces.
Still the problem is how to be notified that a client has been disconnected and the callback interface is invalid without trying to send it a message.
I am not also sure of what to do to keep the connection, except for polling messages.
Thank you a lot for the response.
My main problem is to transform this "Protocol: TWebSocketProtocol" to the callback interface that it is created in the example.
I have found that the callbacks are of type TInterfacedObjectFakeServer are kept in TServiceContainerServer.fFakeCallbacks in the server, but it is much complicated for me with existing knowledge of mormot
And the ClientRestoreCallbacks above can update the connectionID (type TRestConnectionID or THttpServerConnectionID ?) and I am not sure if TOnWebSocketProtocolClosed event should be used
I need more documentation of the idea behind this process
when do we need ClientRestoreCallbacks to have a true value?
I am trying to tune a websocket server to be able to find when a websocket client has closed the connection.
IServiceWithCallbackReleased is ok for clients that close the connection as it should be closed
But when a client looses its connection (due to network or other reasons), the server can not be informed about it, or I am missing how to implemented.
Any ideas? thank you in advance
First time I checked mormot was 5 years ago. The Mormot v1 was a huge investment to start using it. I gave up.
Two years ago, with the 2nd version, I started to read the code and it was really helpful to have different units as it is now.
The documentation is probably the main problem for a new adopter now.
Dividing mormot to different repositories is not going to help more. The increased time it will need, it is better to be invested in more documentation with more examples.
mormot.core.os.security line 1947 and 2056
[dcc32 Error] mormot.core.os.security.pas(1947): E2010 Incompatible types: 'PAnsiChar' and 'PUtf8Char'
Lazarus does not have rtti for records, so you should inform mormot about typeinfo
Check the function: TRttiJson.RegisterFromText
But I am more worried about the fact that Pavel did not send some news from Ukraine since a lot of time.
I believe he is ok. Probably busy with UnityBase
mormot.net.client, line 2232
wraNegotiate:
fOnAuthorize := OnAuthorizeSspi; // as AuthorizeSspiUser()
Gives error in compiling if DOMAINRESTAUTH is not set
thank you a lot @ab
Win10 and I am receiving a RetrieveListJSON output with about 14000 records
How can I make mormot http server to compress the request response?
I am getting this exception but I have not managed to find any problem during runtime:
https://gist.github.com/dkounal/d1a0b1c … ad54746744
The only code that comes from TPCHRclient.setJWTheader is:
245: function TPCHRclient.setJWTheader(Sender: TRestClientUri): boolean; var b:Boolean; st:RawUtf8;
246: begin st:=''; repeat b:=jwl.TryReadLock; if b then try
247: if (Jexp>0) and (Jexp>{UnixTimeUtc}(DateTimeToUnixTime(Now)+jsdif)) then st:=jwth;
248: finally jwl.ReadUnLock; end else Sleep(20); until b;
249: result:=st<>''; if result then sender.SessionHttpHeader:=st; end;
Any contribution is welcome
in case I have a json rawutf8 string (e.g. from a recordsavejson ) when is better to use rawjson and when variant as a parameter type when calling an interface function in a mormot http server?
Thank you a lot @ab.
Array of variant does not work as a parameter when calling a http interface function of a mormot http server.
I remember I read somewhere that the solution is to transform it to tdocvariant and send it as json
Is that correct?
If so, is there a way to do replace it fast?
I can probably replace the array of Variant with an rawutf8 parameter in the functions and then:
a) use _arr(['aaa',1,5]) to get the a json rawutf8 parameter
b) use _JsonFast or DocList functions inside the called http inteface function to get the array members
Do you propose something else?
Delphi 12.1
file mormot.crypt.openssl
line 897 and line 966: E2250 There is no overloaded version of 'AlgoName' that can be called with these arguments
Trying to have a separate TsqlDBZeosConnectionProperties per Trestserverdb I noticed that the server is really lacking available connections. Every Trestserverdb seems to consumes about 20 connections of the external database in a MariaDB server. So, I come back and search the forum to understand. I have some questions and please correct me if needed, in the following statements for a service with https interfaces that do the database works using one or more TrestServerDB instances:
1. Each TrestServerDB instance has a thread to do reading ORM tasks in the DB and a thread to do write ORM tasks in the DB. This is controlled by setting a TRestServerAcquireMode option in each TRestServerUriContextCommand context e.g. TrestserverDB.AcquireExecutionMode[execOrmGet]:=amBackgroundThread;
What I can not understand is how execSoaByInterface will be used. I have a mormot http server that exposes interfaces (derived from TInjectableObjectRest) and all interfaces' methods use a main TrestServerDB instance to do the DB orm tasks. Can different interfaces derived from TInjectableObjectRest use the same one TrestServerDB instance without problems?
2. In past you have proposed me @ab to use AcquireExecution[execOrmWrite].Safe and lock it before doing a db orm task. This stopped nearly all sql errors with desynchronization in zeos but it is much slower than without it. The idea now was to use different TrestServerDB instances but even that it is not very efficient for the database server.
3. I have found a TSqlDBConnectionPropertiesThreadSafe.EndCurrentThread that you also recommended in the past. How this helps, and when should it be used?
4. How to create a json string from a couple of orms to be saved as a batch and put it in a list and send all the batches to the db in a single thread every 100ms? I am using more than 100 different ormclasses that can be used and write records in the db. An what about blobs that could exist in Torm classes? And what will happen if two clients send the same orm record with the same ID to be added? How can I control if an ORM record with an ID is already inserted in the batch? Probably I need a list for Torms to be added and a Tbatch to add only updates/deletes
Yes, I thought the pool creates new dB connections.
So, I have to have a TsqlDBZeosConnectionProperties per Trestserverdb.
Thank you @ab I will report back if I find a problem
I' m getting an exception when using Trestbatch arising from mormot.db.sql line 3702: "SharedTransaction(sessionID=1) with mixed connections: TsqlDBZeosConnection and TsqlDBZeosConnection"
I am using Mariadb with Zeos and I have implemented multiple TrestServerDB instances connecting to the same external database in order to be used by different threads. It was proposed in a past discussion for better performance.
Should I use the same TrestServerDB instance for all batches or is there something else I missed?
For single inserts/updates, it works ok with autocommit=true, but I have problem with batches.
I have a TSynThreadPool that runs all works with batches in TSynThreadPoolWorkThread derived thread.
Should I have a dedicated TrestServerDB instance for each TSynThreadPoolWorkThread to be used for batches?
Just to note, that I am using predefined IDvalue for inserting orm objects, so the autonumbering by mormot for primary key is not a problem
Thank you in advance
Kabiri, I undesrstand you but FKs is a great way to protect yout database against inconsistence. If mormot could grant this integrity will be great. this will remove of programmer many responsability. We know that are programmers and programmers.
Do you use zeos with mormot to connect to the database?
For me, every sql error with zeos and mariadb can result to unpredictable situations, that zeos framework will not resolve.
So, I prefer not to use database's "protections"
I updated the libmariadb.dll to latest version from here (https://dlm.mariadb.com/browse/c_connector/) and I can not reproduce it for the moment.
So, probably this was the problem. I will inform if something new.
Thank you @ab
No. Now I will do.
I am getting exception in the following zeos code in a connection to a mariadb database:
File ZDbcMySqlStatement, line 3201
procedure TZMariaDBBatchDMLPreparedStatement.SetNullArray(
ParameterIndex: Integer; const SQLType: TZSQLType; const Value;
const VariantType: TZVariantType);
var
Bind: PMYSQL_aligned_BIND;
aArray: PZArray;
I: Integer;
begin
inherited SetNullArray(ParameterIndex, SQLType, Value, VariantType);
{$IFNDEF GENERIC_INDEX}
ParameterIndex := ParameterIndex - 1;
{$ENDIF}
{$R-}
Bind := @FMYSQL_aligned_BINDs[ParameterIndex];
{$IFDEF RangeCheckEnabled}{$R+}{$ENDIF}
if (FMYSQL_STMT = nil) then
InternalRealPrepare;
if (FMYSQL_STMT = nil) then
raise EZSQLException.Create(SFailedtoPrepareStmt);
aArray := BindList[ParameterIndex].Value;
if Pointer(Value) = nil
then FillChar(Bind^.indicators^, BatchDMLArrayCount, Char(MySQLNullIndicatorMatrix[False, FUseDefaults]))
else for i := 0 to BatchDMLArrayCount -1 do
{$R-}
Bind^.indicators[I] := MySQLNullIndicatorMatrix[(Bind^.indicators[i] = Ord(STMT_INDICATOR_NULL)) or IsNullFromArray(aArray, I), FUseDefaults];
{$IFDEF RangeCheckEnabled}{$R+}{$ENDIF}
end;
The above is called by mormot.db.sql.zeos line 1085 TZeosArrayBinding.Create(aStatement: TSqlDBZeosStatement) called by TSqlDBZeosStatement.ExecutePrepared for a Trestbatch
The problem above seems to be that Bind^.indicators is null when having an exception with ParameterIndex=0
Is there a way to overcome it?
Thank you in advance
The first error "ECryptCert {Message:"TCryptCertX509.Generate: GeneratePrivateKey failed error"}" is likely to be a timeout issue when generating a RSA keypai.
Is the Win32 emulation somewhat slow for the computing demanding tasks (like RSA prime search)?
No, I could say that it is more fast than I expected. As it is the iPhone emulator in the Mac.
The other issues seem to be related to floating-point issues.
My guess is that Win32 has poor floating point emulation.
As I understood I have to move to 64bit to have it to work ok
Nothing unexpected....
I compiled mormot2 tests using Delphi 12 and FPC 3.2.3 both for w32 & w64 targets
w64 targets finished without errors
w32 targets finished with errors that are available to this gist: https://gist.github.com/dkounal/1887a0e … 8001bf9f27
I compiled as usual a w32 app using delphi 11.3 that has last snapshot from mormot2 in Github and I run it in a Mac mini inside a VM with win 11 arm edition.
I downloaded the SQLite3 command tool for windows x64 and I tested with it.
To be honest I am curious to run the mormot2 tests inside this VM. I will do it
What is "Windows RISC" ?
Apologies... my mistake. Windows 11 Arm version
I can run a VM with windows 11 native arm version for Apple's risc processor
All x86-x64 applications can run inside it with "transparent" emulation. The OS is native, but the application can be translated/emulated
I can confirm that the same happens using an external database with virtual tables and TrestserverDB, so it should not be mormot's problem
I downloaded the SQLITE version 3.44.2 tools and from the command line tool, the query executes ok in the win11 Arm cmd prompt
I am not sure if it is a problem of win11 emulation or a bug in the code of sqlite. Anyway, I broke the query to two queries and it work ok now. I will follow it in the future updates.
I found something really crazy running windows (risc version) in VMware in Mac. Let me explain:
I have a windows 32bit executable using last github mormot2 compiled with Delphi 11.3 that runs the following query in a TRestServerDB object (memory type based on TOrmVirtualTableBinary):
memserver.Orm.RetrievelistJson(TOrmApiIC11, 'RowID in (select distinct (f.value->"$.ic11") as ic from apicols p, json_each(p.ic11s) f where p.RowID='+toutf8(_pr)+') ORDER BY code', '(code || " " || title) as title');
The above works ok in a windows machine.
Now in a Mac, I install the free VmWare Fusion personal edition and then I installed windows 11 Risc edition. The same windows executable does not gives the same results in this enviroment.
I tried to test the following queries and the returned results are OK:
memserver.Orm.executeJson([TOrmApiCols], 'select distinct (f.value->"$.ic11") as ic from apicols p, json_each(p.ic11s) f where p.RowID='+toutf8(_pr)+') ORDER BY code');
memserver.Orm.RetrievelistJson(TOrmApiIC11, 'RowID in (1930,1931) ORDER BY code', '(code || " " || title) as title');
Same results I am having also using Parallels
I am thinking that this should be sqlite's problem and not Mormot. Am I right?
Thank you in advance
No problem. I did in an other way. Thank you
@ab do you consider the above change as acceptable for the Delete as it works for MembereExists?
Thank you in advance
I just post it because MemberExists(TOrmsometing,0), a ORM method, accept it and it works
I tried also to run the tests without errors after changing TRestStorageExternal.EngineDelete to be like the following :
function TRestStorageExternal.EngineDelete(TableModelIndex: integer;
ID: TID): boolean;
begin
if (ID < 0) or
(TableModelIndex < 0) or
(Model.Tables[TableModelIndex] <> fStoredClass) then
result := false
I know that ID:0 has a meaning for Add/AddOrUpdate in Delete as in MemberExists I am not sure.
I am not sure also about using Delete in batches
I have noticed that the following works:
if MemberExists(TOrmsometing,0) then ....
but the following does not:
Delete(TOrmsometing,0)
Is it possible in mormot.orm.sql line 1317 to allow zero IDs to be deleted through orm?
function TRestStorageExternal.EngineDelete(TableModelIndex: integer;
ID: TID): boolean;
begin
if (ID <= 0) or
(TableModelIndex < 0) or
(Model.Tables[TableModelIndex] <> fStoredClass) then
result := false
Thank you in advance
Thank you a lot @ab
Is there a way to have the create sql query from CreateMissingTables to modify it and create the table in a way it will not cause trouble to mormot?
Is it possible to have sqlite generated columns as described here for an Torm in order to be used in queries?
https://www.sqlite.org/gencol.html
Which is the proposed way to create them (if it is possible)?
I did not write the above this to state the "confusion". I wrote it to help.
All the time, I am requesting 'RowID" because that works everywhere. All methods like MultiFieldValues, onefieldvalues, etc follow that.
I am using ExecuteJSON only to deal with json fields using json_each and conditions in where and believe me I love it, it is very powerful.
Probably the main reason I am using Mormot is the possibility to use everything you need. I have already used XData & DMVC in the past.
I did not write anything to blame your work
I tried to request the same with different approaches from two different TRestServerDB, one sqlite memory tables (Srv1) and one that maps to a Mariadb database(Srv2).
I noticed the following differences:
a. Using RetrieveListJson and MultiFieldValues, the Srv1 does not recognize the field name 'ID' (raises exception) and only works with field name 'RowID'. In contrast, Srv2 accepts without exceptions both field names 'ID' and 'RowID' but it always returns the field name 'ID' in json.
b. Running a query with ExecuteJson([Tormtable],'select * from table where field=1'), the Srv1 does NOT return the RowID field in json, in contrast the Srv2 returns it as 'ID'
c. Using again ExecuteJson the query 'select RowID from table' works for both Srv1 and Srv2, but Srv1 returns RowID and Srv2 returns ID as row id field name.
d. Using again ExecuteJson the query 'select ID from table' works only for Srv2, and Srv1 raises exception.
The above are helpful with what we expect to be returned.
Can you provide performance results for Freepascal too?
direct number eg. 2 did not work
Any news about this bug?
I believe it still exists.
Please use zeos from here: https://github.com/marsupilami79/zeoslib
tree: 8.0 patches
Tell me if you continue to have problems