You are not logged in.
I can not reproduce it anymore. probably it had a connection with the uppercase table names that you fix the same day. Now RawJson works ok.
Thank you @ab
Thanks a lot @ab. Please fix also the RawJson type for FireDac.
I am using RawJson in case I want a Json Array orm property not to be a BloB file in the database with setter/getter and DynarrayJSONLoad/save
please add ctid to the reserved column names when using TSqlDBPostgresConnectionProperties
it can not be used as a property in an Orm definition
In a Torm that has a property of type RawJson I am getting an error when adding an object in the database:
ExceptionMessage="[FireDAC][Phys][MySQL]-335. Parameter [LA] data type is unknown. Hint: specify TFDParam.DataType or assign TFDParam value before Prepare/Execute call"
Investigating mormot.db.rad, line 586
DatasetBindSqlParam(arrndx, p, fParams[p]);
I noticed that the VType for this RawJson Field in fParams is ftNull. changing during debugging to ftutf8 it saves the TOrm object in the database without error
Thank you a lot @ab
@ab do you propose something for the above?
It works OK after removing the "Uppercase" in Mormot2's mormot.rad.firedac and I do not have any special config in Mariadb
fixed.
Thank you a lot @ab
Delphi 11.2 after last commits:
[dcc32 Error] mormot.net.sock.windows.inc(1162): E2010 Incompatible types: 'Winapi.Windows.PCCERT_CONTEXT' and 'mormot.core.os.PCCERT_CONTEXT'
[dcc32 Error] mormot.net.sock.windows.inc(1335): E2010 Incompatible types: 'mormot.core.os.PCCERT_CONTEXT' and 'Winapi.Windows.PCCERT_CONTEXT'
supporting and following
Trying to use firedac from a windows client to connect to a Mysql database in a unix server I am getting errors that a table (with name in upper case) does not exist, and trying to create it I am getting errors. the table exists in the filesystem in lowercase.
I tried to fix it without success by adding
var dbmormot: TSqlDBFireDACConnectionProperties;
with dbmormot.Parameters do begin Add('MetaCaseInsCat=True'); Add('MetaCaseIns=True'); end;and also:
procedure checkconnection(Sender: TSqlDBConnection; Event: TOnSqlDBProcessEvent);
begin
if Event<>speCreated then exit;
with TSqlDBFireDACConnection(Sender).Database do begin
Params.Add('MetaCaseInsCat=True');
Params.Add('MetaCaseIns=True');
end;I noticed that mormot2 in mormot.rad.firedac line 332,
meta.ObjectName := Utf8ToString(UpperCase(aTableName));
or in line 367
TableName := Utf8ToString(UpperCase(aTableName));
I remove the Uppercase in both places and it works.
How do you propose me to deal with it? Am I missing an other configuration tunning point?
Thank you in advace
I find I find a small bug:
The first argument in a helper MUST be a variable from Json.
If not, only the first argument exists in the Variant passed to the helper
I was wondering if there is a way to pass the content of a template variable as one argument in a helper.
Something like the following:
{{helpername value1,value2,{{otherVariable}}}}it just works like that:
{{helpername value1,value2,otherVariable}}where otherVariable exists in the json context as name in a name:value pair
I can not miss your tutorials and I enjoy reading and testing them.
The exciting that lead me to start this thread is that a helper can take an array of arguments and that helps a lot because with just one helper you can include optional "modifiers" arguments to the response of the helper
Mustache in Mormot is really helpful, and a not so well documented feature is even more helpful:
In mormot's mustache we could have helpers where a function in our compiled code is called when the helper found in the template like the following:
{{helpername value_todealwith}}A procedure like the following is called that it is predefined and must be added/delete with TSynMustache.HelperAdd/HelperDelete or better with TSynMustache.HelpersGetStandardList that will add also the standard helpers.
procedure(const Value: variant; out Result: variant) of object;The exciting feature shown in the mormot's source is that the variant value passed to the helper can be a comma delimited array of values:
{{helpername value1,value2,value3}}An easy way to parse them, as you can read it in the source for existing standard helpers is:
procedure healper(const Value: variant; out Result: variant);
var dv: PDocVariantData; wasString: boolean; work2do:RawUtf8;
begin SetVariantNull(result{%H-});
if not _SafeArray(Value, dv) then exit;
if dv^.count=2 then
VariantToUtf8(dv^.Values[1], work2do, wasString);
...Thank you a lot @ab for this framework
Looking the source and using the debugger to trace, when using TSQLDBZEOSConnectionProperties, this is derived from TSqlDBConnectionPropertiesThreadSafe and together with Mariadb, this should not be a problem
Reading documentation...: https://synopse.info/files/html/Synopse … ml#TITL_25 in 11.7.1
I have added the following after creating TrestserverDB:
srv.AcquireExecutionMode[execOrmGet]:=amBackgroundThread;
srv.AcquireExecutionMode[execOrmWrite]:=amBackgroundThread;
Also I added in the end of the thread from TSynBackgroundTimer:
dbmormot.EndCurrentThread;
The above existed inside my code and was removed in the past.
Now I do not have the above errors but I still have seldomly errors like the following I reported in the past and probably it is a problem from zeos
https://synopse.info/forum/viewtopic.php?id=6376
My best wishes for the remaining of the week
I started with mormot2 and firedac a year ago. In the mean time I switched to zeos to have the same code base for FPC.
I did not delete the old code with firedac, I just used preprocessor conditionals to keep the firedac use.
Yesterday, I tried to use my codebase that works ok with zeos and mariadb, switching to firedac: TSqlDBFireDACConnectionProperties instead of TSQLDBZEOSConnectionProperties
The above was the first problem. Today I noticed that with firedac, mormot2 when starting, wants to add a database field from Orm that already exists as TinyInt instead of BigInt.
I just want to report that zeos and Firedac are not interchangeable under same common conditions
I am using the latest commit from Github. Problem started after commit 5e6f3685
One of the three cases that do not allow to be compiled is the following:
i := SortFind(pointer(Value), aID, Count);
Removing the pointer substitution it compiles OK:
i := SortFind(Value, aID, Count);
Why do we need the pointer?
I have read the following:
https://synopse.info/forum/viewtopic.php?id=3352
https://synopse.info/forum/viewtopic.php?id=5788
I am still not sure if an interface function should create or not, each time called, a TrestserverDB that maps tables in a external database. I am not using user authentication and sessions.
Can the same TrestserverDB instance (with tables mapped to an external database) be accessed from multiple threads? Should I use a lock when accessing it for read/write?
I have not understand the use of TSynDictionary in the above discussion, do you mean that a thread can pop up a TrestserverDB instance from TSynDictionary, during an interface call and re-insert it when finished?
Thank you in advance
mormot.orm.base line 10566
[dcc32 Error] mormot.orm.base.pas(10566): E2010 Incompatible types: 'TOrmCacheEntryValueDynArray' and 'Pointer'
Ok, so probably most of them can be changed to mediumtext to work with Batches.
Thank you @ab
Trying to use Firedac in a trestbatch I am getting the following error:
Invalid call to TSqlDBFireDACStatement.BindArray(Param=36,Type=ftUnknown)
The exception comes from procedure TSqlDBStatement.BindArray(Param: integer; ParamType: TSqlDBFieldType; const Values: TRawUtf8DynArray; ValuesCount: integer);
in line 5938 of mormot.db.sql where Paramtype is ftUnknown
Looking the Orm class, this parameter is a rawutf8 and in the external mariadb database it is a Longtext field. Changing it to MediumText, the error does not exist anymore
mm=TOrmModel.Create([Tormtable1, Tormtable2],'url');
OrmMapExternal(mm,Tormtable1,dbmormot,'table1');
OrmMapExternal(mm,Tormtable2,dbmormot,'table2');
srv:=TRestServerDB.Create(mm);
srv will be used by different threads. Is this my mistake?
Can I use also from different threads the following?
dbmormot.executeinlined('an sql command',false);
This is the result from a procedure running from a TSynBackgroundTimer using zeos 8.0-fixes trunk and MariaDB
after a fresh start of my program, no exception was reported previously from TsynLog
Exception OS: 20230124 11362514 ! EXCOS EAccessViolation (c0000005) [pchr BgdTmr] at 7328b162
ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.InternalRealPrepare (1441)
ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.Prepare (697)
ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.ExecuteQueryPrepared (1048)
mormot.db.sql.zeos.pas TSqlDBZeosStatement.ExecutePrepared (1179)
mormot.orm.sql.pas TRestStorageExternal.ExecuteDirect (1798)
mormot.orm.sql.pas TRestStorageExternal.TableHasRows (1443)
mormot.orm.server.pas TRestOrmServer.TableHasRows (1846)
mormot.core.threads.pas TSynBackgroundTimer.EverySecond (2481)
mormot.core.threads.pas TSynBackgroundThreadProcess.ExecuteLoop (2387)
mormot.core.threads.pas TSynBackgroundThreadAbstract.Execute (2075)
Zeos connection is created like the following, where zopts has only the location of LibMariadb.dll (32bit):
myzurl:=TZURL.Create('zdbc:mysql://',server,port,db,rootuser,rootpass,zopts);
dbmormot:=TSQLDBZEOSConnectionProperties.CreateWithZURL(myzurl,dMySQL,true);
dbmormot.ConnectionTimeOutMinutes:=450;
dbmormot.DeleteConnectionInOwnThread:=true;
I usually find a lot of exceptions that have to do with multithreading use of zeos connectio like the following:
https://gist.github.com/dkounal/095e653 … 48eb24ead5
Is something needed to zeos to work in threadsafe mode?
Thank you in advance
Date: Fri, 6 Jan 2023 18:05:43 +0100 (CET)
From: Michael Van Canneyt <michael@freepascal.org>
To: Lazarus mailing list <lazarus@lists.lazarus-ide.org>, FPC
development mailing list <fpc-devel@lists.freepascal.org>
Subject: [Lazarus] Unicode RTL for FPC
Hello,
I'm currently working for a company (Tixeo.com) that is preparing to use FPC to recompile their flagship product (written in Delphi) for certain targets.
As part of this work, we're striving to make the Free Pascal RTL and Packages more compatible with recent Delphis.
That means:
- String = UnicodeString, Char=WideChar
- Dotted units.
Before you all get a heart attack:
Because backwards compatibility is important, FPC will of course continue to distribute a backwards-compatible RTL and packages, with single-byte string. All FPC modes will continue to function.
To make this possible a feature called 'subtargets' has been implemented.This can be used for other things than the unicode RTL, but will be used forthe unicode RTL.
The RTL is compiled 'normally' and with subtarget=unicodertl; this will result in 2 separate sets of .o/.ppu files:
The normal CPU-OS directory and a CPU-OS-unicodertl directory.
The first part of the work has been pushed to gitlab in a branch called unicodertl (NOT the svn/unicodertl branch):
- subtarget support is there
- The rtl compiles with make SUB_TARGET=unicodertl with the compiler as it is in that branch.
Needless to say, this is a major change that will need thorough testing. The earlier testing starts, the better.
For those that wish to help in testing:
- Update your git clone
- switch to branch unicodertl.
- in the toplevel FPC directory, do a
make all
- if that went well, next to your fpc.cfg, create a file fpc-unicodertl.cfg with the following contents:
----
-dUNICODERTL
-Municodestrings
----
- to create a Unicode RTL, in the rtl directory do a make clean all SUB_TARGET=unicodertl PP=path/to/the/new/compiler
- if that worked, you can try then a make install SUB_TARGET=unicodertl
Note that this is NOT ready for production use:
- The packages have not yet been converted (working on this now) and will certainly fail...
- a private testsuite has been run and gives no failures, but the complete compiler testsuite still needs to be examined.
- Dotted names will be created after the unicode transition is deemed complete.
if you do wish to test and spot errors in the RTL or compiler, please file a bugreport.
You can use a tag 'UnicodeRTL' to report bugs for this.
Michael.
Also Tevent is needed if Puremormot2 is set in mormot.core.threads
https://github.com/synopse/mORMot2/comm … 19ff2e806e
would change it into a TRWLock which will allow parallel search in the cache, without any spinning.
Mormot.rest.server needs also 3 renames from Trestcache to Tormcache
Thanks a lot @ab, it works now
p^.Value.Kind is rkBool
p^.Prop^ is shown as TRttiProp but I can not have any of each properties in the debugger, and evaluating it as expression in the debugger I am getting "TRttiProp ()"
A mormot2 daemon does not follow the configuration after an update. I using an inherited class from TSynDaemonSettings with extra properties in the published section and the config file is an INI file.
The same source compiled with Delphi it works ok, but with FPC all boolean directives are taking all the false value. I do not know from which version of Mormot it started to happen.
My FPC version is Lazarus 2.2.4 (rev lazarus_2_2_4-0-g06a78ed923) FPC 3.2.2 x86_64-win64-win32/win64 and even with last versions it is the same.
I tried for the first time the debugger from FPC and I found that the problem should be in line 4243 from mormot.core.data, "if p^.Prop^.SetValueText(Instance, v) then" but even with O0 optimization the debugger shows only assembly when calling this function and it is impossible for me to follow.
string and integers work without problem
Any ideas? Thank you in advance
You had right, I tried to find out what caused this error and I found it:
I had a THttpClientSocket object with an unauthorized interface service to create a JWT when needed without destroying and recreating the service interface. If more than 65 secs passed without using it and a request is done after that, using it, the above exception is created.
Thank you a lot @ab. It works perfectly now with the JWTs from the client side.
using OnAuthentificationFailed with result (true/false) and changing TRestClientUri.SessionHttpHeader it does not work
Also, SessionHttpHeader is added to headers once before the loop of OnAuthentificationFailed
Thank you @ab, I have already searched the above forum threads.
I have already created the server side implementation and it works OK
My problem now is the client. From the above links probably you mean the https://synopse.info/forum/viewtopic.ph … 051#p30051 but it is besed on sessions as far as I can understand
Do you propose a solution without sessions?
Using THttpClientSocket to an interface service, I am getting the following in the log, but nothing is catch in except sections in my program, nor any unexpected attitude is trapped from my program.
Is it an internal http error that it is trapped and retried transparently to my code? if so, how can I exclude it from the logs?
Thank you in advance
20221202 07020721 EXC ENetSock {Message:"THttpClientSocket.SockRecvLn error 10053 after 0 chars"} [xxdv0 BgdTmr] at 3be6e5
20221202 07042923 EXC ENetSock {Message:"THttpClientSocket.SockRecvLn error 10053 after 0 chars"} [xxdv0 BgdTmr] at 3be6e5
I am using a TRestHttpClientWinHttp to call inteface services in a Mormot2 server and I am not using sessions.
As far as I can understand, Mormot2 does not support JWT by internal design. There is no TRestClientAuthentication for JWTs.
In the server, I am checking if JWT header exists and if it is valid in each interface call
In the client, is it possible to use OnAuthentificationFailed event to refresh the expired JWT that the client should send in each interface call?
If I implement a timer based renew function running in the background, how can I protect the client from accessing the SessionHttpHeader during the timeframe it is set by the renew function?
Edit:
Is there an OnCall event that runs before a service inteface call is made by a TRestHttpClientWinHttp ? I noticed that SessionHttpHeader is added to headers once before the loop of OnAuthentificationFailed
Thank you in advance
Also:
[dcc32 Error] mormot.app.daemon.pas(868): E2016 Array type required
mormot #30 & #52 position for the moment
Just two errors during compile in Delphi 11.2 to note for future correction
File: mormot.crypt.openssl
[dcc32 Error] mormot.crypt.openssl.pas(2776): E2251 Ambiguous overloaded call to 'BinToHex'
System.Classes.pas(14010): Related method: procedure BinToHex(Pointer; PWideChar; Integer);
System.Classes.pas(14077): Related method: procedure BinToHex(Pointer; PAnsiChar; Integer);
The second probably based on FPC
File: mormot.app.daemon line 753:
fStopper := [low(fStopper) .. high(fStopper)];
Yes, thanks a lot @ab!
Example httpServerRaw with FPCUSEVERSIONINFO enabled in FPC 3.2
Verbose: Compiling C:\delphi\G\mORMot2\src\app\mormot.app.console.pas
Verbose: Compiling C:\delphi\G\mORMot2\src\core\mormot.core.base.pas
Verbose: Compiling C:\delphi\G\mORMot2\src\core\mormot.core.os.pas
mormot.core.os.posix.inc(31,9) Error: Duplicate identifier "FileInfo"
mormot.core.os.posix.inc(31,9) Hint: Identifier already defined in mormot.core.os.pas at line 2602
mormot.core.os.pas(8357,0) Verbose: There were 1 errors compiling module, stopping
Enabling Define FPCUSEVERSIONINFO in mormot.defines.inc causes problem in compilation with FPC and Linux x86 as target
mormot.core.os.posix.inc(31,9) Error: Duplicate identifier "FileInfo"
The problem might be that there is a function also named Fileinfo as the pascal unit.
The service daemon with the last updates works perfectly. I noticed that if you just run from the command line the service executable without options I can log an exception from the windows' Tservice but it is harmless.
Another point that needs to be handled better in windows is the platform independent commands in mormot.core.os to run an executable from a mormot server (RunProcess, RunCommand, etc)
In my scenario, the service periodically looks for updates. If it finds an available update, it downloads it and runs an other executable to do the update, the ServiceUpdater.
I have created this executable, the ServiceUpdater, using mormot framework that stops the service, unzips and replaces the executable and re-runs the service.
This executable does not work, if the services runs it. It waits for the service to stop till timeout.
Looking the code, I noticed that all mormot's commands from windows are based on Createprocess where the host process can not end before the executed process
I change the mormot's RunProcess/RunCommand to ShellExecute with command 'open' and now the service stops and the ServiceUpdater can continue to do the update.
Probably it could be included for windows. I will test this situation also in linux.
#34 the mormot for the moment. I saw it also in #52
OK Thanks a lot
is it better to use TRestServerDB's orm interface directly ?
that's how you're supposed to use it. The only reason you can still call Add/Update/Delete directly from the TRestServerDB instance is because it's there for backwards compatibility with mORMot 1. If you define PUREMORMOT2 flag for your project then you will see those direct methods no longer available.
Thank you for response. I apologize, I am not sure I understand the response
Do you mean that I should use a TrestClientDB.Create(aRunningServer: TRestServerDB) ?
I already have enabled PUREMORMOT2, I create a TRestServerDB instance before the http server and I can still have access to TRestServerDB.orm.add
Looking an example from mormot2 I noticed that for standalone applications it exists also TRestClientDB that can be created to connect to external databases also through SQLITE.
I noticed that TrestClientDB can be created also with the following contructor:
/// initialize the class, for an existing TRestServerDB
// - the client TOrmModel will be cloned from the server's one
// - the TRestServerDB and TSqlDatabase instances won't be managed by the
// client, but will access directly to the server
constructor Create(aRunningServer: TRestServerDB); reintroduce; overload;Having all past problems with multiple threads trying to add to a database new records or updating others I am wondering:
When having multiple threads to use a TRestServerDB instance for the same tables, is it better to use TRestServerDB's orm interface directly ?
Or use one TrestClientDB instance for each thread with the above constructor called with the same TrestServerDB instance ?
What are the pros/cons?
Thank you in advance
PS I can not find the TSQLRestClientRedirect somewhere. Is it implemented?
Note: To help better understand my case:
I have clients that call a rest interface procedure but for a work that will take time and requests info from other API servers too. So the REST request is finished and the job is entered in a TSynThreadPool. So, later a thread will have to do all database work to provide the response through websockets. So, the above discussion is about threads running from a TSynThreadPool and accessing a main-central TRestServerDB instance.
I believe that a REST interface procedure in a server called by a client will not have problems to access a main-central TRestServerDB instance to do database work
Thank you a lot @ab
We have a windows service that runs a second executable that it is the real mormot2 server
The service has the responsibility to automatically update the mormot2 server executable when an update by the developer is available.
What will be a reliable way to update it?
We want to stop receiving new requests till the existing are served and closed, to terminate the server and the mormot2 server executable and to update the executable and restart it.
Killing the windows process is not a solution as I think. What do you propose?
Creating a rest function to request to finish? And if mormot2 is in dead-lock situation? And how do you find that a mormot2 server windows process is exited? By trying to delete or to modify the mormot2 executable as file?
In linux, we have the systemd for a number of linux distos
Do you run mormot2 server executable as a process controlled by systemd (to rely to databases being ready for example) or by eg rc.local?
How do you start it? Using -f or -r switch in TsynDaemon?
How do you stop it? Using SIGSTOP?
Again how is it better to look for updates and auto-update?
Do you run (and how do you initiate it) an other script to stop systemd service, update and restart it?
Let's consider that the update host and procedure to receive the updated version is safe discussing about security considerations
Thank you in advance
Is it possible to set the SectionName when loadin/saving in ini format the TSynDaemonSettings ?
Both IniToObject/ObjectToIni have this option and it needs just to be set in TSynJsonFileSettings.LoadFromJson and TSynDaemon.Create
Thank you in advance
Thank you @ab, it works now