You are not logged in.
This a really interesting proposal @ab
As I can understand, you propose to divide parts of an application to different TrestserverDBs (each one with an SQLITE database) if high level processing does not need to access more than one sqlite's tables for its tasks. That will even allow not to need an external database and simplifies backup and other tasks.
The problem is that for special cases, an sql query needs to combine(join) and have read access to more than one sqlite's tables. SQLITE provides an "attach" feature that allows different sqlite databases' tables to be accessible from an other sqlite db as its tables.
Can a TrestserverDB have an sqlite db with attached tables from other TrestserverDB's sqlite? how reliable is that and what is needed for functioning without troubles? That will be an interesting scenario and example.
You must use https (and not http) from a mobile device. That is a common silent failure reason.
Great job!! Thank you!
We need a blog of your "Angel" implementation and the Letsencrypt status too
Thanks a lot @ab
// Sync with compiler internals
TTypeKind = (tkUnknown, tkInteger, tkChar, tkEnumeration, tkFloat,
tkString, tkSet, tkClass, tkMethod, tkWChar, tkLString, tkWString,
tkVariant, tkArray, tkRecord, tkInterface, tkInt64, tkDynArray, tkUString,
tkClassRef, tkPointer, tkProcedure, tkMRecord);
Delphi added the typeinfo type tkMRecord (managed records) that it is not included in TRttiKind and this causes an out of range error and can not be serialized
thanks a lot @ab
Is it possible to know if a connection is websocket upgraded when executing an interface call? (other authentication is not need in such case, it happens on websocket upgrade)
or, can I allow only websocket upgraded connections to an inteface?
Is it stored somehow in ServiceRunningContext.request?
Thank you in advance
The ID is copied from an other api service. The problem is not only produced by the main ID, it can be produced from other unique keys also.
Usually it does not happen. The problem is that if it happens the only way is to restart all the service.
Using firedac I havent managed to log such errors but firedac is unusable with rawjson and it was not possible to test it in the same way.
ps I will try to check it with firedac.
I have not found a solution to that problem and even trying different approaches it still seldom happens and it is very difficult to reproduce it.
From logging I notice that before this problem starts there are log entries with access violation like the following:
Exception OS: 20230406 13312317 & EXCOS EAccessViolation (c0000005) [Main] at 70f7b162 ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.InternalRealPrepare (1441) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.Prepare (697) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.ExecuteQueryPrepared (1048) mormot.db.sql.zeos.pas TSqlDBZeosStatement.ExecutePrepared (1179) mormot.db.sql.pas TSqlDBStatement.ExecutePreparedAndFetchAllAsJson (6682) mormot.orm.sql.pas TRestStorageExternal.EngineList (1403) mormot.orm.server.pas TRestOrmServer.EngineList (1116) mormot.orm.rest.pas TRestOrm.ExecuteJson (1716) mormot.orm.rest.pas TRestOrm.ExecuteList (1699) mormot.orm.rest.pas TRestOrm.MultiFieldValue (983) mormot.orm.rest.pas TRestOrm.OneFieldValue (891)
Looking in zeos's ZDbcMySqlStatement.pas the line 1441 is the last line the following:
procedure TZAbstractMySQLPreparedStatement.InternalRealPrepare;
var I: Integer;
begin
if (FMYSQL_STMT = nil) then
FMYSQL_STMT := FPlainDriver.mysql_stmt_init(FPMYSQL^);Can FPMYSQL, which is the database connection, be invalid after a pool refreshing or a timeout or something like that?
PS this an old thread that should be moved to mormot2
Using zeos and mysql, I am having the following problem:
If an insert to a table happens to have a duplicate key with existing record,
Exception: 20230406 14135223 " EXC EZSQLException {Message:"SQL Error: Duplicate entry '1007806' for key 'PRIMARY' \r\nCode: 1062 SQL: insert into apiorder (ID,blabla,blabla) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"} [] at 01608282 ZDbcMySql.pas TZMySQLConnection.HandleErrorOrWarning (1267) ZDbcMySql.pas TZMySQLConnection.HandleErrorOrWarning (1267) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.ExecuteUpdatePrepared (1150) mormot.db.sql.zeos.pas TSqlDBZeosStatement.ExecutePrepared (1204) mormot.orm.sql.pas TRestStorageExternal.ExecuteFromJson (2144) mormot.orm.sql.pas TRestStorageExternal.EngineAdd (1272) mormot.orm.rest.pas TRestOrm.Add (1754)
after that, in every insert zeos responds:
Exception: 20230406 14135223 " EXC EZSQLException {Message:"SQL Error: Commands out of sync; you can't run this command now \r\nCode: 2014 SQL: insert into apiorder (ID,blabla) values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"} [] at 01608282 ZDbcMySql.pas TZMySQLConnection.HandleErrorOrWarning (1267) ZDbcMySql.pas TZMySQLConnection.HandleErrorOrWarning (1267) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.FlushPendingResults (1227) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.Prepare (693) ZDbcMySqlStatement.pas TZAbstractMySQLPreparedStatement.ExecuteUpdatePrepared (1123) mormot.db.sql.zeos.pas TSqlDBZeosStatement.ExecutePrepared (1204) mormot.orm.sql.pas TRestStorageExternal.ExecuteFromJson (2144) mormot.orm.sql.pas TRestStorageExternal.EngineAdd (1272) mormot.orm.rest.pas TRestOrm.Add (1754)
How to overcome this problem?
Sometimes, even memberexists does not prevent it, even with writelock.
Thank you in advance
That answers my question. Thanks a lot tbo.
I have a Tdocvariant in a variant variable and I want to assign a part of it ("var1") to an other pascal variable (var1) which is a record with the same structure.
If I have understood ok, tdocvariant is a utf8string in reality, and not a real record inside it. Is it like that?
DynArrayLoadJson can do the job but it is a double conversion or am I missing something?
I have a Tdocvariant that has the structure:
{
"var1":[
{
"v1":[
{
"a":"1",
"b":"2"
}
],
"v2":"something"
}
],
"var2":"aaa"
}How can I assigned the var1 part of the above Tdocvariant to a pascal array of the following record type:
type
Dv1=record
a,b:rawutf8;
end;
Dvar1=record
v1:array of Dv1;
v2:rawutf8;
end;
var var1:array of Dvar1;I noticed the following when trying to retrieve orm instances:
I am using the Retrieve:
function Retrieve(const SqlWhere: RawUtf8; Value: TOrm; const aCustomFieldsCsv: RawUtf8 = ''): boolean;
When defining custom return of fileds in aCustomFieldsCsv, the RowID must be included in order to be returned.
In the following function, if RowID is included in CustomCsvFields then the update call fails:
function Update(Value: TOrm; const CustomCsvFields: RawUtf8; DoNotAutoComputeFields: boolean): boolean;
When using TFieldBits, ID/RowID is not included and it is manipulated separately.
The following function helps also not to have an error if included in the input string:
function FieldBitsFromCsv(const aFieldsCsv: RawUtf8; var Bits: TFieldBits; out withID: boolean): boolean;
I surely understand that in an update nobody will include the RowID to be updated, but in Retrieve this difference it is not clear in the documentation
My proposal, is not to change anything in the code. Just include in the Retrieve's comments (to be included in the future in the documentation) that as aCustomFieldsCsv is injected directly in the Select query, RowID must be included if needed.
Thank you @ab for your time and your amazing work.
It started as a copy-paste function. You have right, now I learn from RetrieveDocVariantArray's code, I will change it.
Concerning the problem in this tread, it works now OK and the problem described here https://synopse.info/forum/viewtopic.php?id=6491 is also fixed.
Thank you a lot @ab
I have tested the function leaving just the if clause and the same happens.
I start the service connecting to a postgres db and this is the first call
I have installed wsl2, ubuntu and I was able to have the exeception in this enviroment.
I installed the FPC for linux to be able to trace it and I found that the exception happens in mormot.soa.server line 1510 when calling
WR.SetText(Ctxt.Call.OutBody);and especially, in mormot.core.base, line 11548,
FastAssignUtf8(Text, fDataString) // FPC expects this As far as I can see Text is zero length empty string and FDataString has a value of '[]'.
My method is:
function TinterP.GetOrders(const sdat, edat: rawutf8; const pid: int64): TServiceCustomAnswer;
var ..... sdat_,edat_:RawUtf8; _pid:Int64; v:variant; p:Tid;
begin ..... p:=0; ......
v:=MyTrestserverDB.Orm.RetrieveDocVariantArray(Tormmyorm,'aa','pid=? and sdat>=? AND edat2<=?',[_pid,sdat_,edat_],'RowID,edat,sdat,extra,lastupd',@p);
if p=0 then result:=prepSCA(204,'[]') else result:=prepSCA(200,VariantToUtf8(v.aa));
end;prepSCA is just a function to return a TServiceCustomAnswer with the arguments.
If the query has results everything is OK. If it is empty an exception happens, only in linux
I am using Lazarus 2.2.7 (rev ada7a90f86) FPC 3.2.3 x86_64-linux-gtk2
I was travelling and it was not possible to debug it since now. In win64 target, there is no exception. in Linux 64, it happens when the resulted query from RetrieveDocVariantArray is empty.
I will try to debug it more tomorrow.
I will try it in windows with FPC, as with Delphi it works OK
But I am not sure it is from my code. I do not have many methods and everything else worked for months till now, without even a restart.
The same interface has an other method with exactly the same code as the problematic method. The only difference is a RetrieveDocVariantArray that uses a different Torm table. No other difference.
both search a different table with the same parameters and return the query results.
I changed in the "working" method the Tormtable used in the query and I got the same problem. I finally succeeded to have the problematic method to work by just removing a search Tdatetime parameter in the query that usually causes the result to be an empty query.
Now it works. I will check it again tomorrow.
the problem here: https://synopse.info/forum/viewtopic.php?id=6491 is not fixed. it is now returning a 400 error
Removing the slash from Tmodel.root did not help, nor anything else.
I have now an other method in the other interface that does not work if compiled with FPC. If compiled with Delphi 11.2, it works without any problem
This method can be just a result assignment to return a result through TServiceCustomAnswer but it stills gives an 500 http error.
I am using Lazarus 2.2.5 (rev lazarus_2_2_4-27-g75ee8b4c01) FPC 3.2.3 x86_64-win64-win32/win64 and cross compiling for Linux but no success with fpc.
it does not compile with PUREMORMOT2, but at least, it works
Thank you a lot @ab
Problem exists even with TOrmModel.Root that does not have a slash
Is only one interface allowed per TRestServerFullMemory ? because, I usually have at least two
I think also, that this is the problem for this too: https://synopse.info/forum/viewtopic.php?id=6491
So, it is supported only for client side as mentioned in https://synopse.info/forum/viewtopic.php?id=6338 (first response) ?
20013718 $ - 08.991.005
20230307 20013718 $ ret mormot.rest.server.TRestServerRoutingRest(05404860) { "errorCode":500, "error": {"EOrmException": { "ClassName": "EOrmException", "Address": "cb791a mormot.rest.server.pas TRestServerUriContext.ExecuteCommand (3029)", "Message": "Unexpected Command=0 in TRestServerRoutingRest.Execute" }} }
20230307 20013718 $ srvr ? POST pchr/v8/restInf=500 out=250 B in 8.99s
20230307 20013718 $ debug mormot.rest.memserver.TRestServerFullMemory(052f90c0) TRestServerRoutingRest.Error: { "errorCode":500, "error": {"EOrmException": { "ClassName": "EOrmException", "Address": "cb791a mormot.rest.server.pas TRestServerUriContext.ExecuteCommand (3029)", "Message": "Unexpected Command=0 in TRestServerRoutingRest.Execute" }} }
20230307 20013718 $ EXC EOrmException {Message:"Unexpected Command=0 in TRestServerRoutingRest.Execute"} [R1 pchrv7 pchrv8] at cb791a mormot.rest.server.pas TRestServerUriContext.ExecuteCommand (3029)
20230307 20012819 $ + mormot.rest.memserver.TRestServerFullMemory(052f90c0).URI POST pchr/v8/restInf in=0 B
20230307 20002920 $ - 14.477.594No, with a POST
I believe the problem is because I am using a slash / inside the base URI 'pchr/v8'
I have a mormot 2.1 server and I am calling methods from an interface
Using postman I call the url with just the name of the URL and without the method
ex The complete URL is http://127.0.0.1:10000/pchr/v8/Restinf/method
where iRestinf is the interface and method is the interface method
when calling then URL http://127.0.0.1:10000/pchr/v8/Restinf/ without the method, I am getting an exception both in Delphi debug mode in mormot.rest.server line 3026
The same code with FPC in linux returns :
<body style="font-family:verdana">
<h1>mORMot2 (Linux) Server Error 500</h1>
<hr>
<p>HTTP 500 Internal Server Error<p>EFastReader: Reached End of Input<p><small>mORMot 2Congratulations!!! Many many thanks for your work
Generally from what I can understand, Delphi's Utf8ToString/UTF8Encode are faster than mormot2's utf8tostring/stringtoutf8 for non latin characters but in FPC it is better to use mormot2's functions
I used the stopwatch for FPC from https://stackoverflow.com/questions/139 … tstopwatch as a fast alternative with the same code
In win11 compiled with FPC 3.2.3 x86_64-win64-win32/win64 and win64 target and string is declared as 'unicodestring'
Using string with Greek Characters
string conversion: 7811
rawutf8 conversion: 6274
mormot utf8tostring conversion: 9197
mormot stringtoutf8 conversion: 7457
RTL Utf8ToString conversion: 10735
UTF8Encode conversion: 5055
Finished with Greek Characters
Using string with Latin Characters
string conversion: 3705
rawutf8 conversion: 3279
mormot utf8tostring conversion: 3910
mormot stringtoutf8 conversion: 3735
RTL Utf8ToString conversion: 3853
UTF8Encode conversion: 3810
Finished with Latin Characters
win11 and Delphi 11.2 with win64 target
Using string with Greek Characters
string conversion: 2561
rawutf8 conversion: 3441
mormot utf8tostring conversion: 5563
mormot stringtoutf8 conversion: 4299
RTL Utf8ToString conversion: 2835
UTF8Encode conversion: 3528
Finished with Greek Characters
Using string with Latin Characters
string conversion: 1680
rawutf8 conversion: 1926
mormot utf8tostring conversion: 1514
mormot stringtoutf8 conversion: 2179
RTL Utf8ToString conversion: 1735
UTF8Encode conversion: 1608
Finished with Latin Characters
In win11 compiled with FPC 3.2.3 x86_64-win64-win32/win64 and win64 target and string is declared as 'string'
The results are completely different:
Using string with Greek Characters
string conversion: 2206
rawutf8 conversion: 2065
mormot utf8tostring conversion: 151
mormot stringtoutf8 conversion: 2097
RTL Utf8ToString conversion: 14705
UTF8Encode conversion: 8931
Finished with Greek Characters
Using string with Latin Characters
string conversion: 1996
rawutf8 conversion: 1947
mormot utf8tostring conversion: 152
mormot stringtoutf8 conversion: 2059
RTL Utf8ToString conversion: 5926
UTF8Encode conversion: 6457
Finished with Latin Characters
Using a simple example as in https://gist.github.com/dkounal/b14ce68 … ffbcb60a53
In win11 with Delphi 11.2 (win32target) I got the following results in ms:
Using string with Greek Characters
string conversion: 2141
rawutf8 conversion: 3307
mormot utf8tostring conversion: 5256
mormot stringtoutf8 conversion: 3834
RTL Utf8ToString conversion: 2083
UTF8Encode conversion: 2514
Finished with Greek Characters
Using string with Latin Characters
string conversion: 1259
rawutf8 conversion: 1555
mormot utf8tostring conversion: 1154
mormot stringtoutf8 conversion: 1715
RTL Utf8ToString conversion: 1097
UTF8Encode conversion: 1134
Finished with Latin Characters
Any comments welcome, just for learning purposes. I am not an expert
Coming from the mysql world..... Bang!!! You have right @mpv....
I have to forget the boInsertOrReplace and most of my habits with mormot's Batches....
Thank you a lot
Trying to insert a Batch to postgres using TSqlDBPostgresConnectionProperties, mormot2, windows11 Delphi 11.2, win32( from pg9)-win64(15.2) Postgres 15.2
res:=TRestBatch.Create(DBsrv.Orm,TormMobMetav,1000,[boInsertOrReplace,boExtendedJson]);The SQL query as I can see from mormot.db.sql.postgres, line 730 is:
'replace into pchrdb.m_mdmetav (ID,typ,sid,nam,nam2,nam3,nam4,act,sor) values (unnest(?::int8[]),unnest(?::int8[]),unnest(?::int8[]),unnest(?::text[]),unnest(?::text[]),unnest(?::text[]),unnest(?::text[]),unnest(?::int8[]),unnest(?::int8[]))'I am getting an exception in mormot.db.raw.postgres.pas line 334.
'ERROR: syntax error at or near "replace"'#$A'LINE 1: replace into pchrdb.m_mdmetav (ID,typ,sid,nam,nam2,nam3,nam4...'#$A' ^'#$A
and an errCode that it must be overflowed
that come in the stack from mormot.db.sql.postgres, line 269
PQ.Check(fPGConn, 'Prepare',PQ.Prepare(fPGConn, pointer(aName), pointer(aSql), aParamCount, nil));How to overcome the above? The log is in this gist: https://gist.github.com/dkounal/e394d2a … 5110ea76ee
if the call is repeated, probably due to the previous failure, the prepared statement does not exist in the cache and I am getting:
'ERROR: prepared statement "08" does not exist'#$A
Is it expected to be handled by me or by the framework?
Last round has mormot results
The problem exists only if the RawJson field's contents is not a valid json. If it is '{}' or '[]', no error is produced
In nginx's logs I noticed now from my previous attempts where I used html content type that nginx has logged the following:
2023/02/25 12:05:40 [error] 289105#289105: *2 upstream sent invalid header: "text/html;\x20..." while reading response header from upstream, client: x.x.x.x, server: xxx.yyy.zzz, request: "POST /pchrd/v0/restsys/nicupdate HTTP/1.1", upstream: "http://127.0.0.1:10480/pchrd/v0/restsys/nicupdate", host: "xxx.yyy.zzz"I give up.... All other methods in the same interface work. I tried to rename it without any change
TRestSys = class(TInjectableObjectRest, IRestSys)
public
function nicupdate(const hostname,ip4,ip6:RawUtf8):TServiceCustomAnswer;
end;
....
function TRestSys.nicupdate(const hostname,ip4,ip6: RawUtf8): TServiceCustomAnswer;
begin Result.status:=200; result.header:=JSON_CONTENT_TYPE_HEADER; Result.content:='{"Hello": "there"}'; end; 20230225 10383249 $ + mormot.rest.memserver.TRestServerFullMemory(7f2a2912cf90).URI POST pchrd/v0/Restsys/nicupdate in=19 B
20230225 10383249 $ call mormot.rest.memserver.TRestServerFullMemory(7f2a2912cf90) RestSys.nicupdate["ddd","ddd","ddd"]
20230225 10383249 $ EXCOS EAccessViolation (416a41) [R1pchrdv1pchrdv0] at
20230225 10383254 $ debug mormot.rest.memserver.TRestServerFullMemory(7f2a2912cf90) TRestServerRoutingRest.Error: { "errorCode":500, "error": {"EAccessViolation": { "ClassName": "EAccessViolation", "Address": "7f2a2790d400", "Message": "Access violation" }} }
20230225 10383254 $ srvr Interface POST pchrd/v0/Restsys/nicupdate=500 out=157 B in 86.10ms
20230225 10383254 $ ret mormot.rest.server.TRestServerRoutingRest(7f2a266743d0) { "errorCode":500, "error": {"EAccessViolation": { "ClassName": "EAccessViolation", "Address": "7f2a2790d400", "Message": "Access violation" }} }
20230225 10383254 $ - 00.086.132 I have the following settings in project's properties:
Compilation and Linking:
Optimization Level 0
Nothing else checked
Debugging:
Generate info for the debugger: I tried both Dwarf2 and Dwarf3
Display line numbers checked
Use Heaptrc unit checked
Custom options
-Xm
I am still getting no more info in debug file
What else should I do to enable it?
Thank you in advance
Usually I write and debug in Delphi (windows) and after that I compile in FPC for Linux.
I am having an exception when compiled in Linux by FPC in an interface method and the result (TServiceCustomAnswer) is not returned to the client (try except end inside the interface method is used, nothing caught)
The exception error in the log with the following line of the exception:
20230224 12215901 % EXCOS EAccessViolation (416a31) [R1pchrdv1pchrdv0] atHow a mab is created on linux compiled with FPC? With windows I am using a sample from Tsynlog in mormot1
Ok, thanks a lot for the almost complete code.
But this one store the week array as a blob, how about having it stored as a JSON string? Wouldn't I need to register the array for JSON? And what more?
with best regards
Lars
Arrays are by default stored as blob. Me too, I need them as JSON. The way I am using is:
Declare the field as RawJSON in Torm definition with getter/setter and use DynArraySaveJson/DynArrayLoadJson inside them.
cweekArray:TArray<TWeekArray>;
property weekArray: RawJSON read getWeekArray write setWeekArray;
....
function TORMmyorm.getWeekArray: rawjson;
begin result:=DynArraySaveJson(cweekArray,typeinfo(TArray<TWeekArray>)); end;
procedure TORMmyorm.setWeekArray(const Value: rawjson);
begin if (value='') or (not DynArrayLoadJson(cweekArray,value,typeinfo(TArray<TWeekArray>))) then weekArray:=nil; end;RawJSON is automatically created as TEXT
Thanks a lot @ab
After last commits, if I map existing primary key field of an external database's table to be the 'ID' in mormot2, when I use RetrieveListJson I am getting the original name as primary key and not the 'ID' or 'RowID' in the produced json
I am mapping the existing primary key field name with: ExternalDB.MapField('ID','myfieldname');
Before last commits, RetrieveListJson returned:
["ID":1,....Now it returns:
["myfieldname":1,....The problem start fromlast commit 0f9f409205e340031fec3da8e709ba246dc9e845
It still happen after a reboot, and Lazarus closed (I never though of that case), but it continues
it is problem of my installation probably, so, thank you, I do not want to bother you more.
Can you imagine a reason for the following?
In a new empty VCL project I just add a mormot.core.base in uses and set to be win64 target plaform
After that, even in debug build configuration, the debugger does not stop to any breakpoint.
I usually do not use mormot2 with win64 in Delphi and now I can find a reason for that
It could be something stupid, but ... Can you imagine any reason for that?
Thank you in advance