You are not logged in.
Can you provide performance results for Freepascal too?
direct number eg. 2 did not work
Any news about this bug?
I believe it still exists.
Please use zeos from here: https://github.com/marsupilami79/zeoslib
tree: 8.0 patches
Tell me if you continue to have problems
Thank you ab , I have seen the special handle from the source and that is the reason for my question.
The last paragraph, is something I do not know in pascal: How can I map a record variable like myclassdata.myid to fID in Tormmyclass?
Thank you again
I'm not sure then where serialization fails, is it when you have a property that writes to a record member or when you have a property that is a record?
Seems you want to have a published properties that are members of record in order to easier exchange those records, that's deinitely non standard behaviour, so it may work or not.
I would suggest to simplify design, define your orm object with fields it needs and have, use simple types for fields, (string, integer) and Variant (json object) where more complex data is required or when you need to exchange data among few classes.
Published properties even when their content is read/written to record members, works like a charm
My problem and my question is about TOrmMyclass's ID that it is double saved to fID and myclassdata.myid
During unserialization with FillFrom and other routines, the SetfID is not used.
I am wondering if a Rtti custom method is used for setting fID and how to change it to use my SetfID procedure
AFAIK only published variables will be serialezed, your record is public variable.
Do note that you can use TDocVariantData instead of record, it's much more powerful and sometimes equally fast since it's stored as Variant and that can also be copied by reference.
My record's selected members can be also published properties and being serialized from TOrm (as happens with "othervar" in the above example). TDocVariantData has the same problem with the TOrmMyclass's ID, so the question is the same for it too.
Why do you want to do it this way? Does not the following also meet your requirements:
I am already using this way, too. But copying a record is much more fast and you can pass it also as a pointer if needed (in a Torm with multiple records)
Let me explain better the question
Here is a TOrm derived definition:
type
Dmyclass=record myid:TID; othervar:PtrInt; end;
TOrmMyclass=Class(TOrm)
private
function GetfID: TID;
procedure SetfID(const Value: TID);
public
myclassdata:Dmyclass;
class function createxml(_pat: TOrmapipatient; author:ixmldocument; const visdata:Dapidocvisit):rawutf8;
class procedure InitializeTable(const Server: IRestOrmServer; const FieldName: RawUtf8; Options: TOrmInitializeTableOptions); override;
property IDValue: TID read GetfID write SetfID;
property ID: TID read GetID write SetfID;
published
property othervar: PtrInt read myclassdata.othervar write myclassdata.othervar;
end;
function TOrmMyclass.GetfID: TID;
begin
result:=myclassdata.myid;
end;
procedure TOrmMyclass.SetfID(const Value: TID);
begin
fID:=Value;
myclassdata.myid:=value;
end;
I am using this record to be able to copy data from one TOrmMyclass instance to an other by just copying the myclassdata record variable between them.
The problem is that if am using TOrmMyclass.FillFrom to fill TOrmMyclass's instance data from JSON, the ID is filled correctly but not the myclassdata.myid
Can this be possible? Should I change something in the Rtti for that, and how?
Thank you in advance
Sure, it was very stupid of myself
b is a local variable inside a rest function. b is used in a TOrmmyclass.OrmProps.FieldBitsFromCsv(bity,bi,b) to set what fields will be retrieved.
In one special case inside the algorithm b was not initialized and for a reason I can not understand all the times the function was used, it got the value 64.
The above value as boolean was shown as true and only if you try to transform it to integer, this value is shown
Finding this case's missing initialization, and set the initial value to true, it was fixed.
The problem was finally in my code. The provided variable b content was the problem
Apologies for the trouble
Changing
n := ord(W.withID);
to
n := ord(integer(W.withID)<>0);
works OK
Delphi 11.3 32bit
procedure TOrmPropertiesAbstract.SetJsonWriterColumnNames(W: TOrmWriter; KnownRowsCount: integer);
var
i, n, nf: PtrInt;
begin
// set col names
nf := Length(W.Fields); -->15
n := ord(W.withID); -->64
SetLength(W.ColNames, nf + n);
if W.withID then
W.ColNames[0] := ROWID_TXT; // works for both normal and FTS3 records
for i := 0 to nf - 1 do -->
begin
W.ColNames[n] := Fields.List[W.Fields[i]].Name;
inc(n);
end;
// write or init field names for appropriate JSON Expand
W.AddColumns(KnownRowsCount); -->KnownRowsCount=0
end;
At the end n=79
I have above the values, the variables get
Mariadb is used as database
ID, 'yid', 'erid', 'dat', 'rid', 'xxID', 'typ', 'person', 'sim', 'pd', 'uvd', 'aym', 'ser', 'endata', 'uFdata', 'user' are correctly the published TormMyclass's properties, the same for table's fields
the ee object has ok its data resulting from a Torm.retrieve
I believe the problem is in mormot.orm.base TOrmPropertiesAbstract.SetJsonWriterColumnNames(W: TOrmWriter; KnownRowsCount: integer);
n := ord(W.withID); produces a n with value 64
Empty fields are not mine.
MAX_SQLFIELDS_128 is set for other torms
I am trying to use the following code:
var b:boolean=true;
bi:TFieldBits=ALL_FIELDS;
ee:TormMyclass
begin
ee.GetJsonValues(true,b,bi);
I am getting exception in TResultsWriter.AddColumns(aKnownRowsCount: integer);
in line 2824:
len := PStrLen(PAnsiChar(c^) - _STRLEN)^; // ColNames[] <> ''
Colnames is:
('"RowID":', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', 'yid', 'erid', 'dat', 'rid', 'xxID', 'typ', 'person', 'sim', 'pd', 'uvd', 'aym', 'ser', 'endata', 'uFdata', 'user')
Thank you in advance
Delphi 11.3 patch 1
I tried today to include a part my code with mormot2 in a delphi package.
During compile I am getting an error [dcc32 Fatal Error] mormot.db.raw.sqlite3.pas(1): F2084 Internal Error: GH8402 and a message box that it can not find the file in the package directory path(????)
I have tried to included it as a unit with its directory path in uses list, but the error persists
I have read problems concerning include it in a dll, but is it free of problems with a delphi package?
Ok, that makes it clear.
So in windows when using the TsynServer (for a client service) we should provide as parameter the -c, but in linux, which is better, -f or -c, for agelize?
Thank you a lot @ab
PS by the way, I noticed that all parameters described as '/parameter' in /new of angelize, it runs them as '\parameter'
Trying to use the Angelize, I noticed that a default settings file is created when it runs for the first time.
When adding the first child service, the Servicename of angelize is changed to the name of the first child service.
That causes trouble if it is already installed with an other name, by modifying the main angelize settings file: you can not uninstall it, as the angelize service name has been changed.
Probably this change in the angelize servicename should not happen if the main angelize sevice is already installed as a service
Also I did not understand how to deal with the following case:
I have a child windows service executable that I want to be monitored/controlled by agl.exe.
I am adding it with agl's /new parameter and after that I install it as windows service by calling agl.exe with /install parameter?
With the above, the agl is installed as a windows service with the service name of the child service and the child windows service is not installed and can not be run.
How do you propose to deal with?
Thank you in advance
I want to sort the elements of IKeyValue by Key, and I don't know any good way to do that.
IKeyValue is a replacement for TDictionary to search and get fast a Value instance. I do not think you should use it if you want to have a list of instances sorted.
If you want a list of instances you can sort in multiple ways you can use ilist<> and assigned a compare method
With Delphi 11.3, last commit, it runs without problem, inside IDE and standalone both for win32 and win64.
In the past, I had such a problem with win64 when the tests run with debugging inside IDE.
Try to remove last #13#10 in headers variable.
I returned back my code to check and I found that now it is not needed even that.
The same code from repository that did not work yesterday morning, it works today for linux.
The only change, is that I upgraded last night fpcupdeluxe and I did also an update to the same fixes version of FPC+Lazarus.
To be honest I am a bit afraid to use fpc....
PS I always do clean, delete and compile. Debugger does not work without that.
Thank you all again
To finish with this thread,
using thttprequest and curl does not works from linux compiled with FPC returning error 56.
Same code compiled with FPC for win64 works OK in windows.
Using the above gist code from inside mormot works ok for linux with linux
Headers were tested for both #13#10 and #10
I will try to check it again in the weekend.
Thank you all for the help. At least with curl it works now.
One more hint using WININET (again now.....) in THttpRequest.Request:
If you include Content-type header in InHeader parameter it is included in the request. If InDataType is set, content-type there is not recognized by my Oracle solution web server I have to deal with
Delphi 11.3
Win11
I am experiencing also corrupted result content from mormot2's http client, if non-latin characters are included in the returned text.
Probably it happens with the http calls that do not return content-length, but I did not have such problems with WINET and if using directly the curl library
it does not work. I think the problem is in mormot.net.client, line 1616
if ctxt.OutStream <> nil then
begin
if (Http.ContentLength > 0) and
(ctxt.Status in [HTTP_SUCCESS, HTTP_PARTIALCONTENT]) then
begin
if ctxt.OutStream.InheritsFrom(TStreamRedirect) then
TStreamRedirect(ctxt.OutStream).ExpectedSize :=
fRangeStart + Http.ContentLength;
GetBody(ctxt.OutStream)
end;
end
else
GetBody(nil);
if stream is defined and header content-length does not exist, it will never call GetBody
I can probably use a TRawBytesStream to get a stream and not bothering you.... I will try it
forget it.....
It was my fault.
No need for something like that
Nope. The result is 301.
So you need to set
h.RedirectMax := 10;
before the request to enable redirection.
Then we got a 200 response, but with no chunked encoding flag, nor content-length.
https://www.rfc-editor.org/rfc/rfc7230#section-3.3.3 states it should read all data until the server closes the connection.
This is a pretty inefficient way of handling the content size, and void the whole HTTP/1.1 interrest - this server is awfully coded for sure, and react as a HTTP/1.0 server.Previous version was only reading till the connection close with HTTP/1.0 - now I have enabled it for HTTP/1.1 also.
See https://github.com/synopse/mORMot2/commit/ddb7c8d4
@ab you probably remember the above case.
it was impossible to get correct encoding with an http call when using:
var s:RawByteString; op:THttpClientSocket;
op.Request(url,'GET',_KeepAlive,headers,'','',true,nil{,ContentStream});
s:=op.content;
so I decided to use the call with a stream:
THttpClientSocket.Request(url,'GET',_KeepAlive,headers,'','',true,nil,ContentStream);
I noticed that the above fix does not work with stream parameter and the stream returns empty.
Is it possible also to declare the default encoding for received content if an oracle web server like the one I am using does not return an encoding flag?
Thank you in advance
Thank you very much for your answers
The following code works OK in Ubuntu when running from an interface in mormot:
https://gist.github.com/dkounal/b8ab592 … 7536a31623
I tried the +#13+#10, error still exists.
Do you recommend something else?
I will try now the THttpClientSocket. Or should I use the TSimpleHttpClient?
Is it possible to use compression with THttpClientSocket?
No and thats a good idea. I will test it. Thank you
I am using thttprequest from mormot2 inside a service to get response from an external API and I am using Tcurlhttp to do the connection (plus the USELIBCURL directive in the project's optionst)
The same code is compiled with delphi (windows) and FPC(Linux Ubuntu).
In windows it works OK with libcurl.dll v8.1.2.0 but in ubuntu linux (v7.81.0) I am getting the following error:
libcurl error 56 (Failure when receiving data from the peer)
I do not believe it is the difference between the curl version, because running the request from command line in linux, it works and it does not provide any error
The problem has probably to do with the input headers I add, as removing a header gives responce back.
I am adding the following way the headers:
headers:='Authorization: Basic '+BinToBase64(uname+':'+passw)+#10+'api-key: '+applkey+#10;
for n:=0 to high(headername) do
headers:=headers+headername[n]+': '+headervalue[n]+#10;
I noticed that leaving Authorization only, or api-key only, I get no curl error, but the usual response page from the rest server requesting both. The rest server I am contacting is not mormot.
it seems not be an FPC problem, compiled in windows 64bit with FPC it works OK.
Does anyone can imagine a reason for this behavior of curl in linux when used through mormot?
@ab is mormot2 using its own code-library for curl, and not ubuntu's curl?
Thank you in advance
Does the Tcurlhttp supoorts authentication based on info in thttprequest?
One possible solution, I was thinking is, before adding a list of records to the TrestBatch, to create a comma delimited string (CSV) with their predefined primary keys and run an sql query like the following:
SELECT RowID from table where RowID in (Csv of ID)
In such a case I can have the existing records in the db, with just one query, to decide if I am going to use INSERT or UPDATE. And I can use Trestbatch with multi-records insert.
Do you recommend something better?
I was wondering if it is possible to have an IOrm.Replace. That will help even when not using Trestbatch in mysql/Mariadb
As I can see the function EncodeAsSqlPrepared does the real job in mormot.orm.base line 3412
For Mysql it is just a change to start with 'Replace' instead of 'INSERT' but it is better to run it as upsert (INSERT ..... ON DUPLICATE KEY UPDATE .....
For Postgres/SQLITE it is a real upsert query.
ID primary key must be excluded from the update part of the upsert, and probably a parameter in IOrm.Replace can add more fields as conflict columns.
Is is possible @ab?
Thank you in advance
I am using Postgresql as external database and with TbatchRest I can not use the boInsertOrReplace option (as it is allowed with Mysql) due to different SQL syntax.
@ab recommends to use TbatchRest even in the server side, as it is more fast using multiple records inserts in one sql command.
Sometimes, a number of records in the batch could exist already in the db table and need an update. Or, we do not know if a record with the same primary key exists already and we have to replace it.
How do you deal with this situation in the server side?
With Mysql I was very satisfied as boInsertOrReplace and Mormot do the job with just a change in the first word in the query.
Now, with postgresql, I am not sure what is better....
Include delete commands in the batch for all records that they are going to be inserted after that?
How do you deal with that situation and what do you propose?
Edit: The records to be inserted have a predefined primary ID and it is not filled by mormot or the datase
In your code above, you don't define any client, so I don't understand how you may have any disconnection error.
So using TRestServerFullMemory instance in TmwbCallback's creation is not the cause. I will try to debug it, to see what happens.
Thanks a lot @ab
Please follow the forum rules, and don't post code in the forum itself, but in a separated gist/pastebin.
I apologize, I try not to post more that 20 lines of code before using gist.
I am trying to transform a legacy desktop application to a service based on Mormot2 and a GUI windows application
To have asynchronous communication between GUI and Service I am using websockets following the rest-websockets example in ex dir of mormot2
The problem is that this transformation happens in small steps. During this transformation I am using a TRestServerFullMemory with the method interfaces without an http server inside the legacy application
What should I use as client rest interface when creating the callback interface?
callback := TLongWorkCallback.Create(Client, ILongWorkCallback);
I am using the following:
ImwbCallback = interface(IInvokable)
['{E59F752F-E961-4493-95A8-A5302CD1D857}']
procedure LogMain(const typ:Integer; const msg:string);
end;
ImwbService = interface(IInvokable)
['{BC6964B7-3D10-4EFB-962E-45348AD71DA9}']
procedure StartWork(const PrID: int64; const onFinish: ImwbCallback);
end;
TmwbCallback = class(TInterfacedCallback, ImwbCallback)
protected
procedure LogMain(const typ:Integer; const msg:string);
end;
begin
memsrv:=TRestServerFullMemory.Create(TOrmModel.Create([],rooturl));
memsrv.Model.Owner:=memsrv;
memsrv.server.CreateMissingTables;
memsrv.ServiceDefine(TmwbService,[ImwbService],sicshared).ByPassAuthentication:=true;
memsrv.Resolve(ImwbService,wbService);
callback:=TmwbCallback.Create(memsrv, ImwbCallback);
wbservice.StartWork(myid,callback);
The above seems to work and I am getting callbacks, but there are cases where error is produced saying the client has been disconnected.
What can cause this "disconnection"? if the above is "illegal" what should I do to have callbacks that will work the following day when my app's transformation is finished?
Thank you in advance
I think Trestserverdb.StatementLastException can provide you the sql error that you should return to the client
As I can understand from mormot.net.sock line 580, I have to use USE_OPENSSL for callbacks to be used. Is it needed, so?
Should I include also libssl-3.dll and libcrypto-3.dll or it can statically linked?
Thank you in advance
I am getting the following errors trying to access a Mormot2 server from a windows 10 machine acting as client:
Exception: 20230503 05435100 ! EXC ENetSock {Message:"THttpClientWebSockets.DoTlsAfter: TLS failed [ESChannel <www.mydomain.com>: ComputeAndSendAnswer returned 80092012 [2148081682], System Error 1397 [Mutual Authentication failed. The server's password is out of date at the domain controller]]"} [Main] at ce1539 mormot.net.sock.pas ......
https://gist.github.com/dkounal/c83413c … 9161984290
The server is behind an nginx with TLS works OK from other clients. With IgnoreTlsCertificateErrors=True the above problem is fixed.
A custom root certificate is needed to be installed in windows for some sites during using classic web browsers in this windows terminal which produces this error as client.
Does it worth to have an event callback to check the certificate provided (and not ignoring all errors) and it that practically possible?
Thank you a lot @ab I will try it
Somewhere in the forum, I read that if a database connection becomes unusable (eg. out of sync messages, transactions problems, etc) it is better to close it.
Is it possible to that for connections used by TrestServerDB? Is it possible to request to refresh all connections of its connection pool?
The database connection could be through zeos or a TSqlDBPostgresConnection
Thank you in advace
As I can see it is returned from Tormtable escaped
I did the following:
var t:ormtable;
v:variant;
n:integer;
t.ToDocVariant(v,false);
for n:=0 to v._count-1 do
v.value[n].dedo:=_jsonfast(v.value[n].dedo);
With the above the v has the correct view
Is there a better or safer way?
@dcount
Thanks for sharing. Even if this code expects string = UnicodeString which is not cross-compiler - so it is Delphi only.
I am missing something here..... I have the same code being compiled in Delphi and FPC. is something really bad?
Some of your helpers are pretty weird, like the one using a temporary string then copy() on it with passed parameters. I don't see how it may work in practice because the indexes expects UTF-16 but you only have a UTF-8 string at hand.
I am using greek text that with utf8 you can not know where to cut it. So, for cases that I know the place to be cut in delphi string but I have a rawutf8 I using this
Also the use of PUtf8Char() over a RawUtf8 is pretty inefficient, because it will make a temporary string allocation.
I changed the PUtf8Char to pointer().
I am using a tormtable result from MultiFieldValues
Can I have a little help more?
Thank you in advance
I was really afraid to present it..... but it help to learn!
Thanks @ab
Record is registered with TRttiJson.RegisterFromText
I tried to have the property dedo as RawJSON but again it gets escaped
Do you recommend an other way to have it without been escaped?