You are not logged in.
My test result for Zeos without mORMot is also depressive:
WriteLn(ZReadOnlyQuery1.FieldDefs[i].Size); // 80
WriteLn(ZReadOnlyQuery1.FieldDefs[i].Precision); // 80while ODBC + TSQLDBConnectionProperties + PostgreSQL is ok.
maybe for Zeos somehow is returned maximum possible count of bytes for UTF8 string:
https://www.postgresql.org/docs/current … ibyte.html
So you receive ColumnLength=80 whereas you expect 20?
yes
I do not know where the error comes from (TSQLDBConnectionProperties.SQLGetField sounds correct), but I guess this is not an issue.
With a PostgreSQL database, you may just ignore the text field length, and consider it without any field limit...
See https://www.postgresql.org/docs/9.3/sta … acter.html
For ODBC (checked in our SynDBExplorer for the same DB) the value of ColumnLength is correct (20). Maybe Zeos bug?
Hi!
we have small problem for TSQLDBZEOSConnectionProperties:
Props := TSQLDBZEOSConnectionProperties.Create(Format('postgresql://%s:%s',[eHost.Text,ePort.Text]), eDBName.Text, eUser.Text, ePasswd.Text);
Prop.GetFieldDefinitions('testtable',Fields,true);
WriteLn(RawUTF8ArrayToCSV(Fields,#13#10' ')); for table like this (I use db first approach)
CREATE TABLE testtable
(
id next_num_global_id NOT NULL,
test character varying(20) NOT NULLwill print:
id [int8 8 0 0] *
test [varchar 80 0 0] *My proposition:

@edwinsn it is still FPC, NewPascal is just fork name with full credits for FPC and with full backward compatibility for the official FPC. We need to somehow distinguish between FPC and the FPC dedicated for our purposes.
github.io is worth to consider
, seems good for me.
Mentioned zip file is special zip - included Lazarus should be ultra easy to run.
To omit confusion and for better identification, new IP is needed. I have domain newpascal.org and I'd like to create modern small start page to link github, zip packages to download and mORMot forum (see below ab).
We are Pascal users so name like Pascal++ is wrong. I think New(Pascal); is what we want. ![]()
@ab
as you know licensing is always my top priority (my discussion about mORMot LGPL license), so library like Orca is rejected by definition. I agree with all points.
I think I can help in topic "how to start work on compiler", maybe some small PDF is good idea.
If all "compiler changes" are marked with comments like:
original_fpc_code;
{ new pascal compiler begin }
our_changes;
{ new pascal compiler end }
original_fpc_code;then maintenance of fork with merging is quite simple. A simple and effective trick used by Typhon developers (I don't like them but this is quite good approach as addition for WinMerge or for other tools). Additional "mark comment" was extremely usefully for my Sparta project.
Any of my compiler change is full backward compatible and optional (unlike ARC objects in Delphi). If you like NewPascal name I think is good to have also more generic conditional define than FPC_HAS_*, I mean NPC (beside the FPC).
I have one question:
Is possible to use part of mORMot forum as "home forum" for our pascal fork? I think the best place is just below "mORMot Framework" subforum.
@AOG
The most frustrating was Sven for management operators. All was developed as he wanted, with Florian cooperation. For final release he told me in shortcut : "sorry I changed my mind, I won't include your changes in trunk"... Fortunately Florian has different opinion than Sven. I have much more examples but I think the more important is to keep positive energy.
AOG you have good ideas, but please give me time.
All of those *.zip can be done automatically for major platforms on github for each commit or for tags. I'd like to finish few scripts for github as our start point. When I take all together I will give you (ab and AOG) access as admins.
We don't need any installer, ever. We can have unique solution because is possible to create zip (Lazarus + FPC + mORMot) which can be unpacked in custom directory (for example is possible to get full individual IDE + Compiler + mORMot for each project on single machine). It requires dark magic knowledge (without improper tricks) but is quite simple.
Additionally after our FPC fork, I can publish simple Lazarus "pendrive" edition used in my company ^^. (for now windows only)
@ab I am working on this already!
"Invoke" topic was raised in 10.02.2013 and in 05.02.2015 - It is almost 3 years... Nothing will change. I'd like to omit strong words so let me not commenting Sven reasoning and core team. It is very frustrating.
I think we need fresh air and MORE POSITIVE energy around FPC/Pascal. Your project inspired me to many open source work (!). Today I have for "our" mormotish FPC compiler few extensions (all already works!):
- management operators
- nullable types / optional types
- new kind of "specialized helpers" for types like interfaces, dynamic arrays etc.
- more usable variant record
- smart pointers
- arc objects
I need just few days to publish this (few tests), let me do that fork on github together with merge for other forks (RTTI for interfaces + RTTI attributes + Generics.Collections + few bug fixes for compiler + mentioned above changes).
About 3.0.2, it may be the occasion to merge the interface RTTI branch to the FPC trunk.
I tried to make this request to the FPC-dev list.
I think it is pointless. That may happen somewhere around 2030-2040. I have terrible experience with ready to merge Generics.Collections which is waiting 2 years (!), RTTI branch (attributes only) 3 years (!).
... ![]()
Thanks for links, I will start my work soon.
http://www.cromis.net/blog/2013/02/tany … container/
TAnyValue was my inspiration for management operators
(Initialize,Finalize,Copy,Clone):
AOG and ab - amazing
. Can I use code from mORMot to implement Invoke function for RTTI module for FPC?
Lazarus is different than Delphi. You need to add additional path to mORMot directory (place where Synopse.inc is located) in:
Project->Project Options->Compiler Options->Paths ; in field "Include files"
I'd like to use WebSockets on my VPS Linux
.
Project->Project Options->Compiler Options->Paths ; in field "Other unit files"
Yay ;D the power of mORMot is scary. All of my mobile app on Android can be used as server...
I agree but is easily to forget about that ![]()
@ab not at all, especially for the result:
For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result.
that won't work for console/service application (eventually "Classes.CheckSynchronize" call is needed but I like to avoid that)
Sometimes I need to create new threads inside rest server methods to perform some actions. For example I'd like to return as soon as possible, so part of work can be redirected to separated thread. Additional thread isn't executed always. I wonder, maybe there is some nice technique in mormot to auto free the threads? I'd like to perform auto free when thread "Execute" was ended.
At the moment I have for that purpose my own structures.
Where is mORmot logo? I'd like to use sticker "build with mORMot" ![]()
Maybe some picture with marmot is good idea for that purpose:
New important bug for RawByteString is fixed:
http://bugs.freepascal.org/view.php?id=30082
fixed in FPC trunk r33597
Notice: No difference for AES-CSPRNG (mORMot v1.18.2589) and for current FPC trunk (r33539). Empty body and ESynCrypto still occurs.
I like your proposition. Idea of TSQLRecordProperties.FieldBitsFromExcludingCSV() method is very useful but... In the case of "DB-First" approach for existing project AddWithExclusion is common functionality.
With your suggestion:
GIID_Server.Add(iidrec, TSQLiid.RecordProps.FieldBitsFromExcludingCSV('a,b,c', soInsert));looks much more complicated than:
GIID_Server.AddWithExclusion(iidrec, 'a,b,c');AddWithExclusion like method is used in my project very often even more often than standard Add.
Hi ab!
I have another problem related to previous patch "Ignoring columns for DB-First" (http://synopse.info/forum/viewtopic.php?id=3252)
when table is really big (even 120 columns) very often more convenient is to exclude selected fields instead of typing all required.
Patch attached:
+AddWithExclusion to create a new member, excluding selected fields
I see that in SynLZcompress1pas is mentioned size (255 + 16) similar to block presented in bug2 (272 bytes one $00 byte and 255 + 16 $FF bytes). Maybe is some nonstandard bitwise-operations in FPC for ARM (http://stackoverflow.com/questions/9044 … ions-to-me). I can try to get asm arm code for SynLZcompress1pas .
Additionally all regression tests associated with SynLZ compression and Sha/Aes encryption pass without error.
I can provide ASM for ARM generated by FPC, but maybe is no sense for that... Seems like is almost impossible to find what is wrong :\ . The last what I can do is to update mORMot to latest versions of SynCrypto and latest FPC trunk.
I can't reproduce bug outside production case.
Thanks for info. I am a little depressed. I need to find what is wrong. Maybe it is some FPC bug :\. Anyway we need wait to tomorrow.
Btw. maybe is worth to use custom random instead of system random for:
{ code from SynCrypto }
Head.SomeSalt := random(MaxInt);as is stated in FPC doc:
The Free Pascal implementation of the Random routine uses the Mersenne Twister to simulate randomness. This implementation has a better statistical distribution than for example a Linear Congruential generator algorithm, but is considerably slower than the latter. If speed is an issue, then alternate random number generators should be considered.
after small research I think that the buggy is EncryptPKCS7Buffer for FPC ARM (SynCrypto.pas) (so probably it is not related to threads). Bug 2 has more sense in this context.
I see that is used Random (maybe this is our bug, for some random values combination?). Maybe usage of
Td0, Td1, Td2, Td3, Te0, Te1, Te2, Te3: array[byte] of cardinal;is somehow wrong.
I will try to create test on my ARM/Android device with many requests to server...
Btw. I can confirm since the hcSynShaAes is disabled, all works fine in restaurant... 0 errors.
Maybe I need to lock ALL TSQLHttpClientWinSock, maybe TSQLHttpClientWinSock per thread is bad approach ?
Are you sure you make a thread-safe use of TSQLHttpClientWinSock
Yes I'am 100% sure. Why? Because all wrongly encoded data comes from GClientUpdate, which is used only in one place in whole project in special thread.
LValue := ObjectToJSON(LRequest);
// the only line where is used GClientUpdate (not counting the declaration, constructor and destructor)
LCodeError := GClientUpdate.CallBack(mPOST, 'SyncRequest', LValue, LResult); as was showed in first topic LValue is always valid, but encoded data sent to server is empty or strange $00 $FF $FF ... $FF block.
The thread is created only once at app start in static constructor so GClientUpdate.CallBack is 100% thread-safe it is never created twice.
, and that your server side is also thread-safe?
I think yes, anyway the problem is on client side as was showed in first post:
Valid OutContent -> call CompressDataAndGetHeaders(fCompressHeader,fCompress,OutContentType,OutContent); -> OutContent is empty or filled by $00 $FF $FF ... $FF.
but this is true only for hcSynShaAes. Since hcSynShaAes is disabled on client side I have the longest period without errors ever.
hypothetical even if something is wrong on server side AFAIK that can't have affect on CompressDataAndGetHeaders on client side in presented way.
ab wrote:Did you try to disable hcSynShaAes?
No. I will do that tomorrow. The acquisition of new data can take a while.
With disabled hcSynShaAes works well. 0 errors. ![]()
When hcSynShaAes is enabled on client side, I am almost sure that CompressDataAndGetHeaders is colliding with part where data are received from another thread from another TSQLHttpClientWinSock instance.
Some important details (maybe that can help to reproduce bug):
For Server side:
aHttpServer := TMyHttpServer.Create(IntToStr(settings.httpPort),[aRestServer],'+',useHttpApiRegisteringURI, 7, secSynShaAes);For my server is used lower "ServerThreadPoolCount" (I need to limit connections to PostgreSQL DB). I think that has important role in bug on client side, because for many requests for many clients some delay may occur (single client can have few connections: TSQLHttpClientWinSock per thread).
For Client side:
GClient: TSQLHttpClientWinSock = nil;
GClientUpdate: TSQLHttpClientWinSock = nil;
GClientPing: TSQLHttpClientWinSock = nil;
GClientIID: TSQLHttpClientWinSock = nil; When I've added new instance of TSQLHttpClientWinSock the bug occurs more often. For example GClientIID is used only to find our bug (but that additional instance of TSQLHttpClientWinSock is important to get bug more often) :
fIID_Process := TSynBackgroundThreadProcess.Create('IID process', OnIID, 10000);
...
procedure TMainForm.OnIID(Sender: TSynBackgroundThreadProcess;
Event: TWaitResult);
var LCodeError: Integer;
LResult: RawUTF8;
begin
LCodeError := GClientIID.CallBack(mPOST, 'SyncIID', '', LResult); I think when data are received and decompressed somewhere in GClientIID.CallBack and in the same time other data in other thread (for example by GClientUpdate) are compressed by CompressDataAndGetHeaders for send purpose then bug occurs.
I do not understand how to reproduce it.
I too can't reproduce this :\. This is log/data from production. The output of CompressDataAndWriteHeaders generally is ok. Maybe some parts of CompressDataAndGetHeaders are not thread safe?
Perhaps the presented data (especcialy second hex data) are some kind of tip?
Is the problem on the server side?
Server is protected. It can ignore bad data from clients. Anyway mORMot is critical part of my system and I need to fix that.
You state it occurs on the client side.
Do you mean on FPC Android ARM target?
Yes.
We do not use this platform here, so it would be difficult to reproduce...
Even acquiring data for this topic was difficult. I don't have direct access to devices. It is big restaurant...
Which version of FPC do you use?
FPC trunk r33326
Did you try to disable hcSynShaAes?
No. I will do that tomorrow. The acquisition of new data can take a while.
Do the regression tests associated with SynLZ compression and Sha/Aes encryption pass on the FPC Android ARM target?
No. That is not that simple.
to be more precise the bug exist in THttpSocket.CompressDataAndWriteHeaders. After investigation I found very strange things. (topic is related to http://synopse.info/forum/viewtopic.php?id=3223 )
How it was debugged? Small modification for THttpSocket.CompressDataAndWriteHeaders in SynCrtSock:
procedure THttpSocket.CompressDataAndWriteHeaders(const OutContentType: SockString;
var OutContent: SockString);
var OutContentEncoding: SockString;
{$IFDEF MORMOT_TEST}
iid: TSQLclient_iid;
id: Int64;
{$ENDIF}
begin
{$IFDEF MORMOT_TEST}
try
iid := TSQLclient_iid.Create;
iid.iid := GUIDToRawUTF8(RandomGUID);
GIIDLock.Lock;
if GIID <> nil then
begin
iid.creation_time := DBServerDateTime;
iid.xkasy := GV_XKASA_ID;
if OutContent <> '' then begin
SetLength(TSQLclient_iidAccess(iid).fraw_body, Length(OutContent));
Move(OutContent[1], TSQLclient_iidAccess(iid).fraw_body[1], Length(OutContent));
end;
id := GIID.AddWithBlobs(iid);
end;
{$ENDIF}
if integer(fCompressHeader)<>0 then begin
OutContentEncoding := CompressDataAndGetHeaders(fCompressHeader,fCompress,
OutContentType,OutContent);
if OutContentEncoding<>'' then
SockSend(['Content-Encoding: ',OutContentEncoding]);
end;
SockSend(['Content-Length: ',length(OutContent)]); // needed even 0
if (OutContent<>'') and (OutContentType<>'') then
SockSend(['Content-Type: ',OutContentType]);
{$IFDEF MORMOT_TEST}
if GIID <> nil then
begin
if OutContent <> '' then begin
SetLength(TSQLclient_iidAccess(iid).fsynlz_body, Length(OutContent));
Move(OutContent[1], TSQLclient_iidAccess(iid).fsynlz_body[1], Length(OutContent));
GIID.Update(iid, 'synlz_body');
end;
end;
SockSend(['IID: ', iid.iid]);
SockSend(['XCashID: ', GV_XKASA_ID]);
finally
GIIDLock.UnLock;
iid.Free;
end;
{$ENDIF}
end;It was necessary to create some local database (available for my server on request) to get logs because the client was installed on android (ARMv6/7) so debugging process was hard. The chance to get the error is approximately 1/100000 requests -,-.
Bug 1:
For data like this (saved as raw_body):
{"ID":0,"sync_zlecenian_otw":false,"sync_zlecenian_otw_id":0,"zlecenian_otw":[],"zleceniap_otw":[],"wiadomosci_otw":[],"delete_wiadomosci_otw":[],"salarestauracji":0,"stoliki":[],"xkasy_top":0,"sender":"TfrmLogOut","client_inner_version":4,"operator":9999918876,"xkasy":999991380}I got NULL (saved as synlz_body). That is mentioned "inlen=0" in topic http://synopse.info/forum/viewtopic.php?id=3223
Bug 2:
For data like this (saved as raw_body):
{"ID":0,"sync_zlecenian_otw":true,"sync_zlecenian_otw_id":0,"zlecenian_otw":[],"zleceniap_otw":[],"wiadomosci_otw":[],"delete_wiadomosci_otw":[],"salarestauracji":999992049683,"stoliki":[],"xkasy_top":0,"sender":"TTableBillFrm","client_inner_version":4,"operator":9999918876,"xkasy":999991380}I got strange BLOB presented here as HEX view (saved as synlz_body). That is our ESynCrypto on server side {"Message":"TAESCFB.DecryptPKCS7: Invalid content"} in topic http://synopse.info/forum/viewtopic.php?id=3223:
00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff The bug occurs on client side: FPC Android ARM6 HF (hcSynShaAes for TSQLHttpClientWinSock).
@AOG that is the part of mORMot, so I think that only ab is able to do that change
. I'd like to omit usage of fossil without ab knowledge
.
Another note to check: maybe error is caused because for FPC:
SockString = type AnsiString(CP_UTF8);implicit conversion, somewhere...
Horray!
mORMot is now very tasty in including to existing project!
As far as I understand "SendData" parameter is useless for our case?
I need to know whether the patch will be included. This is the key feature for my team, so eventually I need keep this somehow outside the core of mORMot library.
thanks, works as expected.
Patch is ready. To be consistent with standard SQL and with other mORMot parts, I must to say that "excluding" is bad idea. Instead of excluding is used standard approach with CustomFields. All old code will work as before (with small exception for one virtual Add, but IMO this is the must to keep clear code base):
https://drive.google.com/file/d/0B4PZhd … sp=sharing
example usage:
function NewMaleBaby(Client: TSQLRest; const Name,Address: RawUTF8): TID;
var Baby: TSQLBaby; // store a record
begin
Baby := TSQLBaby.Create;
try
Baby.Name := Name;
Baby.Address := Address;
Baby.BirthDate := Date;
Baby.Sex := sMale;
result := Client.Add(Baby,true,'Name,Address,Sex'); // (!!!) BirthDate is filled by DB
// also is possible to call:
// result := Client.Add(Baby,true,'*');
// which is equal to:
// result := Client.Add(Baby,true);
finally
Baby.Free;
end;
end;Thanks ab, nice smart solution, but it is missing additional check for decode fields in TSQLRestStorageExternal.ExecuteFromJSON:
should be
// decode fields
if fEngineAddForcedID = 0 then
Decoder.Decode(SentData,nil,pNonQuoted,InsertedID,true) else
Decoder.Decode(SentData,nil,pNonQuoted,0,true);instead of
// decode fields
Decoder.Decode(SentData,nil,pNonQuoted,InsertedID,true);without modification is raised exception: TSQLRestStorageExternal.JSONDecodedPrepareToSQL(TSQLxxx): No column for "ID" field in table public.xxx
btw. I am working on ignoring columns ![]()
I think is worth to track these FPC threads:
http://free-pascal-general.1045716.n5.n … 24746.html
http://free-pascal-general.1045716.n5.n … 24747.html
Calling *W version is recommended for FPC 3.0 (described as tips for Lazarus but usefully for whole FPC):
http://wiki.freepascal.org/Better_Unico … indows_API
maybe this can help us:
that was fine for beginning of including mORMot in my big project (~600 tables and 6 MLOC) for some tables but the fields are used in many tables for update, select etc. So I need to put these fields in the TSQLRecord definition. The insert is the only case in DB-First approach when I need to omit some fields. I'd like to omit declaring two models for each table...
The new parameter for "Add" seems most logically. I don't see any alternative.
I see another potential lack in InitSynCommonsConversionTables in SynCommons.pas. Maybe is worth to set DefaultSystemCodepage to CP_UTF8 for HASCODEPAGE?
Hi!
I need to implement addition (new parameter for TSQLRest.Add method) to ignore selected columns during adding/inserting new record (I have DB-First approach for existing project where mORMot is introduced). I need to let DB to fill few of my columns (and that can't be changed).
Is any sense to preparing patch for this (because it will be ignored)? I'd also like to fix "Insert new record and Bad Request (400)" ( http://synopse.info/forum/viewtopic.php?id=3245 ).
Thanks for advice. After small redesign, my code seems more solid. Hmmm.
Yes I know. I read it several times and IMO mORMot is incomplete in this matter. In my case it is not "just" defining of "few" getters
Lack of something like "After Read/Fill" is big problem. I don't get why we have ComputeFieldsBeforeWrite and we don't have Compute from other perspective like "ComputeFieldsAfterRead".
It is like saying that I need only "finalize" section without "initialize" because I can do the same things just after main begin keyword >.<. I need equivalent of
procedure Loaded; virtual;from TComponent which is called after loading all properties from dfm stream. Using "getters" is not the same like collective event after "fill".
Not at all, I am the author of EngineAddIgnoreID/OnEngineLockedNextID ... ( http://synopse.info/forum/viewtopic.php?id=2842 )
The main idea of usage/introducing EngineAddIgnoreID is final SQL for ORM Add without ID:
insert into x (a, b) values (1, 2)Important for "database-first" approach and for merging mORMot into existing projects. I need to omit OnEngineLockedNextID, 'select max(ID)' and I can't provide ID.
It works perfect for my projects since 2015-09-01 (was introduced to omit ID in insert) but I got 400 error in my "fiddler logs" which is a little confusing since all is correctly inserted into DB.
I'd like to have comparable feature to existing ComputeFieldsBeforeWrite but for "After Read". Why we have ComputeFieldsBeforeWrite? I see lack in framework for something like "ComputeFieldsAfterRead".
ID must be 0 for EngineAddIgnoreID feature, just look at TSQLRestStorageExternal.EngineLockedNextID (which is called from TSQLRestStorageExternal.EngineAdd)
Because EngineAddIgnoreID is set to true for my table. All works as expected (even 0 returned by EngineAdd) except returned HTML code
. ExecuteORMWrite has no info about used "EngineAddIgnoreID".
Hi,
there is small issue for client/server, for ORM table with EngineAddIgnoreID = true .
when I try to add a new record to the database (by using http server POST), always is returned "Bad Request (400)", despite the proper insert into DB.
Can we somehow get ride of this? I think we can adjust the method TSQLRestServerURIContext.ExecuteORMWrite to receive somehow info about EngineAddIgnoreID for table from TableEngine to return correct value.