You are not logged in.
Pages: 1
@AOG
Sure, I have commited the relevant files here: https://github.com/transmogrifix/sx-ddd
But I would not use them for new mORMot2 projects and rather wait for whatever ab is planning DDD/KDD-wise.
I have several projects in production which use mORMot2. As with all new code, a few issues came up (in production of course ) occasionally but were fixed and things are running smoothly now.
The last one being a locking issue in TSynCache which was introduced when RW locks were added.
I was looking forward to RW locks for a long time, it is a great addition. Although this is an example of a feature which can add hard to detect bugs with unit tests, which got me thinking if mORMot2 could use some kind of branching/versioning/whatever strategy for stable and development/bleeding edge?
About DDD:
The projects were all upgraded from mORMot 1 from 6 months to a few months ago and they all used DDD units, it was pretty easy to port mORMot1's DDD units to mORMot2 though (at least the parts I was using relating to persistence, authentication and emails).
I am following your work on that for some time, very nice!
I would like to use it, is it now the recommended server for production?
Thank you for your help and for fixing the issue so fast!
I think it is happening when they expire, my interface services are used by an Angular app only.
It only starts happening after some time after the server is started.
See the following pastebin with the log:
https://pastebin.com/3jpjcpU0
I don't use any callbacks.
I register the services like so:
FMainRest.ServiceRegister(TOrderService, [TypeInfo(IOrderServiceCommand)], sicClientDriven).ResultAsJSONObject := True;
I haven't made any changes to the Angular app or the server in some time, I have only upgraded mORMot2.
Hi ab,
My server (Linux, FPC ) logs are getting filled with:
20220202 11342032 EXCOS EAccessViolation (0f) [Pool31adminauthpublicapi] at 737e00 ../mORMot2/src/soa/mormot.soa.server.pas tservicefactoryserver.instancefree (821) ../mORMot2/src/soa/mormot.soa.server.pas tservicefactoryserver.instancefreegc (842) ../mORMot2/src/soa/mormot.soa.server.pas tservicefactoryserver.retrieveinstance (1045) ../mORMot2/src/soa/mormot.soa.server.pas tservicefactoryserver.executemethod (1354) ../mORMot2/src/rest/mormot.rest.server.pas trestserveruricontext.internalexecutesoabyinterfacecomputeresult (3245) ../mORMot2/src/rest/mormot.rest.server.pas trestserveruricontext.internalexecutesoabyinterface (3298) ../mORMot2/src/rest/mormot.rest.server.pas trestserverroutingrest.executesoabyinterface (4455) ../mORMot2/src/rest/mormot.rest.server.pas trestserveruricontext.executecommand (2941) ../mORMot2/src/rest/mormot.rest.server.pas trestserver.uri (6880) ../mORMot2/src/rest/mormot.rest.http.server.pas tresthttpserver.request (1088) ../mORMot2/src/net/mormot.net.server.pas thttpservergeneric.request (1416) ../mORMot2/src/net/mormot.net.server.pas thttpserver.process (2008) ../mORMot2/src/net/mormot.net.server.pas thttpserversocket.taskprocessbody (2106) ../mORMot2/src/net/mormot.net.server.pas thttpserversocket.taskprocess (2062) ../mORMot2/src/net/mormot.net.server.pas tsynthreadpoolthttpserver.task (2474) ../mORMot2/src/core/mormot.core.threads.pas tsynthreadpoolworkthread.dotask (2834) ../mORMot2/src/core/mormot.core.threads.pas tsynthreadpoolworkthread.execute (2873)
I am on git commit fcc29c07aa0b3585a131bf6c30a2b4681927543b (reviewed all singleton locks), I am not sure when it started but it was also happening with commit f2fd54a8d92a4711acfc6802b25efad2282ffd17 (minor OpenSSL library refactoring)
After a while, the server becomes unusable.
Any idea what the culprit is?
Thanks!
FpDebug does not use gdb, it is a dwarf debugger written in pascal.
See https://wiki.freepascal.org/FpDebug for more info.
In any case, it is currently the best debugger backend for Lazarus (it works the best for me that is).
Hi,
You haven't initialized fLogType. Try using TSynAutoCreateFields as the base class of TMyLog and remove the write part of the logType property.
TMyLog = class(TSynAutoCreateFields)
private
fLogdate: TDate;
fComment: RawUTF8;
fLogType: TLogType;
published
property logDate: TDate read fLogDate write fLogDate;
property comment: RawUTF8 read fComment write fComment;
property logType: TLogType read fLogType;
end;
For the TSQLRecord properties, you have flattened them propertly, but make sure to use the same letter casing.
For example logType.name should be logType_name and not LogType_Name.
The new mORMot is a masterpiece.
Thank you ab!
Hi,
You can exclude the pilSubClassesFlattening flag by overriding TDDDRepositoryRestFactory.GetAggregateRTTIOptions.
I wanted to do the same, so a virtual method was introduced for that purpose. See: https://github.com/synopse/mORMot/pull/179
The new naming scheme seems fine to me.
Prefixing everything with Syn just because someone might have naming conflicts seems a bit of overkill.
Is there an option to allow multiple read single write logic?
Like TMultiReadExclusiveWriteSynchronizer do.
Useful when object must be thread safe with big amount of read operations and rare write operations.
I usually use TPasMPMultipleReaderSingleWriterLock from PasMP which works with both Delphi and FPC (Windows and Linux).
That is currently not the case for external DB.
I have created a pull request on GitHub (317) with a potential fix.
EDIT:
It seems that my fix is not a proper solution. InitializeTable must be called somewhere after because for example TSQLRest.UpdateField() is not possible at that point:
Error SQLITE_LOCKED (6) [UPDATE CISInfo SET Test8=? WHERE Test8=?] using 3.31.0 - vtable constructor called recursively: CISInfo, extended_errcode=6
In file '..\..\fpc\mORMot\SynSQLite3.pas' at line 5392
Ab,
When I override TSQLRecord.InitializeTable, it gets called when the table is created or the table is empty.
But it does not get called when the table already contains rows and I have added a new field.
Am I missing something?
Thank you.
I will do that as well then, thanks ab!
Hi,
I have a TSQLRecord descendant (TMyRecord) which is stored in a an external database (MariaDB via ZEOS).
After table is created by the ORM (CreateMissingTables) and after I insert some rows, if I add another eg. RawUTF8 published property the field is created in the DB automatically.
My issue is that the default value for all existing rows will be NULL.
So if I then try to select an existing row (TMyRecord.CreateAndFillPrepare and FillOne) I get no results.
Currently the only way I have to make those rows available again is to manually connect to the database and change NULL to empty string for all rows.
I haven't been able to find a solution for this in the documentation and the sources.
Any suggestions?
Thank you!
Svetozar Belic.
Hi ab,
I am not sure if you receive notifications for comments on your commits in GitHub.
If not, please see my comment https://github.com/synopse/mORMot/commi … #r38431461
Thanks,
Svetozar Belic.
Hi ab,
My TSQLRecord derived class has one TID property with stored AS_UNIQUE. I don't have any other filters or validators.
The clients call a method of an interface based service IOrderApiService.Add, it is registered like so:
FApiRest.ServiceRegister(TOrderService, [TypeInfo(IOrderApiService)], sicClientDriven);
That method prepares the aggregate and then adds it to the DB via a TDDDRepositoryRestCommand descendant which is registered like this:
FMainRest.ServiceDefine(TInfraRepoOrder, [IDomOrderCommand, IDomOrderQuery], sicClientDriven);
I was under the assumption that I would not need any locks in my code if I registered the services like that.
Does this mean I have to stop using stored AS_UNIQUE and add my own thread-safe TSynValidateUniqueField-like class?
Thanks,
Svetozar Belic.
Hi,
I occasionally get an AV from TSynValidateUniqueField.Process which is called from TSQLRecord.Validate. This happens when I have multiple threads adding/updating records.
Here is a stack trace leading up to the AV : https://pastebin.com/MaxfK1mG
The cause seems to be the use of filters in TSQLRecordProperties.Filters concurrently. More specifically, multiple threads are using and overwriting TSynValidateRest.ProcessRec:
if wasTSynValidateRest then begin // set additional parameters
ValidateRest.fProcessRec := self;
ValidateRest.fProcessRest := aRest;
end;
Am I missing something, or should those two properties be passed to TSynValidateRest.Process as arguments instead to avoid this?
Thanks,
Svetozar Belic.
I use Currency type and have no such problems.
Have you tried using directly just Currency everywhere (ie. don't use TPrice anywhere)? It should work that way.
Alternatively, don't create a new type for your Currency fields and just create an alias:
type
TPrice = Currency;
For the framework to process the property as you would expect, the property type needs to equal TypeInfo(Currency).
By using TPrice = type Currency; you have created a new type, so it is no longer the same type.
Thank you!
I have the same error. I haven't had time to investigate so I've just made a quick fix in SynCommons.pas at line ~60429:
procedure TRawUTF8ListHashed.RehashOnChanged;
...
collisions := fHash.Rehash({forced=}false); // I've set this to false.
...
I can provide a stack trace if needed.
Hi,
1. AS_UNIQUE is meant to be used with TSQLRecord descendants only, it does not have the same meaning elsewhere.
Its value is False, so in your case for an ordinary published property it means "Do not store".
2. Its a code completion bug, I haven't found a solution yet.
trx wrote:Note that if your TSQLRestServer is not registered with your TSQLHttpServer, TSQLRestServer.EndCurrentThread will not be called so you must call it yourself.
I have this scenario and there are no leaks that were reported by FastMM4 or noticed.
By doing this you are not forcing the creation of connections per thread?
Depending on the provider, is this not necessary as mentioned by ab above?
In my case some requests were not processed in the thread pool because of keep-alive, so a dedicated background thread was created.
See the implementation of TSynThreadPoolTHttpServer.Task(aCaller: TSynThread; aContext: Pointer) in SynCrtSock.pas for details.
In that case, when a DB connection (for me via ZEOS to MySQL) is created for each background thread, it is not automatically released when the thread ends.
I have only noticed this because MySQL started complaining about too many connections.
Note that if your TSQLRestServer is not registered with your TSQLHttpServer, TSQLRestServer.EndCurrentThread will not be called so you must call it yourself.
For example, I have two TSQLRestServers:
1. One for external DB which I don't register with TSQLHttpServer.
2. One in memory which is registered with TSQLHttpServer which provides interface based services. Those services then use the first REST server to get the required data and a DB connection is created per thread.
When the thread ends, TSQLHttpServer will call EndCurrentThread of the server 2. but not 1. so the DB connection will not be released.
To solve this I use a TSQLHttpServer descendant:
TMyHTTPServer = class(TSQLHttpServer)
private
FDBRest: TSQLRestServer;
protected
procedure HttpThreadTerminate(Sender: TThread); override;
begin
inherited HttpThreadTerminate(Sender);
if Assigned(FDBRest) then
FDBRest.EndCurrentThread(Sender);
end;
end;
If your compiler supports it, you can do:
myVar := Default(TMyRecord)
Yeap, I forgot to do that, done.
Yeah, I see you fixed the source of the problem.
Thank you!
When commiting the batch, in TSQLRestServerDB.InternalBatchStop that T*DynArray field is determined to be ftBlob at line 2008 (latest master branch) :
Types[f] := Props.Fields.List[prop].SQLDBFieldType;
Later in that same method, a statement is prepared and field values are bound using the determined types:
for f := 0 to valuesCount-1 do begin
if GetBit(ValuesNull[0],f) then
fStatement^.BindNull(f+1) else
case Types[prop] of
ftInt64:
fStatement^.Bind(f+1,GetInt64(pointer(Values[f])));
ftDouble, ftCurrency:
fStatement^.Bind(f+1,GetExtended(pointer(Values[f])));
ftDate, ftUTF8:
fStatement^.Bind(f+1,Values[f]);
ftBlob:
fStatement^.BindBlob(f+1,Values[f]);
end;
This field's data (a json string) is therefore saved as a ftBlob. Later when you retrieve the row in unit SynSQLite3, TSQLRequest.FieldsToJSON the field is a ftBlob which gets encoded to base64. When this base64 encoded value ends up in TSQLPropInfoRTTIDynArray.SetValue, it is not decoded first and fails to be processed.
I am not sure where this should be fixed, you are more familiar with the code. Correcting this in TSQLPropInfoRTTIDynArray.SetValue seems the like cleanest approach to me because I don't see why it wouldn't accept base64 and be a T*ObjArray.
Thanks ab,
Svetozar Belic.
Hi,
I was trying out mORMot's DDD features and have noticed a bug which i believe is in the implementation of TSQLPropInfoRTTIDynArray.SetValue.
For example, using a class like the following (with TSQLRestServerDB and SQLite3):
TTestSubObjectName = type RawUTF8;
TTestSubObject = class(TSynPersistent)
private
FName: TTestSubObjectName;
published
property Name : TTestSubObjectName read FName write FName;
end;
TTestSubObjectObjArray = array of TTestSubObject;
TTestObject = class(TSynAutoCreateFields)
private
FSubObjects: TTestSubObjectObjArray;
published
property SubObjects : TTestSubObjectObjArray read FSubObjects write FSubObjects;
end;
TTestObjectObjArray = array of TTestObject;
TJSONSerializer.RegisterObjArrayForJSON([
TypeInfo(TTestSubObjectObjArray), TTestSubObject,
TypeInfo(TTestObjectObjArray), TTestObject
]);
// Persistence object.
TSQLTestObject = class(TSQLRecord)
protected
FSubObjects: TTestSubObjectObjArray;
published
property SubObjects : TTestSubObjectObjArray read FSubObjects write FSubObjects;
end;
If you create one test object with several subobjects and save (commit) it, everything works as expected, when you select and get the object again, the subobjects are there.
The problem arises when you create, add and commit several such test objects at once. If you select and get them, you will notice that there are no subobjects.
I did some digging around and I see that if you commit one object, its SubObjects value ends up in the database as TEXT whereas if you commit multiple objects, their SubObjects value ends up as BLOB.
I think this is because batch insert uses prepared statements and explicitly sets the field type to BLOB because SubObjects property is a DynArray. Single insert on the other hand saves everything as text.
When retrieving the objects from the DB, the implementation of TSQLPropInfoRTTIDynArray.SetValue expects JSON text.
But because the field was saved as a BLOB, it gets a base64 encoded value instead which does not get handled properly.
procedure TSQLPropInfoRTTIDynArray.SetValue(Instance: TObject; Value: PUTF8Char; wasString: boolean);
var tmp: TSynTempBuffer;
da: TDynArray;
begin
GetDynArray(Instance,da);
if Value=nil then
da.Clear else
try
// fObjArray is not nil, so it will attempt jump to da.LoadFromJSON which does not expect a base64 encoded input.
if (fObjArray=nil) and Base64MagicCheckAndDecode(Value,tmp) then
da.LoadFrom(tmp.buf) else
da.LoadFromJSON(tmp.Init(Value));
finally
tmp.Done;
end;
end;
The following quick fix solves the problem:
procedure TSQLPropInfoRTTIDynArray.SetValue(Instance: TObject; Value: PUTF8Char; wasString: boolean);
var
tmp: TSynTempBuffer;
da: TDynArray;
begin
GetDynArray(Instance,da);
if Value = nil then
da.Clear else
try
if not Base64MagicCheckAndDecode(Value,tmp) then
tmp.Init(Value);
if fObjArray = nil then
da.LoadFrom(tmp.Buf) else
da.LoadFromJSON(tmp.buf)
finally
tmp.Done;
end;
end;
The above problem has another side effect.
For example, you commit one object and after that you commit two in a batch.
You then retrieve all inserted objects, and see that the first object is correct, but the other two are not.
Their SubObjects field has the value of the first object. This happens because when they get filled, the same record (TSQLRecordFill.fTableMap.Dest) is reused.
So the first object gets filled properly, the other two attempt to TSQLPropInfoRTTIDynArray.SetValue but fail, so they end up with the first one's data.
This problem does not occur with eg. TSQLRestServerFullMemory.
I have created a quick test application which reproduces the problem : http://txlab.org/objArrayPublishedPropTest.zip
Thanks,
Svetozar Belic.
Hi,
I have a TSQLRecord like so :
TMzItem = class(TSQLRecordWithNames)
protected
FDescriptions : TMzNameList;
FImages : TMzImageList;
public
constructor Create; override;
destructor Destroy; override;
published
property Descriptions : TMzNameList read FDescriptions write FDescriptions;
property Images : TMzImageList read FImages write FImages;
end;
constructor TMzItem.Create;
begin
inherited;
FDescriptions := TMzNameList.Create;
FImages := TMzImageList.Create;
end;
destructor TMzItem.Destroy;
begin
FImages.Free;
FDescriptions.Free;
inherited;
end;
TMzNameList and TMzName look like :
TMzName = class(TPersistentWithCustomCreate)
private
FName: RawUTF8;
FLanguageID: TID;
public
constructor Create; overload; override;
constructor Create(const AName : RawUTF8; const ALangID : TID); overload;
function GetLanguage(const AClient : TSQLRest): TMzLanguage;
published
property LanguageID : TID read FLanguageID write FLanguageID;
property Name : RawUTF8 read FName write FName;
end;
TMzNameList = class(TObjectList)
protected
function GetItem(Index: Integer): TMzName;
procedure SetItem(Index: Integer; const Value: TMzName);
public
function Add(AObject: TMzName): Integer; inline;
property Items[Index: Integer]: TMzName read GetItem write SetItem; default;
end;
constructor TMzName.Create(const AName: RawUTF8; const ALangID: TID);
begin
Create;
FName := AName;
FLanguageID := ALangID;
end;
constructor TMzName.Create;
begin
inherited Create;
end;
function TMzName.GetLanguage(const AClient: TSQLRest): TMzLanguage;
begin
Result := TMzLanguage.Create(AClient, FLanguageID);
end;
function TMzNameList.Add(AObject: TMzName): Integer;
begin
inherited Add(TObject(AObject));
end;
function TMzNameList.GetItem(Index: Integer): TMzName;
begin
Result := TMzName(inherited Items[Index]);
end;
procedure TMzNameList.SetItem(Index: Integer; const Value: TMzName);
begin
inherited Items[Index] := TObject(Value);
end;
initialization
TJSONSerializer.RegisterClassForJSON([TMzName]);
When i create a TMzItem with some Descriptions they are saved just fine. The problem is with TMzImageList :
TMzImage = class(TPersistentWithCustomCreate)
protected
FId : integer;
FNames : TMzNameList;
FIsDefault : boolean;
FIsVisible : boolean;
public
constructor Create; overload; override;
constructor Create(const AId : integer; const AIsDefault : Boolean; const AIsVisible : Boolean); reintroduce; overload;
destructor Destroy; override;
published
property Names : TMzNameList read FNames write FNames;
property ID : integer read FId write FId;
property IsDefault : boolean read FIsDefault write FIsDefault;
property IsVisible : boolean read FIsVisible write FIsVisible;
end;
TMzImageList = class(TObjectList)
protected
function GetItem(Index: Integer): TMzImage;
procedure SetItem(Index: Integer; const Value: TMzImage);
public
function Add(AObject: TMzImage): Integer; inline;
property Items[Index: Integer]: TMzImage read GetItem write SetItem; default;
end;
function TMzImageList.Add(AObject: TMzImage): Integer;
begin
inherited Add(TObject(AObject));
end;
function TMzImageList.GetItem(Index: Integer): TMzImage;
begin
Result := TMzImage(inherited Items[Index]);
end;
procedure TMzImageList.SetItem(Index: Integer; const Value: TMzImage);
begin
inherited Items[Index] := TObject(Value);
end;
constructor TMzImage.Create(const AId: integer; const AIsDefault, AIsVisible: Boolean);
begin
Create;
FNames := TMzNameList.Create;
FId := AId;
FIsDefault := AIsDefault;
FIsVisible := AIsVisible;
end;
constructor TMzImage.Create;
begin
inherited Create;
end;
destructor TMzImage.Destroy;
begin
FNames.Free;
inherited;
end;
initialization
TJSONSerializer.RegisterClassForJSON([TMzImage]);
When i add some TMzImage to Images of TMzItem they are not saved. Instead that property is empty ('[]'). If i remove Names from TMzImage, then Images are saved properly. I have tried to debug Client.Add(item, True) and i can see that the TMzItem is properly converted to JSON along with its images and their names but not saved as such. Images is [].
Have you any suggestions?
Thank you.
trx.
Pages: 1