You are not logged in.
Greetings!
I became a newbie in the newest mORMot, and I'm currently really interested in understanding if the Swagger template is somehow supported already in mORMot2?
---
Seems much is done for mORMot2 and it is almost out for full release. Incredible work!!
And I do disagree with "Too much material was killing the documentation, so user reported..."
Glad to return to the forum after some absence
Does not work for hex encoded characters for example "
I used SynCommons.HexToChar in my custom SynCommons.TTextWriter as a part of XML unescape add-method.
Also THTMLEncoding.DoDecode might help: http://docwiki.embarcadero.com/Librarie … MLEncoding (haven't tried it myself though)
Service (TSynDaemon) executable can be also launched in console mode with the /console parameter, like
YourService.exe /console
It might be useful, for example, for service debugging from Delphi with the corresponding property in project options:
<Debugger_RunParams>/console</Debugger_RunParams>
Zeos 7.2.10 with latest patches (of 8th of April)
I'm using the trunk Zeos version. You could try it, probably not all fixes go to patches for a stable Zeos.
https://sourceforge.net/p/zeoslib/code- … ree/trunk/ or https://github.com/frones/ZeosLib
I'm also using 10.4.2. I specifically made a test with the direct select statement execution and RowDocVariant through Zeos (WIN1250 DB).
In my case UTF8ToSynUnicode from SynCommons calls UTF8ToWideChar from SynCommons. And it seems working ok... - checked the resulting variant value from the debugger.
I used FB 2.5.9, although I doubt that it matters. Btw, I checked both, with USE_SYNCOMMONS and without it. Probably I misunderstand something, sorry.
I assume there shouldn't be any problem in reading from WIN1250 FB DB. TDocVariant from RowDocVariant should contain a full RawUTF8 string. Could it be that you got the loss afterward? maybe due to direct string assigning
Could you explain the need for IdAtom published property? I just couldn't understand it. TSQLRecord has ID support already, you can use this field easily. ID mapping will also work without additional published property...
Besides, you can't declare a class property named Type because it's a Delphi/Pascal keyword.
As an option, you could use '&' for avoiding Delphi keyword and reducing mappings quantity
TormtDfRmAtom = class(TBaseORMEntity)
...
published
// property IdAtom: TID read fID write fID;
property Name: RawUTF8 read FName write SetName;
property &Type: Integer read FType write FType;
end;
...
.MapField('ID', 'IdAtom');
I think Vitaly was thinking of the following scenario. This is not a real example but shows the scheme.
Yep, exactly. I was trying to express myself with words, but it seems your example explains everything better
Nope, in fact, there is no need for any client for using TSQLRestServerDB. It has everything, ORM, batches - all (or almost all) is threadsafe. You can just create a TSQLRestServerDB instance in your TSQLRestServerFullMemory or provide it there via different ways - constructor of a descendant, property, global variable, something else. So every service/method of TSQLRestServerFullMemory will be able to use it and provide the result for your REST client in the form of DTO, plain data, or maybe even TSQLRecord. Only TSQLRestServerFullMemory will be exposed to the HTTP server, so REST clients will work only with it.
REST Client -----> TSQLRestServerFullMemory:SOA/Methods -----> TSQLRestServerDB (or many TSQLRestServerDB-instances, as in your case, as far as I understood)
I think, such a way will allow adding new typical TSQLRestServerDB-instances at runtime and routing users between them at TSQLRestServerFullMemory-level, which will not change anyhow despite the quantity of DBs. But again, as far as I understood your problem.
But at some point I will need to communicate with TSQLREstServerDB, would I do that with a client connection?
You can use TSQLRestServerDB instance directly from TSQLRestServerFullMemory services/methods implementations.
I'm also using such an approach, works perfectly and no need to worry about DB access since it is internal.
Checked a couple of projects with mORMot2 and Zeos/FB. Seems, all ORM and Batch operations are going well. Thanks!
Delphi 10.4.2, Win64, freshest mORMot2, Zeos r7349, FB 2.5.9
PUREMORMOT2;MORMOT2
Nice!
Yes, all's good now, thanks!
I have updated the repository.
I'm getting the exception about outdated static lib on Delphi 10.4.1 Win64. Is it missing or my build is broken?
(had to pause my migration to 2.0 and return to 1.18 for a while - need to do some work)
Checked it with today's commit 679963edbbae2d6146dd420288ddc8ec854d3cd6. Seems the problem has gone. Thanks!
Isn't it the Rtti.RegisterFromText([TypeInfo(TSynMapSymbol), _TSynMapSymbol,...) which fails?
Nope, checked it with step over - seems it passes the whole mormot.core.log.InitializeUnit well. Exception appears at mormot.core.json.InitializeUnit.
Which compiler are you using?
Delphi 10.4.1 Win64
I was curious why not all projects are failing and I think I found something, hope useful. At least, now I can overcome the exception.
This particular project contained mormot.core.log in uses list in dpr (I was setting LOG_VERBOSE sometimes directly in dpr for having a full log).
Such order causes an exception with the latest commit:
program NextGeoViewSQLite3mORMotDemo;
uses
Vcl.Forms,
mormot.core.log, <--------- before the main form unit
mormot.core.rtti,
uFormNextGeoViewSqlite3MormotDemo in 'uFormNextGeoViewSqlite3MormotDemo.pas' {FormNextGeoViewSqlite3MormotDemo},
uNextGeo in '..\..\Source\uNextGeo.pas',
...
And this order goes well, no exception:
program NextGeoViewSQLite3mORMotDemo;
uses
Vcl.Forms,
mormot.core.rtti,
uFormNextGeoViewSqlite3MormotDemo in 'uFormNextGeoViewSqlite3MormotDemo.pas' {FormNextGeoViewSqlite3MormotDemo},
mormot.core.log, <--------- after the main form unit
uNextGeo in '..\..\Source\uNextGeo.pas',
...
I see, I'm trying to follow ORM approach as much as I can. But I understand that not everything can be handled easily, especially with such big legacy DBs.
Hope, somebody else will help you with advice for your case.
I'm not sure so far what has happened, some (not all) projects stopped working with the latest commit. Such exception appears at the start:
Project NextGeoLoadDemo.exe raised exception class EJSONException with message 'Rtti.Count=5 at mormot.core.json start'.
All works fine with the previous commit, the problem seems to appear with 544078bdf080cd6a80566461567cbc84131a948f only.
Could it be that I'm using logs incorrectly since the last commit concerns only mormot.core.log unit?
If TransactionBegin use is not mandatory, I would use a Batch for such operations: https://synopse.info/files/html/Synopse … l#TITL_100
Just do not point to a particular TSQLRecord class at TSQLRestBatch creation (use nil instead), so you'll be able to update different tables in one batch.
Will definitely study it, thanks! And look into your implementation.
I've learned so much from the mORMot. Although, sometimes it is really hard for me to understand some code from the first, second, or even tenth look
Thanks for the very positive feedback, and the detailed reports whenever you reached a bug of mORMot 2!
Yeah, every time I wished to find the solution by myself to avoid disturbing you much. But the only thing I could do is to locate the problem as much as I can. I guess my level is not high enough so far
About EngineAdd... override, I guess the problem by rather be fixed at mormot.db.sql (or mormot.db.sql.zeos) level.
Feedback is welcome, here!
I'll try a couple of ideas first, although they are pretty weak.
You are right, i18n is not included yet, nor the TDataSet/UI part of the framework.
What do you expect/require to find in mORMot 2?
Well, I'm not using mORMotUI/TDataSet. I have a TDrawGrid-based class (inspired by TSQLTableToGrid) with some additional functionality and improvements, and it works well enough already with mORMot2 Everything else seems to present.
I guess I might easily get some troubles further with WinService, HTTP/REST/SOA-servers/clients migration, especially in our FHIR-specified project, where we've overridden mORMot classes pretty deep (I'll make its migration the last one). But I'm sure it all will be solved
I have reduced the memory needed for TOrmTable by half on 64-bit systems, without touching the performance AFAICT.
I would not say that I ever regretted about mORMot memory consumption, but that sounds Very nice! Even if so far I don't understand the particular advantage of it, I already always know for sure: if Arnaud made something new, it will be better, more optimized, faster, and so on
I guess I can share some results of the first stage of my migration to mORMot2. I Hope, it'll be interesting for somebody.
I have done the migration of only one small project group (6 Delphi projects) and a part of my base classes, using mORMot. It contains several TWinHttp-based REST/JSON clients for different web-services with automatic DTO handling. It also contains some work with external DBs: FB2.5 and SQLite3 (ORM, batches, etc.). It doesn't contain any server/daemon. Custom serialization is frequently used there. Some projects are using threads. And it all works well with mORMot2, now
At first, I made it working in compatible with 1.18 mode, then I switched to PUREMORMOT2 - it was a good opportunity to review, clean, and refactor the old code.
I use my own base classes, based on mORMot classes - probably it made the whole process a bit easier for me.
My personal impressions:
- from the beginning, it was pretty hard to find needed classes and functions, but after several days I became pretty used to it.
- I missed some documentation for mORMot2 like for 1.18 - Arnaud spoiled us The documentation often helped me to understand different ways and choose the best one.
- the framework source code formatting became more readable (just for me, at least).
- naming also became much more convenient (for my taste - "Kotlin-way" and UTF8String are very nice).
My current losses:
- Zeos support (temporary).
- EngineAdd, EngineUpdate, EngineBatchSend overriding for TRestServerDB, which I used for universal transliteration (our legacy FB DB is using ISO8859_1 ) - will try to find another workaround for it. Can't afford this loss.
- i18n, I couldn't find some functions, which I used before. But it is probably will be not needed at all, so I have cut it off.
It took pretty much time to get used to mORMot2, but I consider the first stage of migration to be successful I'll continue the process with other much more complex project groups.
Yes, it works smoothly, now! Thanks
My current personal situation:
- my employer sticks Delphi (a huge old legacy product),
- the client of my employer uses Windows infrastructure.
In addition, I have much less experience with Unix it just happened so.
And I would admit that with all Windows disadvantages (and prices) it is always easier to start/work with it, especially if you have some lack of knowledge/experience.
Just a small non-critical issue, but very strange.
Creating a TRestServerDB instance with an empty model causes AV:
TRestServerDB.CreateWithOwnModel([]);
I hardly can imagine that someday I'll need an empty model, but I guess it'll be better to handle it without AV exception.
Anyway, I tried to debug and got a bit strange result. AV appears in
constructor TRestOrmServer.Create(aRest: TRest);
...
SetLength(fTrackChangesHistoryTableIndex, fTrackChangesHistoryTableIndexCount);
for t := 0 to fTrackChangesHistoryTableIndexCount - 1 do
fTrackChangesHistoryTableIndex[t] := -1; <---- hear
...
where (as I assume) it shouldn't get at all because of fTrackChangesHistoryTableIndexCount = 0. I guess I'm missing something simple...
Maybe we can use mORMot sample 09 + Apace ab to do a side-by-side benchmark?
Maybe we can even ask all member on this forum to share about their VPS statistics?
It would be a piece of nice and useful information I guess, at some point, everybody gets the need to have some cheap and stable standalone server (at least, for testing). And this information can make the choice easier.
Currently, I'm not using any, but I assume I may need one someday in the future.
I used ultravds.com for years: adjustable, fast, no direct limits, cheap. I had no single problem with it.
In my case, it was even cheaper -40%: 20% for yearly payment and 20% for the rare promo code, which one guy gave to me on some forum many years ago (checked it, still works, btw, if somebody needs it).
But! it is located in Russia. So, think twice. Probably, it is just a corrupted opinion of the man, living in Russia, but I would not use it for something important or sensitive.
I have some multithreaded code, where each thread creates and uses TSynAutoCreateFields-based instances of the same types. All these threads are pre-initialized (so they start almost at the same time).
This code works ok with 1.18, but with 2.0 I occasionally get random errors (mostly AV), for example at TRttiCustom.SetAutoCreateFields. Could it be that somehow threads are getting mixed at TSynAutoCreateFields-based class registering?
If I add Create/Destroy procedures for these classes before multithreaded processing (in other words - pre-registering them in the main thread) - all goes fine:
TKommuneInfoCountyGeo.Create.Free;
TKommuneInfoMunicipalityGeo.Create.Free;
yeah, I had no doubts that it is just a "technical" issue. Decided to notify just in case
Just a small report. The last commit probably doesn't contain changes for mormot.orm.core. Line 15390:
[dcc64 Error] mormot.orm.core.pas(15390): E2003 Undeclared identifier: 'Utf8FirstLineToUnicodeLength'
the current version performs similar, with the same popup ;-)
Also had this warning, but then downloaded static files from another link: https://github.com/synopse/mORMot2/rele … 2static.7z
Perhaps even mORmot 1.18 didn't work when USE_SYNCOMMONS was not defined.
I liked your supposition, so I rollbacked the actual "migration" project to 1.18 use and tested it with the same Zeos and without USE_SYNCOMMONS. It works fine, it is hard to say how fast/slow in comparing, but it works.
I can continue migration/testing with FireDAC without any problem, so Zeos support easily can wait if you'll have more important things (I might already distract you enough with my current 2.0 feedback).
Ah, I see. Tested the new version, all works, nice! Thanks for the fix and the explanation!
btw, if somebody is interested in my current results for external DBs:
TSQLDBSQLite3ConnectionProperties - fine
TSQLDBFireDACConnectionProperties (FB 2.5.9) - fine
TSQLDBZEOSConnectionProperties (FB 2.5.9, ZeosLib git master f347957defc8e89c381f5d06596ac3a170c5994e 2021-01-21):
with USE_SYNCOMMONS - nothing compiles since 1.18 units namings in Zeos;
without USE_SYNCOMMONS - compiles but throws errors during connection, like
Error SQLITE_ERROR (1) [Step] using 3.34.0 - TSqlDBZEOSConnectionProperties: unexpected TZSQLType "stUnknown", extended_errcode=1
Ok, probably this decision is ugly, but at least it works for my case.
Instead of (1.18)
TSQLRestStorageExternal(StaticVirtualTable[LModel.Tables[i]]).OnEngineAddComputeID :=
DoOnEngineAddComputeIDFirebird;
for 2.0 I used
TRestStorageExternal(TRestOrmServer(server).StaticVirtualTable[LModel.Tables[i]]).OnEngineAddComputeID :=
DoOnEngineAddComputeIDFirebird;
Overriding EngineAdd, EngineUpdate, EngineBatchSend for the whole TRestServerDB-based class, unfortunately, seems to be impossible, so far at least.
I'm not sure that the problem is there, but I got stuck (or tired) in debugging.
It seems that the problem appears only for big JSON arrays. Procedure _JL_DynArray exits with an invalid context result after guessing the array count(=0).
Considering the logic of the code (like result := -result and so on) I tried to change
function JsonArrayCount(P, PMax: PUtf8Char): integer;
begin
result := -1;
...
It helps for the huge input but breaks the parsing of smaller arrays...
In 1.18 I use TSQLRestServerDB-based class and model with optional different "external" TSQLDBConnectionProperties (particularly TSQLDBSQLite3ConnectionProperties, TSQLDBZEOSConnectionProperties, TSQLDBFireDACConnectionProperties). There (for some DBMS) I can access TRestStorageExternal(StaticVirtualTable[]).OnEngineAddComputeID and override EngineAdd, EngineUpdate, EngineBatchSend and so on. But I can't do the same with TRestServerDB in 2.0
I'm a bit confused, probably I got it totally wrong. Will appreciate any clarification.
I may add a Win1251 explicit collation... since there is already a Win1252 dedicated version...
Personally, I'm not interested in win1251 since I'm not using it in the development, although I see questions/troubles concerning it on the forum from time to time.
Just curious, what will be the disadvantage of adding win1251 collation for other non-cyrillic developers?
Joining all kind words!
I will try to test migration starting from some simple projects. Will mORMot2 stay located at the same repository https://github.com/synopse/mORMot2?
As a quick workaround, I would suggest trying to change the collation to WIN32NOCASE for the search field in your DB.
Also, this might be helpful:
https://synopse.info/files/html/Synopse … #TITLE_155
https://synopse.info/forum/viewtopic.php?id=4342
Sorry, I'm not sure about ServiceResultStart/End since I haven't used them. I took a quick look at these methods, I am afraid it might be not enough. Hope, Arnaud or other guys will advise you something better.
We're using custom ServicesRouting and TServiceCustomAnswer in services for a heavy project (FHIR-specified), where we had to override almost everything to make it automatic and compatible with third-party software: fhir+json/fhir+xml output/input data formats, different http-methods, custom meaningful headers, and much other stuff...
You can create your own TSQLRestRoutingREST-based class and assign it to your <TSQLRestServerFullMemory-instance>.ServicesRouting property. Not an easy way, but there you'll be able to adjust almost everything.
I guess this code might be slightly better since it will get the pointer just one time:
with _Safe(vJSON.id)^ do
for intX := 1 to 10 do
AddItem(intX);
or local variable can be also an option.
Anyway, these are just details, of course.
Well, if it is FPC-only related problem, then maybe it is a case of https://synopse.info/files/html/Synopse … l#TITLE_40
Unfortunately, my installed FPC (CodeTyphon) is pretty much outdated (haven't used it for years) and it doesn't work as it is with my current mORMot
Thanks for your reply but it still does not work. I still get the same error.
Yeah, sorry, you're right. I took a look at _Json function. Quotes are not necessary:
// - in addition to the JSON RFC specification strict mode, this method will
// handle some BSON-like extensions, e.g. unquoted field names:
I made a test. In my case (D10.4.1) your code works without any error in both cases.
upd: tested it with 10.3.3 and 10.2.3 also - all goes fine
I guess the error comes from
vJSON := _Json('{id:[]}');
Try
vJSON := _Json('{"id":[]}');
Also I think there is no need in
TDocVariant.New(vJSON);
Vitaly, your solution works fine... and makes me more and more smarter;-)
Glad to hear that
In the callback function, the data must be read (or write) in differently depending on the "type". Alternatively, the coordinates could be delivered as json-string and evaluated separately.
Yes, I also came to these two paths. I haven't found the best approach yet (that's why I was particularly interested in this thread), but for the temporary solution, I decided to parse coordinates on-demand ("alternative way", as you described). The reason is that I have to work with rather big .geojson files (up to 400 Mb) and I do not need all GeoJSON objects coordinates at once. I do not need serialization in GeoJSON format, so it also makes it a bit easier.
One more thing, which might simplify type-handling. I assume you may know this already, so it is just in case. Using enumerations, like this:
TGeoJsonType = (gjtUnknown, gjtPoint, gjtMultiPoint, gjtLineString, gjtMultiLineString, gjtPolygon, gjtMultiPolygon,
gjtGeometryCollection, gjtFeature, gjtFeatureCollection, gjtName, gjtLink);
And, for example:
TGeometry = packed record
coordinates: RawJSON;
&type: TGeoJsonType;
public
function GetLineString(out AResult: TPoint2DList): Boolean;
function GetMultiPolygon(out AResult: TMultiPolygon): Boolean;
....
end;
This option will trim all textual enumerations "gjtLineString" -> "LineString":
TJSONSerializer.SetDefaultEnumTrim(True);
Enumerations will allow to simplify (and speedup) types checking: &type=gjtLineString instead of &type='LineString'. And "case &type of ..." might also make things easier in some cases.
LCoord.lng := GetNextItemDouble(P); LCoord.lat := GetNextItemDouble(P, ']'); // this works.
Noted for the future, thanks!
Sorry, but I don't understand how text-based definition (if we're talking about it) can result [[13.3568, 52.6183],...] from TGeometry/TPoint2D(lng,lat:double) records.
Anyway, it makes me feel more and more stupid I guess I should leave the discussion for saving my poor mental health
You don't need to put a RawUTF8 within the record.
Agree, it left there from another earlier try. needed_for_rtti can be easily removed.
Don't register the record, but the dynamic array of record.
Then you can write TypeInfo(TPoint2DDynArray).
Yeah, also thought so, but somehow it didn't work for me. Maybe I have mistaken somewhere. But it doesn't make much difference as far as I understand. At least custom reader/writer will be the same, I guess.
And you don't need custom serialization, you can use plain text serialization with the dynamic array.
a dynamic array of records? wouldn't "lng"/"lat" names appear then in output?
Regarding the original question of the post, it seems that there is really no way to do this, since the parser only accepts "array of" simple or record types.
Well, something always can be done it depends on the needs. For example, such a workaround can be considered
Records definitions:
TPoint2D = packed record
lng, lat: double;
needed_for_rtti: RawUTF8;
end;
TGeometry = packed record
coordinates: array of TPoint2D;
&type: RawUTF8;
end;
Custom serialization:
TTextWriter.RegisterCustomJSONSerializer(TypeInfo(TPoint2D), TGeometryCustom.TPoint2DReader,
TGeometryCustom.TPoint2DWriter);
Custom writer:
class procedure TGeometryCustom.TPoint2DWriter(const aWriter: TTextWriter; const aValue);
var
LCoord: TPoint2D absolute aValue;
begin
aWriter.Add('[% , %]', [LCoord.lng, LCoord.lat], twJSONEscape);
end;
Custom reader:
class function TGeometryCustom.TPoint2DReader(P: PUTF8Char; var aValue; out aValid: Boolean;
CustomVariantOptions: PDocVariantOptions): PUTF8Char;
var
LCoord: TPoint2D absolute aValue;
LArray: TDoubleDynArray;
begin
aValid := false;
result := nil;
if (P = nil) or (P^ <> '[') then
exit;
inc(P);
{ LCoord.lng := GetNextItemDouble(P);
LCoord.lat := GetNextItemDouble(P); - doesn't work, don't know why}
DynArrayLoadJSON(LArray, FormatUTF8('[%]', [GetNextItem(P, ']')]), TypeInfo(TDoubleDynArray));
if Length(LArray) > 1 then
begin
LCoord.lng := LArray[0];
LCoord.lat := LArray[1];
end;
Finalize(LArray);
if P = nil then
exit;
aValid := true;
result := P; // ',' or ']' for last item of array
end;
GetNextItemDouble didn't work as I expected for the second value - probably I misunderstood it. So I used an additional dynarray.
I haven't got deep in all that records custom parsing stuff (prefer objects) - my almost first quick experience, but at least it seems working Hope it'll help @robs
I meant that Delphi reserved word "type" can be declared as "&type" property in the record
TGeometry = packed record
coordinates: ...;
&type: RawUTF8;
end;
I was just trying to say that reserved word shouldn't be a problem. Probably I just misunderstood something about it, don't mind.
Anyway, it seems that @robs is working with GeoJSON. I was interested in that because I'm working with it on my actual project.