You are not logged in.
But even working in REST mode, SQL has its advantage and we need to execute complex SQL to generate reports.
IMHO, the JSON response returned from the server should include the field type info.
So do you have a plan to fix it?
Thanks ab, I'll utilize the SetFieldType method. I'm curious - doesn't the sqlite engine return field type info for the result set?
A sample JSON response I got while debugging this issue:
[{"DocId":2078946242580065800,"DocDate":"2015-09-15","EventType":"e_po","ProductNr":"000020"}]note 1: the DocId value is long...
note 2: I'm using the latest version from github.
The table:
CREATE TABLE CustInv (
ID INTEGER PRIMARY KEY AUTOINCREMENT,
...
);The view:
CREATE VIEW v_event_inv AS
SELECT header.ID AS DocId,
...
FROM CustInv headerThe delphi code:
var
myFieldType: TSQLFieldType;
myFieldTypeName: string;
myFieldValue: string;
tmpTable: TSQLTableJSON;
begin
tmpTable := AppCtx.Db.ExecuteList([], 'SELECT * FROM v_event_inv');
//some testing code
{$IFDEF DEBUG}
myFieldType := tmpTable.FieldType(0);//expected: sftID. Actual: sftFloat
myFieldTypeName := GetEnumName(TypeInfo(TSQLFieldType), Integer(myFieldType));
myFieldValue := tmpTable.FieldValue('DocId', 1); //correct
CodeSite.Send(myFieldValue, myFieldTypeName);
{$ENDIF}PS: I noticed the bug after using ToClientDataSet, but tracked it down in ExecuteList()...
Update: SQLite has already the json1 extension and it's expected to be integrated into the core once it's stabilized: https://www.sqlite.org/draft/json1.html
Obviously can be very useful for people who need very flexible, even nested database schema.
Great! Thanks AB!
Hi Ab,
Is there a way to get all field names in CSV format for a TSQLRecord?
I need such a function because I want to avoid updating some of the fields via the '' parameter when calling Update().
It'll be idea to be able to:
const
cFieldNotToUpdate = 'field1';
var
fieldsToUpdate: string;
myRec: TSQLPerson;
begin
fieldsToUpdate := myRec.GetAllFieldNamesInCSVFormat;
//so that I can:
fieldsToUpdate := ReplaceStr(fieldsToUpdate, cFieldNotToUpdate , '');
aDb.Update(myRec, fieldsToUpdate);
...I hope'll you get my point.
Or even better, in future versions to provide a new parameter for the Update() method called 'fieldsNotToUpdate' which does the opposite of the 'customFields' param ![]()
I'm glad to know there is the new fts5! Even it's experiential, I suggest to add it to the mORMot framework anyway!
I think you can do that with mORMot, but I maybe a queue like this http://zeromq.org/ fits your needs more?
Sounds interesting. Thanks for sharing.
You are so fast! Thanks ab!
Hi Arnaud,
Would you add the "DoNotAutoComputeFields" param to the BatchUpdate method? There are situation where you don't need to update the TModTime fields, like doing a backup or syncing.
PS, do you accept github pull requests for source code contribution?
Thanks to the help by the author of OTL Primoz, problem's fixed for approach A, OtlTaskControl.CreateTask needs the .Unobserved modifier!
The related OTL community post: https://plus.google.com/u/0/11514338247 … Ys5?cfem=1
Well, after some more testing, it turned out the problem has nothing to do with TSQLHttpClient (because I replaced the http client connection with Sleep(5000) to rule it out), but there seems to be some incompatibility between OmniThreadLibrary (latest github SVN) and mORMot (I'm using the latest version about 1 day ago).
My server is configured like this:
FDbSvr := TDbServer.Create(FDataModel, GetDbFileName, False);
SetupServerLogging;
FDBSvr.PublishAllServices;
(*
WAL might increase the risk of data losses!
http://liferea.blogspot.jp/2012/03/morpheus-suggested-in-comment-to-check.html
http://www.sqlabs.com/sqlitedoctor_details.php
http://stackoverflow.com/a/20970876/133516
*)
FDbSvr.DB.WALMode := False;
FDbSvr.DB.Synchronous := smFull;//ensure max data stability!
FHttpSvr := TSQLHttpServer.Create(IntToStr(gServerPort), [FDbSvr], '+', HTTP_DEFAULT_MODE);
FHttpSvr.AccessControlAllowOrigin := '*'; // allow cross-site AJAX queries;and I have an interface-based service on the server side, with default configuration:
/// this is the main interface exposed by the server
TServerApi = class(TInterfacedObject, IServerApi)
public
function StartCommonDataDownload(aManualCall: Boolean): Boolean;
end;And in this StartCommonDataDownload() method, a thread will be started, like this:
/// approach A, the method will not return until 5 seconds are passed
function TServerApi.StartCommonDataDownload(aManualCall: Boolean): Boolean;
begin
OtlTaskControl.CreateTask(
procedure (const task: IOmniTask)
begin
Sleep(5000);
end
).Run;
end;
/// approach B, the method will not return until 5 seconds are passed
function TServerApi.StartCommonDataDownload(aManualCall: Boolean): Boolean;
begin
TThread.Queue(nil, procedure
begin
Sleep(5000);
end
);
end;
/// approach C, works!
Parallel.Async(procedure
begin
Sleep(5000);
end);Only approach C worked. Does it have anything to do with the way the TSQLHttpServer is running in? I run it with administrator and I guess it runs in the http.sys mode? (update: : tried useBidirSocket just now, same problem...) I also noticed the OmniThreadLibrary 's communication sub-system also is not working inside an interfaced service method...
Thanks for any help.
Update 1: This post has been updated to reflect the latest findings, it seems that the OmniThreadLibrary's CreateTask() is in a way not compatible with mORMot, please do directly to http://synopse.info/forum/viewtopic.php?id=2852#p17618 for more info.
Hi Arnaud,
In my program there is a background thread which creates a TSQLHttpClient and calls the ServerTimeStampSynchronize() method test the connection, and while it's reasonable that it emits the "EWinHTTP exception with message 'winhttp.dll error 12002 (timeout)'" when connecting to a computer where the server software is not running, I found it's strange that it hang the entire program, ie. the UI of the program is not responding, when calling the ServerTimeStampSynchronize() or other methods such as ServiceRegister().
Is there a fix? If I understand it correctly, a http connection in a background thread (not the main thread) should not lock up the UI. Thanks.
I Arnaud,
I haven't been updated my mORMot source for a while, I see the newly added OnEngineAddComputeID event for TSQLRestStorageExternal - looks like it's for **external** DB only. Is there already a similar event for customizing the record ID value for new records on the server side?
Hi Arnaud,
In file SynSQLite3Static, we can find such a comment "under Win64, expects an external sqlite3-64.dll file to be available".
Is there a reason not to provide static linking for the 64bit engine? Thanks.
Great software is about passion, which you have on your open source projects! And I'm really glad to see the progress you have made to support FPC/Lazarus in the past years for mORMot!
@ab, in addtion to updating json properties, we'll also need the ability to add property values.
I think you can investigate the recently added JSON support in postgresql, in order to plan a better support for mORMot.
Hi Arnaud,
It seems that the current mORMot implementation only provides functions (which are accessible within a SQL statement) for reading JSON data stored in a DB, I think the SQLITE3's own functions also support updating the JSON data, eg. updating a property.
HI Arnaud, before I read through the document you pointed out and the new blog post you just written (I guess it's a reaction of this forum post), I'd like to say that you are awesome, you are so fast! ![]()
I'm been wanting an embedded document-based store (don't confuse this with a key-value store) for a long time!
How did your database corrupt?
Hi Arnaud,
Can we either assign an event handler to, or override a method for TSQLRestClientURI, that gets called once LastErrorCode, LastErrorMessage or LastErrorException is set.
The requirement is that I need a centralized place to handle all the errors received from the server, instead of having all error-handling logic scattered around the entire application, something like the try...except block does. Thanks.
By "dirty state" of a TSQLRecord, I mean any property of it is changed (by the UI) and thus needs to be saved to the DB.
What would be a proper and simple way of implementing such a logic? Thanks.
I wish you have done this before I started using mORMot ![]()
Arnaud,
I mean something like this: http://www.generatedata.com/
Hi Arnaud,
Just in case this idea makes sense - integrate a sample record generation feature (for testing) into the framework.
Use ServiceContext: TServiceRunningContext threadvar or the CurrentServiceContext function (from a package).
Hi Arnaud,
I'm using the latest rev. committed today, when being accessed on the server side, ServiceContext.Request is always nil. Any comments? Thanks.
Thanks for your reply. You are right. And now if I think a little bit more about it, actually your commit comments are informative enough to identify which rev. to use. ![]()
Arnaud,
I'm not sure if it makes sense, but obviously you are aware that mORMot is being actively developed (https://github.com/synopse/mORMot/commits/master), sometimes a commit is for bugfixes while another is for new functions, and it's not feasible to keep updated on a daily basis.
May I suggest you to distinguish the revisions by categories, for example, "for beta test only", "considered stable", "new function in progress", etc? That'll help us to decide which revision to use for production.
Have a nice weekend!
Hi Arnaud,
Looks like TWinHTTP includes all functionalities needed to implement a generic REST client?
Hi,
That showcase thread doesn't necessarily represents the mORMot adoption status by the Delphi community.
The Delphi community is a small one when compared to mainstream languages such as Java and C#, that being said, you might be able to have a feel by looking at:
- The stars of the github repository (https://github.com/synopse/mORMot);
- How active this forum is.
- How active the development of the project (check the commits of the above-mentioned github repository).
Got it, thanks for your answers ab!
Hi Arnaud,
I see some of the recent commits contains the word "AsciiDoc", which seems to be one of the supported formats of the REST API document generation. I don't quite understand this part, and I'm out of curiosity, will there be any thing like an "AsciiDoc parser" or something like that appear in the future?
2 and 3. Use a front head, e.g. nginx, and our socket based server.
This is a very common and efficient, safe solution under Linux.
AB, Can you point me to the corresponding paragraph in the document on the details of such setup? I mean, how a mORMot-powered web app works together with Nginx? Maybe you mean the mORMot-powered binary works as a REST API server, and the JS inside HTML pages (served by NginX) communicate with the mORMot server with Ajax?
In my case, I did a quick check, my Samatra PDF Reader seems render all pages, but no bookmarks.
Thanks alpinistbg, per-node increasing Ids for all tables sounds like a simple good idea ![]()
One advantage over full random number is that the UUIDs are timestamp-based and thus sort-able.
Nobody said they're not. Every number is.
Please, see that thread: http://synopse.info/forum/viewtopic.php?id=2160 and particularly that post: http://synopse.info/forum/viewtopic.php … 529#p13529
The main property of each random number generator is to have uniform distribution. In fact, I am not sure what will be the properties when you call Randomize/Random each time in pairs. May be you will get a dependence between the left 42-bits and right 22-bits because probably it gets the clock as source of entropy.
I must say I am really not good at this kind of thing, what I actually mean is that as opposed using pure random numbers for PKs, the timestamp+random PKs increase over time - in other worlds, a new record in most cases will have a greater ID value than the old ones.
You are right on the safe perspective, but since in my scenario the data column is quite low so it should be OK.
@alpinistbb, yes, there is no true random number generation without resorting to some kind of physical devices, and I guess that's out of the scope of the topic.
So my latest method is a time sortable 64bit UUID generation: http://synopse.info/forum/viewtopic.php?id=2522
This is a result of my searching for a UUID algorithm that fits in a Int64, inspired by node-druuid (https://github.com/recurly/node-druuid)
The result is suitable for my scenario where data in branch offices will be synced to a single master database, and I don't want other human-error-prone methods such as allocating UID for each database.
One advantage over full random number is that the UUIDs are timestamp-based and thus sort-able.
Collision testing code (a console program, tests 10 millions times):
var
i, total: Integer;
ids: TDictionary<Int64, Int64>;
begin
try
total := 10000000; // ten millions
ids := TDictionary<Int64, Int64>.Create(total);
for i := 1 to total do
begin
myId := EyUtils.Generate64BitUUID();
Writeln(humansInt(i) + ': ' + IntToStr(myId));
ids.Add(myId, i);
end;
except
on E: Exception do
Writeln(E.ClassName, ': ', E.Message);
end;
ids.Free;
ReadLn;
end.The function, which is actually quite simple:
(*
Generates a time-sortable, 64-bit UUID.
Inspired by: https://github.com/recurly/node-druuid
An uuid comprises:
- A 42-bit timestamp = mill secs. between Now and 2000/01/01 00:00:00:000
- 22 random bits.
*)
function Generate64BitUUID(aOffsetFromYear: Integer = 2000): Int64;
var
startYear: TDateTime;
begin
//first part is 42-bit = timestamp = milliseconds since 2000/01/01 00:00:00:000
startYear := EncodeDateTime(aOffsetFromYear, 1, 1, 0, 0, 0, 0);
Result := MilliSecondsBetween(Now, startYear) shl 22;
//second part = 22 random bits
Randomize;//must, to reduce the chance of random number collision
Result := Result or Random($3FFFFF);
end;Comments and suggestions are welcomed!
Arnaud,
Thanks for your comment. Yes, we discussed about it here earlier, but I'm worried about the "office offset" approach that the users might input wrong Ids and it'll won't be detected until data syncing is under way. At the end, it's the program that has to handle it. We want to avoid this.
re. timestamp-based Id's generation, I'm not sure if the chance of collision is lower, all after all, we'll still have to combine it with random numbers.
I also have found this simple node.js algorithm for generating low-collision UUID: https://github.com/recurly/node-druuid its worth looking at, but I'm not sure how to port it to Pascal...
Testing code below, inserted one million records without a problem using sqlite's random() function to generate the IDs.
I'm going to use this simple approach for generating **unique** Ids for tables that'll be synced from multiple branch offices to a single master server. I think it'll be workable in most cases. Any comments?
program Project5;
{$APPTYPE CONSOLE}
{$R *.res}
uses
Forms,
SysUtils,
SynCommons,
mORMot,
mORMotSQLite3,
SynSQLite3Static;
{$R *.res}
type
TSQLRandomNr = class(TSQLRecord)
private
FNumber: Int64;
published
property Number: Int64 read FNumber write FNumber;
end;
var
myDb: TSQLRestServerDB;
myModel: TSQLModel;
myTable: TSQLTableJSON;
myId, cnt: Int64;
myRec: TSQLRandomNr;
total: Integer;
begin
myModel := TSQLModel.Create([TSQLRandomNr]);
myDb := TSQLRestServerDB.Create(myModel, 'D:\testRandomNr.db3');
myDb.CreateMissingTables;
cnt := 0;
total := 1000000;
Writeln(Format('Inserting %d random numbers', [cnt]));
myDb.TransactionBegin(TSQLRandomNr);
while cnt < total do
begin
myDb.Execute('INSERT INTO Randomnr(ID, Number) SELECT ABS(RANDOM()), ABS(RANDOM())');
Writeln(IntToStr(cnt));
Inc(cnt);
end;
myDb.Commit;
ReadLn;
end.Info about the SQLITE random nr. generator: https://www.sqlite.org/c3ref/randomness.html