#851 Re: mORMot 1 » Bug - int64 values from a database view wrongly parsed as float values » 2015-10-02 11:57:23

But even working in REST mode, SQL has its advantage and we need to execute complex SQL to generate reports.

#852 Re: mORMot 1 » Bug - int64 values from a database view wrongly parsed as float values » 2015-10-01 03:37:36

IMHO, the JSON response returned from the server should include the field type info.

#854 Re: mORMot 1 » Bug - int64 values from a database view wrongly parsed as float values » 2015-09-30 12:33:19

Thanks ab, I'll utilize the SetFieldType method. I'm curious - doesn't the sqlite engine return field type info for the result set?

#855 Re: mORMot 1 » Bug - int64 values from a database view wrongly parsed as float values » 2015-09-29 17:10:55

A sample JSON response I got while debugging this issue:

[{"DocId":2078946242580065800,"DocDate":"2015-09-15","EventType":"e_po","ProductNr":"000020"}]

note 1: the DocId value is long...
note 2: I'm using the latest version from github.

#856 mORMot 1 » Bug - int64 values from a database view wrongly parsed as float values » 2015-09-29 16:45:47

edwinsn
Replies: 17

The table:

CREATE TABLE CustInv (
    ID                INTEGER PRIMARY KEY AUTOINCREMENT,
...
);

The view:

CREATE VIEW v_event_inv AS
    SELECT header.ID AS DocId,
           ...
      FROM CustInv header

The delphi code:

var
  myFieldType: TSQLFieldType;
  myFieldTypeName: string;
  myFieldValue: string;
  tmpTable: TSQLTableJSON;
begin
  tmpTable := AppCtx.Db.ExecuteList([], 'SELECT * FROM v_event_inv');

  //some testing code
    {$IFDEF DEBUG}
    myFieldType := tmpTable.FieldType(0);//expected: sftID. Actual: sftFloat
    myFieldTypeName := GetEnumName(TypeInfo(TSQLFieldType), Integer(myFieldType));
    myFieldValue := tmpTable.FieldValue('DocId', 1); //correct
    CodeSite.Send(myFieldValue, myFieldTypeName);
    {$ENDIF}

PS: I noticed the bug after using ToClientDataSet, but tracked it down in ExecuteList()...

#857 Re: mORMot 1 » BIG news: SQLite looks to be adding JSON support! » 2015-09-23 06:02:09

Update: SQLite has already the json1 extension and it's expected to be integrated into the core once it's stabilized: https://www.sqlite.org/draft/json1.html

Obviously can be very useful for people who need very flexible, even nested database schema.

#859 mORMot 1 » Any method to get all field names in CSV format for a TSQLRecord? » 2015-09-18 13:14:51

edwinsn
Replies: 2

Hi Ab,

Is there a way to get all field names in CSV format for a TSQLRecord?

I need such a function because I want to avoid updating some of the fields via the '' parameter when calling Update().

It'll be idea to be able to:

const
  cFieldNotToUpdate = 'field1';
var
  fieldsToUpdate: string;
  myRec: TSQLPerson;
begin
  fieldsToUpdate := myRec.GetAllFieldNamesInCSVFormat; 
  //so that I can:
  fieldsToUpdate := ReplaceStr(fieldsToUpdate, cFieldNotToUpdate , '');
  aDb.Update(myRec, fieldsToUpdate);
  ...

I hope'll you get my point.

Or even better, in future versions to provide a new parameter for the Update() method called 'fieldsNotToUpdate'  which does the opposite of the 'customFields' param smile

#860 Re: mORMot 1 » SQLite FTS5 and unicode support » 2015-09-18 13:08:41

I'm glad to know there is the new fts5! Even it's experiential, I suggest to add it to the mORMot framework anyway!

#861 Re: mORMot 1 » Is this possible with mORMot ? » 2015-09-16 06:59:27

I think you can do that with mORMot, but I maybe a queue like this http://zeromq.org/ fits your needs more?

#862 Re: mORMot 1 » Contribution: TSynRestDataset » 2015-09-13 08:06:21

Sounds interesting. Thanks for sharing.

#864 mORMot 1 » BatchUpdate should support "DoNotAutoComputeFields" & "aCustomFields" » 2015-09-09 07:55:42

edwinsn
Replies: 2

Hi Arnaud,

Would you add the "DoNotAutoComputeFields" param to the BatchUpdate method? There are situation where you don't need to update the TModTime fields, like doing a backup or syncing.

PS, do you accept github pull requests for source code contribution?

#865 Re: mORMot 1 » OmniThreadLibrary's not working in mORMot's server-side method? » 2015-09-07 17:21:44

Thanks to the help by the author of OTL Primoz, problem's fixed for approach A, OtlTaskControl.CreateTask needs the .Unobserved modifier!

#867 Re: mORMot 1 » OmniThreadLibrary's not working in mORMot's server-side method? » 2015-09-07 12:12:23

Well, after some more testing, it turned out the problem has nothing to do with TSQLHttpClient (because I replaced the http client connection with Sleep(5000) to rule it out), but there seems to be some incompatibility between OmniThreadLibrary (latest github SVN) and mORMot (I'm using the latest version about 1 day ago).

My server is configured like this:

  FDbSvr := TDbServer.Create(FDataModel, GetDbFileName, False);
  SetupServerLogging;
  FDBSvr.PublishAllServices;

  (*
    WAL might increase the risk of data losses!
    http://liferea.blogspot.jp/2012/03/morpheus-suggested-in-comment-to-check.html
    http://www.sqlabs.com/sqlitedoctor_details.php
    http://stackoverflow.com/a/20970876/133516
  *)
  FDbSvr.DB.WALMode := False;
  FDbSvr.DB.Synchronous := smFull;//ensure max data stability!

  FHttpSvr := TSQLHttpServer.Create(IntToStr(gServerPort), [FDbSvr], '+', HTTP_DEFAULT_MODE);
  FHttpSvr.AccessControlAllowOrigin := '*'; // allow cross-site AJAX queries;

and I have an interface-based service on the server side, with default configuration:

  /// this is the main interface exposed by the server
  TServerApi = class(TInterfacedObject, IServerApi)
  public
    function StartCommonDataDownload(aManualCall: Boolean): Boolean;
  end;

And in this StartCommonDataDownload() method, a thread will be started, like this:

/// approach A, the method will not return until 5 seconds are passed
function TServerApi.StartCommonDataDownload(aManualCall: Boolean): Boolean;
begin
  OtlTaskControl.CreateTask(
    procedure (const task: IOmniTask)
    begin
        Sleep(5000);
    end
  ).Run;
end;

/// approach B, the method will not return until 5 seconds are passed
function TServerApi.StartCommonDataDownload(aManualCall: Boolean): Boolean;
begin
    TThread.Queue(nil, procedure
      begin
        Sleep(5000);
      end
    );
end;

  /// approach C, works!
  Parallel.Async(procedure
  begin
      Sleep(5000);
  end);

Only approach C worked. Does it have anything to do with the way the TSQLHttpServer is running in? I run it with administrator and I guess it runs in the http.sys mode? (update: : tried useBidirSocket just now, same problem...) I also noticed the OmniThreadLibrary 's communication sub-system also is not working inside an interfaced service method...

Thanks for any help.

#868 mORMot 1 » OmniThreadLibrary's not working in mORMot's server-side method? » 2015-09-07 08:30:43

edwinsn
Replies: 5

Update 1: This post has been updated to reflect the latest findings, it seems that the OmniThreadLibrary's CreateTask() is in a way not compatible with mORMot, please do directly to http://synopse.info/forum/viewtopic.php?id=2852#p17618 for more info.

Hi Arnaud,

In my program there is a background thread which creates a TSQLHttpClient and calls the ServerTimeStampSynchronize() method test the connection, and while it's reasonable that it emits the "EWinHTTP exception with message 'winhttp.dll error 12002 (timeout)'" when connecting to a computer where the server software is not running, I found it's strange that it hang the entire program, ie. the UI of the program is not responding, when calling the ServerTimeStampSynchronize() or other methods such as ServiceRegister().

Is there a fix? If I understand it correctly, a http connection in a background thread (not the main thread) should not lock up the UI. Thanks.

#869 Re: mORMot 1 » OnEngineLockedNextID and EngineIgnoreID patch » 2015-09-04 03:58:47

I Arnaud,

I haven't been updated my mORMot source for a while, I see the newly added OnEngineAddComputeID event for TSQLRestStorageExternal - looks like it's for **external** DB only. Is there already a similar event for customizing the record ID value for new records on the server side?

#870 mORMot 1 » Sugesstion - also provide .obj files for the 64bit engine. » 2015-09-04 03:52:45

edwinsn
Replies: 1

Hi Arnaud,

In file SynSQLite3Static, we can find such a comment "under Win64, expects an external sqlite3-64.dll file to be available".

Is there a reason not to provide static linking for the 64bit engine? Thanks.

#871 Re: Delphi » GetIt.RealName := 'GetItIfItWontHurtOurSells' » 2015-08-30 08:59:13

Great software is about passion, which you have on your open source projects! And I'm really glad to see the progress you have made to support FPC/Lazarus in the past years for mORMot!

#873 Re: mORMot 1 » BIG news: SQLite looks to be adding JSON support! » 2015-08-25 06:38:06

@ab, in addtion to updating json properties, we'll also need  the ability to add property values.

I think you can investigate the recently added JSON support in postgresql, in order to plan a better support for mORMot.

#874 Re: mORMot 1 » BIG news: SQLite looks to be adding JSON support! » 2015-08-24 09:34:59

Hi Arnaud,

It seems that  the current mORMot implementation only provides functions (which are accessible within a SQL statement) for reading JSON data stored in a DB, I think the SQLITE3's own functions also support updating the JSON data, eg. updating a property.

#875 Re: mORMot 1 » BIG news: SQLite looks to be adding JSON support! » 2015-08-23 12:25:38

HI Arnaud, before I read through the document you pointed out and the new blog post you just written (I guess it's a reaction of this forum post), I'd like to say that you are awesome, you are so fast! smile

#876 mORMot 1 » BIG news: SQLite looks to be adding JSON support! » 2015-08-23 09:15:49

edwinsn
Replies: 12

I'm been wanting an embedded document-based store  (don't confuse this with a key-value store) for a long time!


https://www.reddit.com/r/programming/co … n_support/

https://news.ycombinator.com/item?id=10103695

#879 mORMot 1 » Can we get notified once LastErrorMessage is set? » 2015-07-28 08:19:07

edwinsn
Replies: 4

Hi Arnaud,

Can we either assign an event handler to, or override a method for TSQLRestClientURI, that gets called once LastErrorCode, LastErrorMessage or LastErrorException is set.

The requirement is that I need a centralized place to handle all the errors received from the server, instead of having all error-handling logic scattered around the entire application, something like the try...except block does. Thanks.

#880 mORMot 1 » Need advice on implementing "dirty state checking" » 2015-07-27 14:00:43

edwinsn
Replies: 0

By "dirty state" of a TSQLRecord, I mean any property of it is changed (by the UI) and thus needs to be saved to the DB.

What would be a proper and simple way of implementing such a logic? Thanks.

#881 Re: mORMot 1 » Taming the Mormot » 2015-07-15 04:54:17

I wish you have done this before I started using mORMot big_smile

#884 mORMot 1 » Suggestion, if it makes sense - sample record generator » 2015-07-07 11:20:13

edwinsn
Replies: 4

Hi Arnaud,

Just in case this idea makes sense - integrate a sample record generation feature (for testing)  into the framework.

#885 Re: mORMot 1 » Simple Interface based server with database ! » 2015-05-31 14:20:12

ab wrote:

Use ServiceContext: TServiceRunningContext threadvar or the CurrentServiceContext function (from a package).

See http://synopse.info/files/html/api-1.18 … ICECONTEXT

Hi Arnaud,

I'm using the latest rev. committed today, when being accessed on the server side, ServiceContext.Request is always nil. Any comments? Thanks.

#886 Re: mORMot 1 » Mark "recommended to upgrade" github revisions? » 2015-05-30 16:10:13

Thanks for your reply. You are right. And now if I think a little bit more about it, actually your commit comments are informative enough to identify which rev. to use. big_smile

#887 mORMot 1 » Mark "recommended to upgrade" github revisions? » 2015-05-29 12:04:57

edwinsn
Replies: 2

Arnaud,

I'm not sure if it makes sense, but obviously you are aware that mORMot is being actively developed (https://github.com/synopse/mORMot/commits/master), sometimes a commit is for bugfixes while another is for new functions, and it's not feasible to keep updated on a daily basis.

May I suggest you to distinguish the revisions by categories, for example, "for beta test only", "considered stable", "new function in progress", etc? That'll help us to decide which revision to use for production.

Have a nice weekend!

#888 Re: mORMot 1 » How make Post to a Web Services that response with http code and Json » 2015-05-25 14:26:21

Hi Arnaud,

Looks like TWinHTTP includes all functionalities needed to implement a generic REST client?

#889 Re: mORMot 1 » Usage figures of mORMot » 2015-05-25 14:21:22

Hi,

That showcase thread doesn't necessarily represents the mORMot adoption status by the  Delphi community.

The Delphi community is a small one when compared to mainstream languages such as Java and C#, that being said, you might be able to have a feel by looking at:
- The stars of the github repository (https://github.com/synopse/mORMot);
- How active this forum is.
- How active the development of the project (check the commits of the above-mentioned github repository).

#890 Re: mORMot 1 » AsciiDoc support? » 2015-05-03 15:04:41

Got it, thanks for your answers ab!

#891 mORMot 1 » AsciiDoc support? » 2015-05-03 03:42:02

edwinsn
Replies: 2

Hi Arnaud,

I see some of the recent commits contains the word "AsciiDoc", which seems to be one of the supported formats of the REST API document generation. I don't quite understand this part, and I'm out of curiosity, will there be any thing like an "AsciiDoc parser" or something like that appear in the future?

#892 Re: mORMot 1 » Interface based Server on Lazarus + FPC + HTTPS » 2015-04-29 07:48:29

ab wrote:

2 and 3. Use a front head, e.g. nginx, and our socket based server.
This is a very common and efficient, safe solution under Linux.

AB, Can you point me to the corresponding paragraph in the document on the details of such setup? I mean, how a mORMot-powered web app works together with Nginx? Maybe you mean the mORMot-powered binary works as a REST API server, and the JS inside HTML pages (served by NginX) communicate with the mORMot server with Ajax?

#893 Re: mORMot 1 » Synopse mORMot Framework SAD 1.18.pdf is corrupted » 2015-04-26 14:00:19

In my case, I did a quick check, my Samatra PDF Reader seems render all pages, but no bookmarks.

#894 Re: mORMot 1 » Time-sortable, 64bit UUID generation, 10 millions without collision » 2015-04-19 14:35:36

Thanks alpinistbg, per-node increasing Ids for all tables sounds like a simple good idea smile

#895 Re: mORMot 1 » Time-sortable, 64bit UUID generation, 10 millions without collision » 2015-04-19 03:34:23

alpinistbg wrote:

One advantage over full random number is that the UUIDs are timestamp-based and thus sort-able.

Nobody said they're not. Every number is.

Please, see that thread: http://synopse.info/forum/viewtopic.php?id=2160 and particularly that post: http://synopse.info/forum/viewtopic.php … 529#p13529

The main property of each random number generator is to have uniform distribution. In fact, I am not sure what will be the properties when you call Randomize/Random each time in pairs. May be you will get a dependence between the left 42-bits and right 22-bits because probably it gets the clock as source of entropy.

I must say I am really not good at this kind of thing, what I actually mean is that as opposed using pure random numbers for PKs, the timestamp+random PKs increase over time - in other worlds, a new record in most cases will have a greater  ID value than the old ones.

#896 Re: mORMot 1 » Time-sortable, 64bit UUID generation, 10 millions without collision » 2015-04-18 10:30:11

You are right on the safe perspective, but since in my scenario the data column is quite low so it should be OK.

#897 Re: mORMot 1 » 1M records tested, SQLite's random no. generator's really high quality » 2015-04-18 05:35:15

@alpinistbb, yes, there is no true random number generation without resorting to some kind of physical devices, and I guess that's out of the scope of the topic.

So my latest method is a time sortable 64bit UUID generation: http://synopse.info/forum/viewtopic.php?id=2522

#898 mORMot 1 » Time-sortable, 64bit UUID generation, 10 millions without collision » 2015-04-18 05:32:50

edwinsn
Replies: 9

This is a result of my searching for a UUID algorithm that fits in a Int64, inspired by node-druuid (https://github.com/recurly/node-druuid)

The result is suitable for my scenario where data in branch offices will be synced to a single master database, and I don't want other  human-error-prone methods such as allocating UID for each database.

One advantage over full random number is that the UUIDs are timestamp-based and thus sort-able.

Collision testing code (a console program, tests 10 millions times):

var
  i, total: Integer;
  ids: TDictionary<Int64, Int64>;
begin
  try
    total := 10000000; // ten millions
    ids := TDictionary<Int64, Int64>.Create(total);

    for i := 1 to total do
    begin
      myId := EyUtils.Generate64BitUUID();
      Writeln(humansInt(i) + ': ' + IntToStr(myId));
      ids.Add(myId, i);
    end;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
  ids.Free;
  ReadLn;
end.

The function, which is actually quite simple:

(*
  Generates a time-sortable, 64-bit UUID.
  Inspired by: https://github.com/recurly/node-druuid
  An uuid comprises:
    - A 42-bit timestamp = mill secs. between Now and 2000/01/01 00:00:00:000
    - 22 random bits.
*)
function Generate64BitUUID(aOffsetFromYear: Integer = 2000): Int64;
var
  startYear: TDateTime;
begin
  //first part is 42-bit = timestamp = milliseconds since 2000/01/01 00:00:00:000
  startYear := EncodeDateTime(aOffsetFromYear, 1, 1, 0, 0, 0, 0);
  Result := MilliSecondsBetween(Now, startYear) shl 22;

  //second part = 22 random bits
  Randomize;//must, to reduce the chance of random number collision
  Result := Result or Random($3FFFFF);
end;

Comments and suggestions are welcomed!

#899 Re: mORMot 1 » 1M records tested, SQLite's random no. generator's really high quality » 2015-04-17 15:39:02

Arnaud,

Thanks for your comment. Yes, we discussed about it here earlier, but I'm worried about the "office offset" approach that the users might input wrong Ids and it'll won't be detected until data syncing is under way. At the end, it's the program that has to handle it. We want to avoid this.

re. timestamp-based Id's generation, I'm not sure if the chance of collision is lower, all after all, we'll still have to combine it with random numbers.

I also have found this simple node.js algorithm for generating low-collision UUID: https://github.com/recurly/node-druuid its worth looking at, but I'm not sure how to port it to Pascal...

#900 mORMot 1 » 1M records tested, SQLite's random no. generator's really high quality » 2015-04-17 14:32:43

edwinsn
Replies: 5

Testing code below, inserted one million records without a problem using sqlite's random() function to generate the IDs.

I'm going to use this simple approach for generating **unique** Ids for tables that'll be synced from multiple branch offices to a single master server. I think it'll be workable in most cases.  Any comments?

program Project5;

{$APPTYPE CONSOLE}
{$R *.res}

uses
  Forms,
  SysUtils,
  SynCommons,
  mORMot,
  mORMotSQLite3,
  SynSQLite3Static;

{$R *.res}

type
  TSQLRandomNr = class(TSQLRecord)
  private
    FNumber: Int64;
  published
    property Number: Int64 read FNumber write FNumber;
  end;

var
  myDb: TSQLRestServerDB;
  myModel: TSQLModel;
  myTable: TSQLTableJSON;
  myId, cnt: Int64;
  myRec: TSQLRandomNr;
  total: Integer;
begin
  myModel := TSQLModel.Create([TSQLRandomNr]);
  myDb := TSQLRestServerDB.Create(myModel, 'D:\testRandomNr.db3');
  myDb.CreateMissingTables;

  cnt := 0;
  total := 1000000;
  Writeln(Format('Inserting %d random numbers', [cnt]));
  myDb.TransactionBegin(TSQLRandomNr);
  while cnt < total do
  begin
    myDb.Execute('INSERT INTO Randomnr(ID, Number) SELECT ABS(RANDOM()), ABS(RANDOM())');
    Writeln(IntToStr(cnt));

    Inc(cnt);
  end;
  myDb.Commit;

  ReadLn;

end.

Info about the SQLITE random nr. generator: https://www.sqlite.org/c3ref/randomness.html

Board footer

Powered by FluxBB