Usage of a logging framework

mjustin · 2010-12-14 11:55:18

Hello,

For some areas in the SQLite3 framework, it might be interesting to use some kind of logging framework, for example for

* collecting performance information
* writing generated SQL statements
* notifications about special events
* ...

(a logging framework could also be used by the other parts of the application)

This could be implemented using an existing open source logging library (if the licenses are compatible)

Cheers

Michael

ab · 2010-12-14 14:42:41

At the application level, logging is made easy via the TSQLRecordLog record type.

At the framework level, logging could make a difference.
I've already some code dedicated to this task available. I'll merge it to the main trunk.
It's definitively a good idea.

Thanks for your interest!

edwinsn · 2011-03-09 08:39:53

ab wrote:

At the application level, logging is made easy via the TSQLRecordLog record type

Hi ab,

Can you give me some hint or a few lines of sample code to get me started using TSQLRecordLog?

Can I store the logs to an external database, rather than the main database?

Thank you!

ab · 2011-03-09 09:37:56

The best for a record-based log could be to use our TSynBigTable.
It will keep your main database small, and be very efficient.

The TSQLRecordLog stores the logs as JSON in the main database.
It's purpose was to store some logs associated to another record.

I'll post some sample code later.

edwinsn · 2011-03-09 10:35:59

Great advise! I'll check TSynBigTable!

ab · 2011-03-09 19:41:18

Here is some sample code:

type
  /// the table used for each log entry
  TSQLRecordLogTestOneLog = class(TSQLRecord)
  private
    fStatus: integer;
    fStatusMessage: RawUTF8;
    fTime: TTimeLog;
  published
    /// Time of event (Int64 encoded)
    property Time: TTimeLog read fTime write fTime;
    /// log status, just an integer here (should be an enumeration in your
    // program, to have direct caption on grids)
    property Status: integer read fStatus;
    /// status message (optional)
    property StatusMessage: RawUTF8 read fStatusMessage;
  end;

  /// a table containing some fields, including a log list encoded as JSON
  TSQLRecordLogTest = class(TSQLRecordLog)
  private
    fText: RawUTF8;
    fStatusLog: RawUTF8;
  protected
    /// this entry is used during the logging
    fOneLog: TSQLRecordLogTestOneLog;
  public
    /// release used memory
    destructor Destroy; override;
    /// add one entry to the internal logging status array
    // - use LogTableJSON method to retrieve it as JSON, and save it into
    // StatusLog property in the table
    procedure AddLog(aStatus: Integer; const aMessage: RawUTF8);
    /// retrieve the StatusLog field content, as a TSQLTable
    function LogTable: TSQLTableJSON;
  published
    /// some record field
    property Text: RawUTF8 read fText write fText;
    /// status log, i.e. TSQLRecordLogTestOneLog table encoded as JSON
    property StatusLog: RawUTF8 read fStatusLog write fStatusLog;
  end;

{ TSQLRecordLogTest }

procedure TSQLRecordLogTest.AddLog(aStatus: Integer;
  const aMessage: RawUTF8);
begin
  if fOneLog=nil then
    fOneLog := TSQLRecordLogTestOneLog.Create;
  fOneLog.Time := Iso8601Now;
  fOneLog.Status := aStatus;
  fOneLog.StatusMessage := aMessage;
  Log(fOneLog); // add this content as JSON-log row
end;

destructor TSQLRecordLogTest.Destroy;
begin
  fOneLog.Free;
  inherited;
end;

function TSQLRecordLogTest.LogTable: TSQLTableJSON;
begin
  result := TSQLTableJSON.Create([TSQLPhDEventLog],'',fClientStatusLog);
  result.IDColumnHide;
end;

Take a look at the comments to guess how it works.

In short: you create some logging records, appended in some JSON field which is stored in a table record.

Then you can get the logging list from the JSON field, with the LogTable method, the assign it to the UI Grid of the framework for direct display.

edwinsn · 2011-03-10 06:12:27

Thanks for the sample code! But I don't understand, what's the advantage of using TSQLTableJSON?

One more question, for some reasons, I can't use tbigtable but want to store the logs to a SQLITE DB instead. I'll use TSQLRestClientURINamedPipe server component, and I'm wondering if I can have multiple TSQLRestClientURINamedPipe instances - one for the main data while another for logs only, so that I can save my logs to a separate database. But I read the following line in the source code:

// - allows only one ExportServer*() by running process

Does it mean it's not possible? And I found this similar thread about TSQLite3HttpServer

ab · 2011-03-10 07:00:48

From the client side, TSQLTableJSON will contain all logs records, decoded from its JSON field.
It can be used to populate a Grid to view all logs.

This grid is very fast, and will do most of the formating work for you, thanks to its link to the ORM properties:
http://synopse.info/forum/viewtopic.php?id=140

edwinsn · 2011-03-10 08:41:16

I got it!

And you might overlooked my new question - "is it possible to have multiple TSQLRestClientURINamedPipe instances in a single process?"

Thanks!

ab · 2011-03-10 10:20:13

You can have only multipleTSQLRestServerNamedPipe per process, if you specify diverse ApplicationName parameter values to TSQLRestServer.ExportServerNamedPipe(), one per each TSQLRestServer you create.

For TSQLRestClientURINamedPipe, it only allow one concurrent access.
You could try to change the following code:

constructor TSQLRestClientURINamedPipe.Create(aModel: TSQLModel;
  const ApplicationName: TFileName);
var Pipe: cardinal;
procedure CreatePipe;
begin
  Pipe := CreateFile(pointer(fPipeName), GENERIC_READ or GENERIC_WRITE,
{$ifdef ANONYMOUSNAMEDPIPE}
    FILE_SHARE_READ or FILE_SHARE_WRITE,
    nil, OPEN_EXISTING, SECURITY_SQOS_PRESENT or SECURITY_ANONYMOUS, 0);
{$else}
    FILE_SHARE_READ or FILE_SHARE_WRITE, // this is added to allow sharing
    @fPipeSecurityAttributes, OPEN_EXISTING,
    FILE_ATTRIBUTE_NORMAL or FILE_FLAG_OVERLAPPED, 0);
{$endif}
end;

Only HTTP/1.1 allows multiple concurrent connections at once.

edwinsn · 2011-03-10 11:26:25

Hi ab,

Thanks for the info. I think I've come up with an alternative (maybe better) approach - The multiple clients send the logs to the single server via a RESTful JSON method on the server side, and in that method the log will in turn be stored into another database by via another TSQLRest object.

ab · 2011-03-10 18:54:04

That was a good workaround, perhaps more multi-tier compatible: the client don't have to know how it's stored on the server side, but use its own method.

ab · 2011-04-14 09:49:45

mjustin wrote:

For some areas in the SQLite3 framework, it might be interesting to use some kind of logging framework, for example for
* collecting performance information
* writing generated SQL statements
* notifications about special events
* ...

You had a dream... and it's now reality!
See http://synopse.info/forum/viewtopic.php?pid=1809#p1809

mORMot Open Source

#1 2010-12-14 11:55:18

Usage of a logging framework

#2 2010-12-14 14:42:41

Re: Usage of a logging framework

#3 2011-03-09 08:39:53

Re: Usage of a logging framework

#4 2011-03-09 09:37:56

Re: Usage of a logging framework

#5 2011-03-09 10:35:59

Re: Usage of a logging framework

#6 2011-03-09 19:41:18

Re: Usage of a logging framework

#7 2011-03-10 06:12:27

Re: Usage of a logging framework

#8 2011-03-10 07:00:48

Re: Usage of a logging framework

#9 2011-03-10 08:41:16

Re: Usage of a logging framework

#10 2011-03-10 10:20:13

Re: Usage of a logging framework

#11 2011-03-10 11:26:25

Re: Usage of a logging framework

#12 2011-03-10 18:54:04

Re: Usage of a logging framework

#13 2011-04-14 09:49:45

Re: Usage of a logging framework

Board footer