Enhanced Log Viewer

ab · 2011-08-20 08:22:55

Since the log files tend to be huge (for instance, if you set the logging for our unitary tests, the 6,000,000 unitary tests create a 280 MB log file), a log viewer was definitively in need.

The log-viewer application is available as source code in the "Samples" folder, in the "11 - Exception logging" sub-folder.

Open log files

You can run it with a specified log file on the command line, or use the "Open" button to browse for a file. That is, you can associate this tool with your .log files, for instance, and you'll open it just by double-clicking on such files.

Note that if the file is not in our TSynLog format, it will still be opened as plain text. You'll be able to browse its content and search within, but all the nice features of our logging won't be available, of course.

Log browser

The screen is divided into three main spaces:
- On the left side, the panel of commands;
- On the right side, the log events list;
- On the middle, an optional list of method calls (not shown by default).

The command panel allows to Open a .log file, see the global Stats about its content (customer-side hardware and software running configuration, general numbers about the log), and even ask for a source code line number and unit name from an hexadecimal address available in the log, by browsing for the corresponding .map file (could be handy if you did not deliver the .map content within your main executable - which you should have to, IMHO).

Just below the "Open" button, there is an edit field available, with a ? button. Enter any text within this edit field, and it will be searched within the log events list. Search is case-insensitive, and was designed to be fast. Clicking on the ? button (or pressing the F3 key) allows to repeat the last search.

In the very same left panel, you can see all existing events, with its own color and an associated check-box. Note that only events really encountered in the .log file appear in this list, so its content will change between log files. By selecting / un-selecting a check-box, the corresponding events will be instantaneously displayed / or not on the right side list of events. You can click on the Filter button (or right click on the events check-box list) to select a predefined set of events.

The right colored event list follows the events appended to the log, by time order. When you click on an event, its full line content is displayed at the bottom on the screen, in a memo.

Having all SQL and Client-Server events traced in the log is definitively a huge benefit for customer support and bug tracking.

Customer-side profiler

One distinctive feature of the TSynLog logging class is that it is able to map methods or functions entering/leaving (using the Enter method), and trace this into the logs. The corresponding timing is also written within the "Leave" event, and allows application profiling from the customer side. Most of the time, profiling an application is done during the testing, with a test environment and database. But this is not, and will never reproduce the exact nature of the customer use: for instance, hardware is not the same (network, memory, CPU), nor the software (Operating System version, [anti-]virus installed)... By enabling customer-side method profiling, the log will contain all relevant information. Those events are named "Enter" / "Leave" in the command panel check-box list, and written as + and - in the right-sided event list.

The "Methods profiler" options allows to display the middle optional method calls list. Several sort order are available: by name (alphabetical sort), by occurrence (in running order, i.e. in the same order than in the event log), by time (the full time corresponding to this method, i.e. the time written within the "Leave" event), and by proper time (i.e. excluding all time spent in the nested methods).

The "Merge method calls" check-box allows to regroup all identical method calls, according to their name. In fact, most methods are not called once, but multiple time. And this is the accumulated time spent in the method which is the main argument for code profiling.

I'm quite sure that the first time you'll use this profiling feature on a huge existing application, you'll find out some bottlenecks you would have never thought about before.

The source file is available in our source code repository, and a compiled version can be downloaded directly from http://synopse.info/files/LogView.zip

ab · 2011-08-26 17:47:39

I didn't mention this, but the Open button is able to load either .log or .synlz files.

The .synlz files are the compressed .log files, using our fast SynLz algorithm, as created by the archival feature embedded into SynCommons.pas

You can have auto-archival into .zip files, but the log viewer won't open them directly - yet.

ab · 2012-12-29 18:08:56

Thread information has been added to the logs.
If new property TSynLogFamily.PerThreadLog is set to ptIdentifiedInOnFile, a new column will be added for each logged row.

LogViewer has been updated to allow easy and efficient multi-thread process logging.

See http://synopse.info/forum/viewtopic.php?pid=5869#p5869

nedko · 2013-02-19 10:12:38

Hello AB, I am very fascinated by the possibilities your logclasses/logviewer open at hand. It seems like a great amount of work done here. Thank you for sharing all this treasure with the community!
The thing the urged me to write you here is that I seem to not be able to make logview.exe work. It just does show anything. I feed it the log file that TSynLog created but nothing appears in logview. The log file itself contains some small info: test logging + exception crash info.
What could I possibly do wrong?

ab · 2013-02-19 14:02:49

Did you compile the latest 1.18 version from http://synopse.info/fossil ?

Are you giving the .log file via command line?
Or the folder browser available on the left side of the tool?
By using "Open" button?

nedko · 2013-02-20 11:37:13

I compiled it from the sources. It's now ok. Thank you.

ChrisTT · 2013-02-26 00:21:11

Is the file size limit necessary? I have a 4.7GB log file that won't open, but if I make a copy of the first 50MB of it that opens fine despite the last line being truncated arbitrarily. A 1.4GB file also will not open (that is, it shows in the list of files but clicking on it leaves the view area blank). If I grab the first 1GB of the file it loads and I can click "merge method calls". which is what I want.

Tracing into LogViewer when it tries to load that file it is trying to open it as text but not displaying anything.

    if FLog.EventLevel=nil then begin // if not a TSynLog file -> open as plain text

That line is going the "not a log file" path.

Can this be fixed (obviously at the cost of log file parsing taking a very long time)?

I'm using SynLogging as a cheap profiler and for what I'm doing I'd rather be able to see time breakdowns for the whole operation I'm profiling if I can. Unfortunately for the code I'm working on the way the input grows is through increased complexity rather than just more repetitions of the same basic data. (viz, we know the processing cost is O(A*B*C*D*E*F*G), where all those factors are increasing at similar rates, but we don't know exactly where the time goes as they grow - the exponents on each factor).

(I've also had to build LogViewer to avoid the AV on start - the zipped exe seems to be broken. Also, for other readers, the project is in "SQLite3\Samples\11 - Exception logging" within the main source zip file).

Last edited by ChrisTT (2013-02-26 02:15:09)

ab · 2013-02-26 05:31:00

In some cases, size matters.
Such huge files can't be opened by the Log Viewer, since it can not fit in memory.
For performance, log files are mapped in RAM.
Under 32 bit Windows, it is not possible to map such a big amount of data.
64 bit compilation will be able to open such huge files. It is on track...

But log file autorotation is certainly the best answer to your problem.
It is on the roadmap - see http://synopse.info/fossil/tktview?name=72feb66d45

ChrisTT · 2013-02-26 22:36:07

ab wrote:

But log file autorotation is certainly the best answer to your problem.
It is on the roadmap - see http://synopse.info/fossil/tktview?name=72feb66d45

Yeah, that sounds excellent. I can't wait

Also, at the risk of sounding like someone too lazy to RTFM about submitting code, I added a popup menu to the List with a "copy" item on it, so I can copy the currently visible rows out of the list. The code follows in case you are willing to add it to the project.

procedure TMainLogView.ListMenuCopyClick(Sender: TObject);
var s:string;
    i,Index:integer;
begin
  // based on ListDrawCell
  s := '';
  for i := 0 to FLogSelectedCount-1 do
  begin
    Index := FLogSelected[i];
    s := s + TimeToStr(FLog.EventDateTime(Index)) + #9
           + FEventCaption[FLog.EventLevel[Index]] + #9;
    if FLog.EventThread<>nil then
      s := s + IntToString(cardinal(FLog.EventThread[Index])) else
      s := s + UTF8ToString(StringReplaceAll(FLog.EventText[Index],#9,'   '));
    s := s + UTF8ToString(StringReplaceAll(FLog.EventText[Index],#9,'   ')) + sLineBreak;
  end;
  Clipboard.AsText := s;
end;

ab · 2013-03-05 08:14:14

Good idea.

I've added a new list popup menu with "Copy" action to LogViewer tool.
(implementation a bit diverse than yours, which did copy all the content, not only selected lines)

See http://synopse.info/fossil/info/4abd573b92

Thanks for the proposal and code!

ChrisTT · 2013-03-07 02:01:07

Cool, thanks for that. Happy to have made a useful suggestion too

clopman16 · 2019-10-26 09:42:49

Hello,

I want to make the log viewer was updated when new event has arrive into .log file. How can this be done?
Thanks.

ab · 2019-10-26 09:48:51

This is not supported from file yet...

You need to create a client/server connection to have real-time logs. But it is more complex.

clopman16 · 2019-10-26 11:14:02

OK,
how I can read last 10, 100, 1000 lines from .log file?

mORMot Open Source

#1 2011-08-20 08:22:55