mORMot Open Source

ASiwon

Member

From: Poland

Registered: 2015-01-30

Posts: 82

Problem with opening compressed log files.

Hello,

I have tried to read log records from compressed files using LogView tool. But files are not opened correctly. In LogView just nothing happens after selecting synlz log file - empty list is shown. I have checked where is the problem in source code and I found that in function: StreamUnSynLZ hash value is different in uncompressed data and header:

  if (SynLZdecompress1(P,Head.CompressedSize,result.Memory)<>Head.UnCompressedSize) or
     (Hash32(result.Memory,Head.UnCompressedSize)<>Head.HashUncompressed) then
    FreeAndNil(result);

For now I have only two synlz files but for both it is impossible to read their content. Those two files are not important to me but in the future this may make me some problems. Is it possible to check where exist a problem?

---

best regards
Adam Siwon

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

We never saw this yet.

What is the uncompressed content, before FreeAndNil(result)?
Is the content more or less as expected?

ASiwon

Member

From: Poland

Registered: 2015-01-30

Posts: 82

Re: Problem with opening compressed log files.

For testing purposes I have changed end of function StreamUnSynLZ to:

  if (SynLZdecompress1(P,Head.CompressedSize,result.Memory)<>Head.UnCompressedSize) or
     (Hash32(result.Memory,Head.UnCompressedSize)<>Head.HashUncompressed) then
  begin
    Result.Seek(0, soFromBeginning);
    Result.SaveToFile('Test.log');
    FreeAndNil(result);
  end;

The file after saving has: 102438644 bytes and Head.UnCompressedSize has value: 102404552. Saved file is readable and I don't see any damaged data inside.

---

best regards
Adam Siwon

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

I do not get it...
The result size is set like this (line just above your quote):

result.Size := Head.UnCompressedSize

So I do not understand how result.Size may not match Head.UnCompressedSize...

Could you put a link to the synlz compressed file?

ASiwon

Member

From: Poland

Registered: 2015-01-30

Posts: 82

Re: Problem with opening compressed log files.

Wrong values it is my fault. I have problems with two files. Size from header was from second file and size from stream was from first file. Thats why values was different. So sizes are equals but still are differences on hash values.
You can download compressed files from:
http://gidex.com.pl/download/BkSerwerService.1.synlz
http://gidex.com.pl/download/BkSerwerService.2.synlz

---

best regards
Adam Siwon

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

Could you try http://synopse.info/fossil/info/536ecbc9f7 ?

I will look at your files.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

I checked the files.

The hash does not match...
Pretty weird, I've never seen this behavior.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

After more investigation, the hash algorithm is correct.

But we reached an issue with SynLZ.

The following lines:

20150320 09481911 debug 		Czas następnego wykonania nie został osiągnięty.
20150320 09481911  -    	00.000.013
20150320 09481911 info  	Zakończono przetwarzanie okresowe.
20150320 09481911  -    00.000.017
20150320 09482011  +    TBkSerwisWagiThread(015FC480).PrzetwarzajOkresowo
20150320 09482011  +    	TBkWagaSerwisShared(015DFA80).PrzetwarzajOkresowo

were compressed as such:

20150320 09481911 debug 		Czas następnego wykonania nie został osiągnięty.
20150320 09481911  -    	00.000.013
20150320 09481913 info  	Zakończono przetwarzanie okresowe.
20150320 09481911  -    00.000.017
20150320 09482011  +    TBkSerwisWagiThread(015FC480).PrzetwarzajOkresowo
20150320 09482011  +    	TBkWagaSerwisShared(015DFA80).PrzetwarzajOkresowo

As you can see, '09481911 info' became '09481913 info'...
Pretty awful!
I will fix it ASAP!

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 14,655

Website

Re: Problem with opening compressed log files.

The issue was not with the SynLZ algorithm itself, but with FileSynLZ/FileUnsynLZ functions

It should be fixed by http://synopse.info/fossil/info/de48bcfedd

Any .synlz log file compressed before this commit may be affected by the issue (which was very difficult to reproduce).
The implementation has been fixed, but the format on disk did non change.

Thanks for the feedback, and supplying the log files!
Without them, I would not have been able to reproduce the problem.

ASiwon

Member

From: Poland

Registered: 2015-01-30

Posts: 82

Re: Problem with opening compressed log files.

I have download the current version from github. If I will find any problem then I will let you know. Thank you very much for great support. As always.

---

best regards
Adam Siwon