#1 Low level and performance » SynLZ vs QuickLZ » 2012-01-03 15:58:12

BeginEnd;
Replies: 8

QuickLZ's site (quicklz.com) says:

QuickLZ is the world's fastest compression library, reaching 308 Mbyte/s per core

how SynLZ compares? smile

#2 Low level and performance » Plaintext visible in SynLZ compressed data? » 2012-01-03 15:54:33

BeginEnd;
Replies: 1

plaintext:

Why Use Fossil? There are plenty of open-source version control systems available on the internet these days. What makes Fossil worthy of attention? Bug Tracking And Wiki - In addition to doing distributed version control like Git and Mercurial, Fossil also supports

zip compressed: (fastest)

PK   a§#@~®Op´   
     whyusefossil3.txteŽ1nAE¯ò¸JŠ 
:ÐÖfÇa-Æ+ÛÚÛÇ+…*…ëý÷†iÅÕßê.õ€ËÄÆ ¼¥r‹ú]¸í\»Œ'›‹6ŒÚ´ÂW~8èIRéVÆĐlc{r¿Ðê{4³ÿñR‹TH
E$0·øêw\ŒÆYÚŸ-`Y°Ã©J'-„PÝò"&·\þÙU™G    PŽœÙÆnBõã
§ê
ïË'þPK    a§#@~®Op´   
                  whyusefossil3.txtPK      ?   ã     

synlz compressed:

  Ç Why Use Fossil? There aAwplenty o    f open-source version control sy  └stems available æ╞the internet !ΓQw  ┴days. What makesE
 wort±╢æFatt1Ʊÿ? B   Çug Tracking And Wiki - In addit±ÿ@   to do≥üdistributedlike Git aüfMe   rcurial,F
also supports 

What I say is, on synlz compressed data a few plain text is visible (especially those few from the start of file?) on the top of compressed data. This is a concern for very small text files for which whole plaintext is visible.

In the case of zip compression (even with fastest mode) the plaintext is never visible no matter how small the source file is.

Is there any way you can adopt this in synlz?

#3 Re: Low level and performance » Synopse vs ICS » 2011-08-27 08:18:28

the Thread Pool facility is available only if I am going to use a HTTP based class (say THttpServer)

What if I need thread pool when I am using only direct socket class? (i.e TCrtSocket)
Can you provide a class/function to achieve this?

#4 Low level and performance » Synopse vs ICS » 2011-08-15 15:36:18

BeginEnd;
Replies: 3

In your code, you mentioned that Synopse is faster than indy or synapse. I think all these three are blocking sockets.

  /// Fast low-level Socket implementation
  // - direct access to the OS (Windows, Linux) network layer
  // - use Open constructor to create a client to be connected to a server
  // - use Bind constructor to initialize a server
  // - use direct access to low level Windows or Linux network layer
  // - use SockIn and SockOut (after CreateSock*) to read or write data
  //  as with standard Delphi text files (see SendEmail implementation)
  // - if app is multi-threaded, use faster SockSend() instead of SockOut^
  //  for direct write access to the socket; but SockIn^ is much faster than
  // SockRecv() thanks to its internal buffer, even on multi-threaded app
  // (at least under Windows, it may be up to 10 times faster)
  // - but you can decide whatever to use none, one or both SockIn/SockOut
  // - our classes are much faster than the Indy or Synapse implementation
  TCrtSocket = class
  public
    /// initialized after Open() with socket
    Sock: TSocket;
   ...
   ...

In ICS (commet from ICS code included below), it is claimed that it is of very high performance due to its asynchronous non-blocking behaviour.

How your socket's performance compare with ICS, mainly when using multiple connections (sockets) to achieve better speed? (like a download manager using 10 connections to download a file at top speed; instead of using a single connection like browsers do)

ICS's multi connection and your's multi thread approach (say 10 threads to DL a file) both achieve more or less same speed?

Comments from ICS code:

{
About multithreading and event-driven:
    TWSocket is a pure asynchronous component. It is non-blocking and
    event-driven. It means that when you request an operation such as connect,
    the component start the operation your requested and give control back
    immediately while performing the operation in the background automatically.
    When the operation is done, an event is triggered (such as
    OnSessionConnected if you called Connect).

    This asynchronous non-blocking behaviour is very high performance but a
    little bit difficult to start with. For example, you can't call Connect and
    immediately call SendStr the line below. If you try, you'll have an
    exception triggered saying you are not connected. Calling connect will start
    connection process but will return long before connection is established.
    Calling SendStr at the next line will not work because the socket is not
    connected yet. To make it works the right way, you have to put your SendStr
    in the OnSessionConnected event.

    The asynchronous operation allows you to do several TCP/IP I/O
    simultaneously. Just use as many component as you need. Each one will
    operate independently of the other without blocking each other ! So you
    basically don't need multi-threading with TWSocket, unless YOUR processing
    is lengthy and blocking.

    If you have to use multithreading, you have two possibilities:
    1) Create your TWSocket from your thread's Execute method
    2) Attach a TWSocket to a given thread using ThreadAttach.
    In both cases, you must set MultiThreaded property to TRUE.
    If you don't use one of those methods, you'll end up with a false
    multithreaded program: all events will be processed by the main tread !
    For both methods to work, you MUST have a message loop withing your thread.
    Delphi create a message loop automatically for the main thread (it's in
    the Forms unit), but does NOT create one in a thread ! For your convenience,
    TWSocket has his own MessageLoop procedure. You can use it from your thread.

    Sample program MtSrv uses first method while ThrdSrv uses second method.
    Sample program TcpSrv is much the same as ThrdSrv but doesn't use any
    thread. You'll see that it is able to server a lot of simultaneous clients
    as well and it is much simpler.


 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
...

#5 Re: Low level and performance » Hash32: Lesser collisions? » 2011-08-14 10:43:22

Oh no! smile  I shouldn't have underestimated your amount of research

#6 Low level and performance » Hash32: Lesser collisions? » 2011-08-12 18:52:26

BeginEnd;
Replies: 7

If subtraction is used alternatively (or in a some more advanced manner),
and that sum too used in the result, then your Hash32() would have
even less collisions? May be also a cumulative sum of s3 would be of use?

s3 := 0;
...
      inc(s1,P^[0]);
    inc(s3, P^[0]);
      inc(s2,s1);
      inc(s1,P^[1]);
    dec(s3,P^[1]);
      inc(s2,s1);
      inc(s1,P^[2]);
    inc(s3,P^[2]);
      inc(s2,s1);
      inc(s1,P^[3]);
    dec(s3,P^[3]);
      inc(s2,s1);
      inc(PtrUInt(P),16);

In http://burtleburtle.net/bob/c/checksum.c , the author (Bob Jenkins)
returns 8 integers as checksum in hash(), so very less chance for collisions?.

Likewise, in your Hash32 (may be you have a better name like BouchezHash()),
if you return multiple integers (say s2,s3) , may be it becomes even less in collisions.
(I don't know why you would mix s2 and s1, losing the fine value of s2 in the result)

May I know the significance of "result := s1 xor (s2 shl 16);"
I think s2 is the one that avoids the collions more (as far as s1, s1 would be same
even if all the bytes were swapped in any combination). So, why not use s2 alone?


JFYI:
Bob Jenkins home page: http://burtleburtle.net/bob/
He also has a hash called lookup3 (http://burtleburtle.net/bob/c/lookup3.c),
64bit: http://burtleburtle.net/bob/c/lookup8.c
which he claims faster than crc32 (http://burtleburtle.net/bob/hash/examhash.html)
and claims 0 collisions.

His ISAAC has been ported in many languages including Pascal/Delphi
(http://home.netsurf.de/wolfgang.ehrhardt/misc_en.html) by
Wolfgang Ehrhardt another Delphi/Speed expert like you smile
(http://home.netsurf.de/wolfgang.ehrhardt/)

#7 Re: Low level and performance » DJB Hash usable? » 2011-01-28 07:18:30

I never imagined  there would be this much stuff to work with to achieve speed.


ab wrote:

About the Hash32() code, s1 and s2 would never have the same value, because there is a inc(s2,s1) every two instructions. Use the step by step debugger, and you'll see how it works.

My mad. smile It became clear that s2 is cumulative sum when I manually gone through the loop just for 2 iterations.


If you try to achieve best speed, take a look at some recent reference documentation.
You've some very good reference material available at http://www.agner.org/optimize

I don't know assembly and have no plans to learn smile
BTW, While you are there..., why I have to worry? smile

#8 Re: Low level and performance » DJB Hash usable? » 2011-01-25 18:28:44

I tried reading data by DWORDs with DJB, but still Hash32 is faster than DJB (approx 2.5 times). The reason: it seems add is faster than bit shift and Hash32 does only add (inc())

// only to test performance; few bytes at end not used in hash
function DjbHash(data: pointer; len: integer): cardinal;
type
  PCardinal = ^cardinal;
function SubHash(P: PCardinal; L: integer): cardinal;
var
  i: cardinal;
begin
  result := 5381;
  if p = nil then
    Exit;
  // few bytes at end discarded, as it wouldn't affect performance test
  for i := 1 to L div sizeof(cardinal) do begin
    result := ((result shl 5)+result) + p^;
    inc(p);
  end;
end;
begin
  result := SubHash(data, len);
end;

My understanding correct?



PS: I observed performance with difference in QueryPerformanceCounter value before and after calling the hash function. I think it is fair.


PS2: How s1 and s2 differ?

s1 := 0;
s2 := 0;
for i := 1 to L shr 4 do begin // 16 bytes (4 DWORD) by loop - aligned read
  inc(s1,P^[0]);
  inc(s2,s1);
  inc(s1,P^[1]);
  inc(s2,s1);
  inc(s1,P^[2]);
  inc(s2,s1);
  inc(s1,P^[3]);
  inc(s2,s1);
  inc(PtrUInt(P),16);
end;

In your Hash32() code, after the end of first(main) loop s1 and s2 differ (it should by purpose I think). But by looking at the code plainly it seems both s1 and s2 would have the same value. Where do I overlook? smile

Thanks.

#9 Re: synopse.info » Please help us! » 2011-01-08 17:47:43

I did 2 votes; I think you are #1 in the list. Really???

BTW, excellent work regarding your Delphi stuff.
Are you any super human?

Board footer

Powered by FluxBB