mORMot Open Source

Usage purpose is for sharing of secure notes, but without a centralized server controlling access to the notes (and thus having access to everything), so something closer to a password vault or PGP.

> If you change your password, you also change your key pair

In the scheme, the password/seed phrase define everything: the "login" is public and untrustable (and amenable to DB admin abuse), tieing password changes to pubkeys is a feature.

If a password is suspected to be compromised, then the keys should be revoked ASAP, as they could be compromised and they are what matters.
Merely changing a weak password without revoking its keys would be pointless.

> Good password hashing requires a salt, which is not a secret value,

The "login" would be something universally unique and human-friendly, like an email, and the salt would be that unique login + random salt provided per user by the server.
The client app can also use a pepper, stored on the user hardware, and backed up by the user wherever he/she wants.

At setup (or reset) of an "account", you would register the pubkey(s) on the server, which other users could use as a "phonebook" in the first steps of sharing notes.

Any changes in pubkeys for a given login would be tracked / detectable, and serve as a first line of defense against impostors.

For an actual password change, the user would post a notification of the new pubkey(s) with the old key(s), then it's up to interlocutors to accept the change (or not, leaving the possibility to confirm through other channels).

If a basic user forgets a password completely, then recovery would be through the notes he shared, that other users could share back again.
In a corporate setting, automatic sharing to a recovery account could be used (with the recovery seed kept offline or on a secure element)

> The resulting public key would, by construction, be usable in an offline dictionary attack.

Yes, but unlike in a password-protected access, there is neither a clear-text database nor a server-side decryption key to be leaked (or attacked).

This is a tradeoff: the cost of an offline attack would have to be way higher than more classic attacks against the server, backups or plain old social engineering.

The cost comes from the dictionary attack needing to be against each user's keys, with an access to the per-user random salt 'and pepper), and attacking a user's data will not help attacking another user's data.

On the password hashing side, I'm currently looking at a Argon2id and BalloonHash for password hashing (memory hard all the way, logic gets ever faster and more parallel, but memory access keeps lagging).

About one year later, MS AV is still present, with varying degrees of occurrence depending on binary...
I got my executable whitelisted, but it's not enough, apparently the only surefire way is to add an exclusion.

No difference between Win32 et Win64...

I simplified the code to just this, to try variation of the locking and access mode (with no effect)

var buf := z.UnZip(i);
var h := CreateFile(PChar(fileName), GENERIC_READ or GENERIC_WRITE,
                    0, nil, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0);
WriteFile(h, buf[1], Length(buf), nb, nil);
Assert(nb = Length(buf));
CloseHandle(h);   // <-------- this is the slow part, and only it

No digital signature on the unzipper, but running a full scan on the extracted binaries is very fast, barely registering in CPU usage, and AFAICT it happens for 7zip as well.
(but the unzipped binaries are signed)

Apparently some other zip extractors run into the same issue (https://thomasmullaly.codes/2017/11/19/ … hocolatey/)

I have tried renaming the file, no effect.
I also tried writing the first 16 kB header with zeroes, and then the rest of the file with content, this seems enough to prevent the slowdown. But when opening the file again to write the missing 16 kB headers, the MsMpEng slowdown kicks in again.

I also investigated when the slowdown occurs, it's on the FileClose, writing the data itself is fast.

When delaying the close, by not freeing the TFileStream and not doing the FileClose, then it is possible to unzip all the files at high speed.
Tthe solution might be to defer all the FileClose to a background thread or an asynchronous process of some sort...

Someone made a patch for System.Zip to add ZIP64 format support

https://github.com/ccy/delphi-zip

Diff'ing with Delphi's System.Zip will pinpoint the changes in headers.

Only noticed it here when investigating a storage oddity for one sensor which got de-connected (so sending only zeroes), and became was the #1 user of storage space.

Is there any description of how SynLZ beyond looking at the code of SynLZcompress1pas ?

Thanks!

The reason it's called twice is I have some units and projects using only SynWinSock, so they have their own call InitSocketInterface, and when used in projects that also use SynCrtSock, then InitSocketInterface gets called twice.

If it's not meant to be called more than once, I guess the SynSockCount mechanism could be removed, and the initialization performed directly by SynWinSock ?

For a patch, if you want to restrict support to Windows 8.1 and Windows Server 2012 R2, the cleanest option would simply be to remove SockEnhancedApi and all accompanying code (either for all versions, or just for Win64).

If you still want to support the legacy API, I guess the "SockEnhancedApi := False;" could be moved within the "if SynSockCount = 0 then begin" (and same with SockSChannelApi)

function InitSocketInterface(const Stack: TFileName = ''): Boolean;
begin
  result := False;
  EnterCriticalSection(SynSockCS);
  try
    if SynSockCount = 0 then begin
      SockEnhancedApi := False;
      SockSChannelApi := False;
      SockWship6Api := False;
      if Stack = '' then
        LibHandle := LoadLibrary(DLLStackName)
      else 

But since there were explicitly left outside of the "if SynSockCount = 0" I guess there may be side effects ?

Also there seems to be an issue with the Firebird driver, as it allocates a second connection during the course of executing a statement on an existing connection, minimal test case below

program Test;

{$APPTYPE CONSOLE}

uses
  SysUtils, SynDB, SynDBFirebird;

var
   props : TSQLDBFirebirdConnectionProperties;
   conn : TSQLDBConnection;
   stmt : TSQLDBStatement;
begin
   props := TSQLDBFirebirdConnectionProperties.Create('yourserver:yourdatabase.fdb', '', 'user', 'password');
   conn := props.NewConnection;
   stmt := conn.NewStatement;
   stmt.Prepare('select field from sometable', True);  // fails with -502 here
   stmt.ExecutePrepared;
   while stmt.Step do
      Writeln(stmt.ColumnString(0));
end.

When tracing the code, there is a second connection initiated from within inherited Connect... or is there something wrong with my snippet above (besides not releasing stuff)

Well, it was worth a try

I have no idea, I have not been able to reproduce it predictably. The only safe assumption would be 'any of them', and maybe even "asynchronously' (ie. not withing an API call).

Also any chance of having TWinHttpProgress be declared as "of object"?

Works fine, thanks!

Could you consider a couple other change for TWinHttpAPI (and other Http queries) ?

The first should be simple and innocuous enough, it is to have InternalRetrieveAnswer use instance fields for content length & bytes Read rather than local variable. The idea is that when you are downloading something big, and the request is in a thread, it allows to provide some form of download progress feedback.

The second is a bit more structural, it is to allow the downloaded data to be passed to a callback/event, rather than accumulated by InternalRetrieveAnswer. The idea here is also for largish downloads that may not fit in RAM (or are just undesirable in RAM) to be directly written to a file, directly parsed by a linear parser, streamed to something else etc. When such an "OnPartialDataReceived" event is defined, the event would be fully responsible for the data, and Request method OutData parameter would be left empty.

However I am using neither WCF nor IIS

Windows 2012 R2 for the production server.

In other tests (Win 2008 R2) it never occurred, but the test machine is not able to handle the same amount of load or run for days, so it's no apple-for-apple comparison. The backup server did not exhibit it either, but it's mostly inactive in comparison to the production one, so no surprise.

The load is high enough in terms of I/O that a non-virtualized OS is required for production, which makes full-load testing a bit more involved (attempts by different specialist on hyper-v, vmware & kvm all failed miserably, under virtualization performance ended up around 10% of bare-metal, it's a very peculiar workload with relatively little CPU usage, but loads of small disk & network I/O).

TZipRead.UnZip is declared as returning a string, but it seems to be a relic of pre-Unicode Delphi, shouldn't it be a an AnsiString or a byte array? it's in PasZip, which apparently is a relic

In SynZip, TZipRead.Create raises an exception when a an entry has zero bytes, but a zero size file in a zip is valid.

Well done, this is much more readable than PDF on any screen AFAICT, even for the first part.

PDF is good for displaying on dried-dead-tree-paste, but annoying for any other use

Multi-page HTML would be even better, at least at the top levels. It would be more friendly on mobile, and simpler to navigate, also you would get better search engine indexation & lookups. Ideally if you had a permanent page/url structure that would be even better (i.e not something based on fragile section or chapter numbers), as it could then be linked externally, support 3rd party annotations/comments (using any of the tools that already exist for that).

But don't let the above distract from the fact that it's a *huge* step forward IMHO, there was a lot of doc, but being PDF, it was kind of a "ghetto doc" if you take my meaning. This now looks "pro"