Calculate SHA256 hash of large file

Mokofay · 2017-12-27 16:19:03

Hi Folks,

I am new to Synopse.

I would like to calculate the SHA256 hash of some large files (up to 800MB) in order to verify their integrity so converting them into strings is not really an option.

Is there any way to perform this calculation on a stream instead?

Thanks in advance

Mokofay

ab · 2017-12-27 16:28:01

You can simply use the TSHA256.Update on buffers from the file.

Mokofay · 2017-12-27 16:31:36

I am sorry but I do not understand.

Can you please provide some sample code?

ab · 2017-12-27 17:25:25

I've just added the HashFile() function which does what you expect.
See https://synopse.info/fossil/info/906b6f74c4

Mokofay · 2017-12-27 18:31:27

Thank you much!

George · 2021-12-29 09:47:38

Hi!
HashFile method accept string type instead of rawutf8.
I found issue in case when windows encoding is set to United States and file path is rawutf8 (includes utf8 cyrillic characters).

{$mode delphi}

var
  FilePathStr: TFileName;
  Result:        RawUtf8;
begin
  FilePathStr := SrcPath; // 'C:\Program Files (x86)\'#208#161#208#181#209#128#208#178#208#181#209#128'\test.txt'
  // FilePathStr := UTF8ToString(SrcPath);
  // FilePathStr := UTF8ToSys(SrcPath);
  Result := HashFile(FilePathStr, THashAlgo.hfMD5) // empty hash
end;

Compiler options: -FcUTF8

How to pass rawutf8 to HashFile?

Last edited by George (2021-12-29 09:56:19)

ab · 2021-12-29 10:03:31

Did you try Utf8ToWinAnsi(FilePathStr) ?

But anyway, if the system code page is WinAnsi 1252 then it won't be able to open a file name with cyrilic chars, which are not available in this CP1252...

George · 2021-12-29 10:10:48

Same with Utf8ToWinAnsi, cyrillic symbols become "?".
UTF8ToString(SrcPath) works only when system encoding is russian.

George · 2021-12-29 10:43:32

So, the only option is to load file (using ut8 ready methods) and then calculate hash from buffer (because there is no option to pass stream)?

Last edited by George (2021-12-29 10:44:38)

ab · 2021-12-29 11:08:35

I am afraid this is a FPC RTL limitation.
If your system is WinAnsi, then you can't use string=TFileName encoded outside the WinAnsi charset.

We don't have any UnicodeString file name functions in mORMot - we rely on the TFileName type... which is UnicodeString only on Delphi 2009+.

George · 2021-12-29 13:05:08

Code below works, but require to load entire file.

{$mode delphi}

var
  FilePathStr: RawUtf8;           // 'C:\Program Files (x86)\'#208#161#208#181#209#128#208#178#208#181#209#128'\test.txt'
  FileData:    RawByteString;
  Result:      RawUtf8;
begin
  FileData := ReadFileToString(FilePathStr);
  Result := MD5(FileData);
end;

Last edited by George (2021-12-29 14:09:04)

ab · 2021-12-29 14:44:56

You may try after https://github.com/synopse/mORMot2/commit/3a3734ba

HashFile() should work as you expect with a RawUtf8 file name.

George · 2021-12-29 15:56:50

I still use 1.18..

ab · 2021-12-29 16:15:45

Fix is sadly not available in 1.18.
And difficult to port.

George · 2021-12-29 16:27:05

Thanks, i will use suitable temp fix.
Would be cool if mORMot v2 will be fully Unicode aware (for FPC, including file paths and file operations).

ab · 2021-12-29 17:45:55

With my commit above, in mORMot 2 you can use RawUtf8 as TFileName and it will convert any UTF-8 encoded name into UTF-16 proper buffer before calling the Windows API.
Just as with the plain FPC RTL.

So no need of UnicodeString overloaded versions of functions and methods.

mpv · 2021-12-29 21:59:40

There is an experimental utf8 support for file names in Windows 10 (checkbox on the language setting in control panel). If it is checked - Windows should see file, so mORMot1 HashFile will work.

Last edited by mpv (2021-12-29 22:00:38)

mORMot Open Source

#1 2017-12-27 16:19:03

Calculate SHA256 hash of large file

#2 2017-12-27 16:28:01

Re: Calculate SHA256 hash of large file

#3 2017-12-27 16:31:36

Re: Calculate SHA256 hash of large file

#4 2017-12-27 17:25:25

Re: Calculate SHA256 hash of large file

#5 2017-12-27 18:31:27

Re: Calculate SHA256 hash of large file

#6 2021-12-29 09:47:38

Re: Calculate SHA256 hash of large file

#7 2021-12-29 10:03:31

Re: Calculate SHA256 hash of large file

#8 2021-12-29 10:10:48

Re: Calculate SHA256 hash of large file

#9 2021-12-29 10:43:32

Re: Calculate SHA256 hash of large file

#10 2021-12-29 11:08:35

Re: Calculate SHA256 hash of large file

#11 2021-12-29 13:05:08

Re: Calculate SHA256 hash of large file

#12 2021-12-29 14:44:56

Re: Calculate SHA256 hash of large file

#13 2021-12-29 15:56:50

Re: Calculate SHA256 hash of large file

#14 2021-12-29 16:15:45

Re: Calculate SHA256 hash of large file

#15 2021-12-29 16:27:05

Re: Calculate SHA256 hash of large file

#16 2021-12-29 17:45:55

Re: Calculate SHA256 hash of large file

#17 2021-12-29 21:59:40

Re: Calculate SHA256 hash of large file

Board footer