Pages: 1

#1 2024-01-03 01:45:57

NigelThomas
Member
Registered: 2023-12-11
Posts: 2

MGet fails silently if Content-Length unknown

One of my servers, proxied by Cloudflare, doesn't return Content-Length when queried (I don't know why, that's a whole other ball of fluff). When I attempt downloading a file from that server with MGet the download fails but there is no error indication, just a zero-length file created locally.

Debug output:

D:\mORMot\src\tools\mget\Win32\Debug\Output>mget https://www.simplysup.com/download/dl/trjsetup.exe /debug
20240103 01420959  ! info  SetThreadName 3ae4=Main
20240103 01420959  !  +    mormot.tools.mget.TMGetProcess(02f52a40).Execute https://www.simplysup.com/download/dl/trjsetup.exe
20240103 01421001  ! trace      mormot.net.client.THttpClientSocket(02edffe0) open(www.simplysup.com:443) sock=0268 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLSv1.2
20240103 01421001  ! trace      mormot.net.client.THttpClientSocket(02edffe0) WGet download/dl/trjsetup.exe: start downloading D:\mORMot\src\tools\mget\Win32\Debug\Output\trjsetup.exe.part
20240103 01421001  ! trace      mormot.net.client.THttpClientSocket(02edffe0) RequestInternal GET www.simplysup.com:443//download/dl/trjsetup.exe flags= retry=0
20240103 01421003  ! trace      mormot.net.client.THttpClientSocket(02edffe0) GetHeader   flags=ptk len=-1 application/x-msdownload
20240103 01421003  ! trace      mormot.net.client.THttpClientSocket(02edffe0) RequestInternal status=200 keepalive=30000 flags=ptk in 43.08ms
20240103 01421003  ! trace      mormot.tools.mget.TMGetProcess(02f52a40) Execute: WGet=D:\mORMot\src\tools\mget\Win32\Debug\Output\trjsetup.exe
20240103 01421003  !  -    00.105.011

WGet (1.21.1) downloads the file OK.

Offline

#2 2024-01-03 10:46:10

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,659
Website

Re: MGet fails silently if Content-Length unknown

The server uses

Transfer-Encoding: chunked

which should be supported.

The regular Get() calls works as expected, but I guess wget requires a length to work as expected...
I will investigate further.

Offline

#3 2024-01-03 11:33:57

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,659
Website

Re: MGet fails silently if Content-Length unknown

Please try
https://github.com/synopse/mORMot2/commit/0a01bc35

There are Internet issues on my location currently, so I am not able to fully validate the fix.
Your feedback is welcome.

Offline

#4 2024-01-03 17:58:44

NigelThomas
Member
Registered: 2023-12-11
Posts: 2

Re: MGet fails silently if Content-Length unknown

Thanks Arnaud, I can confirm the fix works:

D:\mORMot\src\tools\mget\Win32\Debug\Output>mget https://www.simplysup.com/download/dl/trjsetup.exe /debug
20240103 17504852  ! info  SetThreadName 4fc0=Main
20240103 17504852  !  +    mormot.tools.mget.TMGetProcess(03122b30).Execute https://www.simplysup.com/download/dl/trjsetup.exe
20240103 17504858  ! trace      mormot.net.client.THttpClientSocket(030affe0) open(www.simplysup.com:443) sock=0268 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLSv1.2
20240103 17504858  ! trace      mormot.net.client.THttpClientSocket(030affe0) WGet download/dl/trjsetup.exe: start downloading D:\mORMot\src\tools\mget\Win32\Debug\Output\trjsetup.exe.part
20240103 17504858  ! trace      mormot.net.client.THttpClientSocket(030affe0) RequestInternal GET www.simplysup.com:443//download/dl/trjsetup.exe flags= retry=0
20240103 17504861  ! trace      mormot.net.client.THttpClientSocket(030affe0) GetHeader   flags=ptk len=-1 application/x-msdownload
20240103 17504861  ! trace      mormot.core.buffers.TStreamRedirect(0303de90) trjsetup.exe 25.7KB read 547.9KB/s ...
20240103 17504961  ! trace      mormot.core.buffers.TStreamRedirect(0303de90) trjsetup.exe 7.3MB read 7MB/s ...
20240103 17505061  ! trace      mormot.core.buffers.TStreamRedirect(0303de90) trjsetup.exe 13.7MB read 6.7MB/s ...
20240103 17505161  ! trace      mormot.core.buffers.TStreamRedirect(0303de90) trjsetup.exe 20.1MB read 6.6MB/s ...
20240103 17505201  ! trace      mormot.net.client.THttpClientSocket(030affe0) GetBody len=21425456
20240103 17505201  ! trace      mormot.net.client.THttpClientSocket(030affe0) RequestInternal status=200 keepalive=30000 flags=ptk in 3.09s
20240103 17505201  ! trace      mormot.core.buffers.TStreamRedirect(0303de90) trjsetup.exe 20.4MB done in 3.07s ( 6.6MB/s )
20240103 17505201  ! trace      mormot.tools.mget.TMGetProcess(03122b30) Execute: WGet=D:\mORMot\src\tools\mget\Win32\Debug\Output\trjsetup.exe
20240103 17505201  !  -    03.176.051

Offline

#5 2024-01-03 18:05:01

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,659
Website

Re: MGet fails silently if Content-Length unknown

Thanks for testing.

The more tests, the better.
big_smile

Offline

#6 2024-02-07 18:45:38

vs
Member
Registered: 2019-10-21
Posts: 36

Re: MGet fails silently if Content-Length unknown

I'm afraid to be wrong, but maybe it makes sense to add the ability to download files over SMB to MGet.
Or move this feature into a separate utility, something like

smbget - wget-like utility for download files over SMB

https://manpages.ubuntu.com/manpages/tr … get.1.html

Last edited by vs (2024-02-07 18:50:07)

Offline

#7 2024-02-07 20:53:17

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,659
Website

Re: MGet fails silently if Content-Length unknown

A pure SMB client is out-of-scope...
But https://searchcode.com/codesearch/view/64390136/ may help. wink

A network file system access (i.e. using plain FileOpen/FileRead) instead of sockets + HTTP, with our P2P automated fallback, is (easily) feasible.
My guess is that we could a "protocol-plugin-system" to work with schemes other than http/https in mget URIs.
We could had e.g. file://.... and smb:// protocols. wink

Offline

#8 2024-02-07 22:50:12

vs
Member
Registered: 2019-10-21
Posts: 36

Re: MGet fails silently if Content-Length unknown

Great as always, ab!
Thank you!

Generally speaking also plus FTP, FTPS, and SFTP wink 

ftp://[user[:password]@]server[:port]/path/to/remote/resource.mpeg
sftp://[user[:password]@]server[:port]/path/to/remote/resource.mpeg

From
https://ffmpeg.org/ffmpeg-protocols.html#ftp

Offline

#9 2024-02-08 18:26:01

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,659
Website

Re: MGet fails silently if Content-Length unknown

Preliminary 'file://' support:
https://github.com/synopse/mORMot2/commit/489436af

Note that on Windows, it would use proper file URIs so could redirect to a remote folder:
https://learn.microsoft.com/en-us/archi … in-windows
On POSIX, you need to have mounted the file to a given position.

Offline

#10 2024-02-08 18:56:23

vs
Member
Registered: 2019-10-21
Posts: 36

Re: MGet fails silently if Content-Length unknown

Incredible. Thank you!
I'll try to do some tests.

Offline

Pages: 1

Board footer

Powered by FluxBB