#1 2016-05-18 13:14:09

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Default User-Agent for outgoing http queries

Currently the User-Agent for HTTP queries is like

Mozilla/4.0 (compatible; MSIE 5.5; Windows; Synopse mORMot 1.18.2634 TdwsWinHTTP)

it should probably be changed to something like

Mozilla/5.0 (Windows; Synopse mORMot 1.18.2634 TdwsWinHTTP)

The reason is that I am using it to fetch RSS feeds on https://www.beginend.net, and I have noticed that at least "Mozilla/4.0" and "MSIE 5.5" are getting filtered and you end up with 403 errors.

Offline

#2 2016-05-18 13:59:13

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,713
Website

Re: Default User-Agent for outgoing http queries

Using which class/function?

Offline

#3 2016-05-19 07:25:32

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

The default User-Agent is defined in function DefaultUserAgent (SynCrtSock)

It affects THttpClient, THttpRequest and all descendants that do not explicitly override it (in my case, it is a TWinHTTP descendant)

Offline

#4 2016-05-19 08:22:17

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,713
Website

Re: Default User-Agent for outgoing http queries

Please try http://synopse.info/fossil/info/53a1209bdb

Thanks for the feedback!

Offline

#5 2016-05-20 06:53:13

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

Works fine, thanks!

Could you consider a couple other change for TWinHttpAPI (and other Http queries) ?

The first should be simple and innocuous enough, it is to have InternalRetrieveAnswer use instance fields for content length & bytes Read rather than local variable. The idea is that when you are downloading something big, and the request is in a thread, it allows to provide some form of download progress feedback.

The second is a bit more structural, it is to allow the downloaded data to be passed to a callback/event, rather than accumulated by InternalRetrieveAnswer. The idea here is also for largish downloads that may not fit in RAM (or are just undesirable in RAM) to be directly written to a file, directly parsed by a linear parser, streamed to something else etc. When such an "OnPartialDataReceived" event is defined, the event would be fully responsible for the data, and Request method OutData parameter would be left empty.

Offline

#6 2016-05-22 08:17:09

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,713
Website

Re: Default User-Agent for outgoing http queries

Please try http://synopse.info/fossil/info/5dc060a372
introducing TWinHttpAPI.OnProgress and OnDownload properties.

Offline

#7 2016-05-24 07:38:52

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

Very nice!

Got a few more issues smile

While digging around the SynCrtSock internals, I noticed THttpRequest.Create takes an aHttps boolean parameter, which is hard-coded (at least for WinHTTP) to WINHTTP_FLAG_SECURE_PROTOCOL_MODERN, which is actually not very modern (it allows SSL3 and all TLS versions).

Ideally, this should be changed to an enumeration, so that you get the option to allow all https protocols (when you do not care about security, so even SSL2), use only moderately secure https protocols (TLS but not SSL3), or only the secure one (TLS 1.2 right now).

Also I think that I have found the source of my occasional WinHTTP Error 12175 (http://synopse.info/forum/viewtopic.php?id=2709), it is the WinHttpSetStatusCallback which goes directly to WinHTTPSecurityErrorCallback, which raises an exception. Apparently the callback mechanism and Delphi exceptions do not play nice: the thread that handles the callback becomes incapable of issuing further WinHTTP queries, and subsequently errors with 12175. The issue is very infrequent, probably related to network issues during handshake or something else that is transient, I have been unable to reproduce it at will.
Rather than raising an exception, the callback should probably just set an error state on the TWinHTTP instance, which could then later be handled in the proper context.

Offline

#8 2016-05-24 08:16:18

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

Also any chance of having TWinHttpProgress be declared as "of object"? smile

Offline

#9 2016-05-24 18:17:56

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,713
Website

Re: Default User-Agent for outgoing http queries

Fixed TWinHttpProgress and TWinHttpDownload by http://synopse.info/fossil/info/ffa7fd4d1f

About WinHTTPSecurityErrorCallback, which kind of notification do you expect?
On which place may we check the TWinHTTP instance error flag?

Offline

#10 2016-05-25 09:20:19

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

I am not sure there is a single place where it would be appropriate... I would expect the exception to be raised in the context of the thread that made use of THttpRequest (rather than the callback's), and it may  need to be checked/raised in the various internals so that they can gracefully abort when the security error is encountered.

However, it is quite possible that without a Delphi exception being raised in the callback, the API functions would naturally fail and cleanup properly, in which case checking the security error and raising the exception somewhere high-level (like at the end of THttpRequest.Request or InternalREST) could be enough.

Offline

#11 2016-05-25 18:20:24

ab
Administrator
From: France
Registered: 2010-06-21
Posts: 14,713
Website

Re: Default User-Agent for outgoing http queries

During which API call may this callback be executed?

Offline

#12 2016-05-30 14:55:48

Eric
Member
Registered: 2012-11-26
Posts: 129
Website

Re: Default User-Agent for outgoing http queries

I have no idea, I have not been able to reproduce it predictably. The only safe assumption would be 'any of them', and maybe even "asynchronously' (ie. not withing an API call).

Offline

Board footer

Powered by FluxBB