You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 2016-05-18 13:14:09
Default User-Agent for outgoing http queries
Currently the User-Agent for HTTP queries is like
Mozilla/4.0 (compatible; MSIE 5.5; Windows; Synopse mORMot 1.18.2634 TdwsWinHTTP)it should probably be changed to something like
Mozilla/5.0 (Windows; Synopse mORMot 1.18.2634 TdwsWinHTTP)The reason is that I am using it to fetch RSS feeds on https://www.beginend.net, and I have noticed that at least "Mozilla/4.0" and "MSIE 5.5" are getting filtered and you end up with 403 errors.
Offline
#3 2016-05-19 07:25:32
Re: Default User-Agent for outgoing http queries
The default User-Agent is defined in function DefaultUserAgent (SynCrtSock)
It affects THttpClient, THttpRequest and all descendants that do not explicitly override it (in my case, it is a TWinHTTP descendant)
Offline
#4 2016-05-19 08:22:17
Re: Default User-Agent for outgoing http queries
Please try http://synopse.info/fossil/info/53a1209bdb
Thanks for the feedback!
Offline
#5 2016-05-20 06:53:13
Re: Default User-Agent for outgoing http queries
Works fine, thanks!
Could you consider a couple other change for TWinHttpAPI (and other Http queries) ?
The first should be simple and innocuous enough, it is to have InternalRetrieveAnswer use instance fields for content length & bytes Read rather than local variable. The idea is that when you are downloading something big, and the request is in a thread, it allows to provide some form of download progress feedback.
The second is a bit more structural, it is to allow the downloaded data to be passed to a callback/event, rather than accumulated by InternalRetrieveAnswer. The idea here is also for largish downloads that may not fit in RAM (or are just undesirable in RAM) to be directly written to a file, directly parsed by a linear parser, streamed to something else etc. When such an "OnPartialDataReceived" event is defined, the event would be fully responsible for the data, and Request method OutData parameter would be left empty.
Offline
#6 2016-05-22 08:17:09
Re: Default User-Agent for outgoing http queries
Please try http://synopse.info/fossil/info/5dc060a372
introducing TWinHttpAPI.OnProgress and OnDownload properties.
Offline
#7 2016-05-24 07:38:52
Re: Default User-Agent for outgoing http queries
Very nice!
Got a few more issues 
While digging around the SynCrtSock internals, I noticed THttpRequest.Create takes an aHttps boolean parameter, which is hard-coded (at least for WinHTTP) to WINHTTP_FLAG_SECURE_PROTOCOL_MODERN, which is actually not very modern (it allows SSL3 and all TLS versions).
Ideally, this should be changed to an enumeration, so that you get the option to allow all https protocols (when you do not care about security, so even SSL2), use only moderately secure https protocols (TLS but not SSL3), or only the secure one (TLS 1.2 right now).
Also I think that I have found the source of my occasional WinHTTP Error 12175 (http://synopse.info/forum/viewtopic.php?id=2709), it is the WinHttpSetStatusCallback which goes directly to WinHTTPSecurityErrorCallback, which raises an exception. Apparently the callback mechanism and Delphi exceptions do not play nice: the thread that handles the callback becomes incapable of issuing further WinHTTP queries, and subsequently errors with 12175. The issue is very infrequent, probably related to network issues during handshake or something else that is transient, I have been unable to reproduce it at will.
Rather than raising an exception, the callback should probably just set an error state on the TWinHTTP instance, which could then later be handled in the proper context.
Offline
#9 2016-05-24 18:17:56
Re: Default User-Agent for outgoing http queries
Fixed TWinHttpProgress and TWinHttpDownload by http://synopse.info/fossil/info/ffa7fd4d1f
About WinHTTPSecurityErrorCallback, which kind of notification do you expect?
On which place may we check the TWinHTTP instance error flag?
Offline
#10 2016-05-25 09:20:19
Re: Default User-Agent for outgoing http queries
I am not sure there is a single place where it would be appropriate... I would expect the exception to be raised in the context of the thread that made use of THttpRequest (rather than the callback's), and it may need to be checked/raised in the various internals so that they can gracefully abort when the security error is encountered.
However, it is quite possible that without a Delphi exception being raised in the callback, the API functions would naturally fail and cleanup properly, in which case checking the security error and raising the exception somewhere high-level (like at the end of THttpRequest.Request or InternalREST) could be enough.
Offline
Pages: 1