Running Fiddler prevents SignalR (websocket) connection dropouts from happening

[Richard]

I've read Eric Lawrence's great blog post (https://www.telerik.com/blogs/help!-running-fiddler-fixes-my-app-) on the most common classes of issues "fixed" by running Fiddler. I seem to have another example.

I'm investigating an issue for a client where SignalR connections (over websockets) are dropping out regularly every hour. The dropouts are affecting roughly half the machines in their network, and happen after a connection has been up for an hour (i.e. not all machines are dropping out at the same time). The machines are heavily locked down windows 10 running Chrome in kiosk mode and are (in theory at least) identical builds. The dropouts have started happening as the client sites have moved to a new network provider - but only affecting about half of the sites. All sites have (in theory at least) identical network configurations.

To try to learn more about the cause of the dropouts (specifically whether it is a difference in the clients or a difference in the network) I have been running Fiddler on a machine that does experience the dropout. However, the dropout stops happening while Fiddler is running and resumes once Fiddler has stopped.

The dropouts happen whether or not there is application traffic occurring on the SignalR connection. Overnight there may be periods of several hours where the only traffic on the connection is the websocket-level keep-alive (ping/pong) and the SignalR level heartbeat ("type":6 messages). The devices experiencing the issue drop out once per hour, 24 hours per day.

Can you think of any possible way(s) in which Fiddler can be influencing the behaviour that would cause the dropouts to stop?

 

Thanks
Richard

 

[Richard]

We've mad a bit of headway on this. By watching with developer tools we can see that without Fiddler in the mix the browser is failing to establish a websockets connection, so SignalR automatically falls back to Server Sent Events. With Fiddler running for some reason SignalR manages to get its first choice protocol (websockets).

My guess (which I'm about to try to prove by experiment and/or research) is that the hourly connection drops are in some way caused by the use of Server Sent Events.

I'm still intrigued to know what Fiddler can be doing to enable the use of websockets. That may provide some clues as we try to find the reason for the websocket connection failing and fix that.

[Nick Iliev]

Hi Richard,

 

It is hard to say what might be causing your issue and the differences in the behavior when Fiddler is running. You could also use the WebSocket inspector (available only when selecting a WebSocket session) to investigate further. When Fiddler is the intermediary, the connect tunnel is configured to Fiddler. So it might be that your client connection (to the server) is not stable enough - you could refer to this documentation on how to manage connection lifetime: https://docs.microsoft.com/en-us/aspnet/signalr/overview/guide-to-the-api/handling-connection-lifetime-events.

Here is some more information on how WebSocket is working in Fiddler

The HTML5 specification introduces WebSockets, a technology that enables real-time socket communication between a client and server. To create a WebSocket, the client first establishes a HTTP(S) connection to the server. Next, the client and server handshake and agree to use the WebSocket protocol for subsequent traffic on the connection. The ws:// and wss:// URI schemes are used for plaintext and secure WebSockets respectively, even though the initial handshakes take place over HTTP or HTTPS.
To help ensure that WebSocket traffic flows unimpeded through a proxy (like Fiddler), clients will first establish a CONNECT to the proxy, requesting a tunnel to the destination server. If the WebSocket URI uses the wss:// scheme, a HTTPS handshake then occurs. If the unsecure ws:// scheme is used, then the HTTPS handshake is skipped.
The client then uses the newly-established connection to send a HTTP GET request with an Upgrade header proposing a switch to the WebSocket protocol. If the server agrees to change protocols, it sends a HTTP/101 Switching Protocols response. You can easily observe this process using the Web Sessions list and Inspectors.

After the handshake, the client and server may send one another WebSocket messages in any order. Unlike HTTP’s request-response pattern, the server may send WebSocket messages to the client without the client first sending a WebSocket message to the server. HTTP messages are no longer exchanged on a connection which has been upgrad-ed to the WebSocket protocol, and you will not see the WebSocket messages in the Web Sessions list or the request or response Inspectors.
To view WebSocket messages, double-click or hit Enter on a HTTP/101 WebSocket entry in the Web Sessions list to display and activate the WebSockets tab introduced in Fiddler 2.5....

The above is a reference from the Debugging with Fiddler book from Eric Lawrance which covers in dept the functionalities of the classic Fiddler.

 

Regards,
Nick Iliev
Progress Telerik

[Richard]

Thanks for the helpful response, Nick.

I'd actually been reading that Microsoft document on handling connection lifetime events just this morning - and I've been using the Fiddler support for websockets extensively (and have the Fiddler book!), so I'm comfortable with all that - it was just working out how Fiddler was allowing the connection to happen when it didn't without Fiddler that had me stumped.

I actually think I have now got to the bottom of it - but have yet to get back on site to try out my theory.

I've discovered that the client has a proxy PAC file which explicitly checks for a scheme of http: or https: before setting the proxy. So what I'm pretty sure is happening is that all of the SignalR handshake traffic is being routed via the proxy and is working, but the actual websocket connection (which has a scheme of wss:) is not being routed to the proxy and attempts to go direct (which is not allowed by the firewall rules).

When Fiddler is in the mix I think it must be treating the websocket connection as https: rather than wss: for the purposes of the PAC file parsing (which it just does with a standard WinHttp call I believe). So that would result in the actual websocket connection going via the proxy as well. I can see from the properties of all the sessions in a Fiddler recording that the websocket does indeed go via the proxy. Without Fiddler in the mix I can see an unsuccessful websocket connection trying to go direct (by watching with wireshark and also looking in developer tools in Chrome).

So I believe this is another case for you to add to your list of problems that Fiddler solves by magic. :-)

[Richard]

One other point I didn't mention in that last post is that we have also proved that the hourly dropouts only happen when SignalR is going over Server-Sent Events (SSE) rather than websockets. We can reproduce that easily in a test environment simply by blocking websockets (with Fiddler script, of course!).

Each time the SSE connection drops out SignalR initiates a reconnect, starting by trying websockets. This takes a while to time out before it falls back to SSE - so we typically see the dropouts lasting for anywhere from 10 or 20 seconds to a couple of minutes.

[Nick Iliev]

Hi Richard,

 

Great observations - thanks for letting us know about your specific case!

 

Regards,
Nick Iliev
Progress Telerik

[Richard]

Can you confirm if that is actually what is happening with Fiddler with that PAC file - it's hard for me to get access to the site to prove my theory at the moment.

The PAC file is structured like this:

function FindProxyForURL(url, host) {
              if (url.substring(0, 5) == "http:" || url.substring(0, 6) == "https:") {
                     <specific rules>
              }
              return "DIRECT";
}

I'm hoping that we just need to add references to "ws:" and "wss:"to that first if statement.

[Nick Iliev]

Hi Richard,

 

Yes it seems that when the application uses proxy settings for HTTP/HTTPS and is making a WS connection, then first an HTTP CONNECT is made (to the proxy that uses https)

When no proxy is set the WebSocket protocol is directly used

GET ws://server.com HTTP1.1
...
Connection:Upgrade
Upgrade: websocket

From that point, the rules set in the PAC files are returning DIRECT which makes the firewall/security tool to drop the connection. It is probably dropping only after a certain time because by default an SSE is used over the HTTP. When for some reason the SSE fails, the SignalR tries to reconnect via different protocols starting with WS, then SSE, then long polling, etc and this is a likely reason why the connection is failing only after a random amount of time (hitting the PAC rules).

 

Regards,
Nick Iliev
Progress Telerik