SSL negotiation doesn't initiate for some traffic

[Sebastien]

Hi,

First, I have to say this is a great tool and I'm very impressed.  I managed to get everything set up to intercept SSL traffic from a Symbian device, through my local hotspot, and redirected to another server (still over HTTPS).  FYI, this is to reroute traffic to a test environment using a device that, unfortunately, has only production URLs burned in (don't get me started on that!).

On the device, I have three options to test my setup:  Opera Mini, the native browser, and the application I wish to test.  If I access my target URL via OM, everything works fine.  If I access it via the native browser or the application, the connection is closed immediately after the SYN, SYN/ACK, ACK exchange.  At first I thought that perhaps the device had app-level configurations for the certificates, but they all have the same settings.  Besides, the certificate hasn't yet been exchanged between the two endpoints.

Looking at Fiddler, the only difference I can see is that OM makes an HTTP/1.0 request, and the other two make an HTTP/1.1 request.  Looking at a tcpdump, one difference I noticed was in the TCP options, where in the successful request the TCP timestamps (tsval and tsecr) are specified and they are absent in the request that fails.  I wonder if the client might try to use those when performing the SSL handshake and, if they are absent, simply aborts?

Unfortunately, I can't see exactly what is sent from the device to Fiddler because I'm on Windows 7 using a local hotspot with Microsoft's virtual wifi miniport adapter.  Unless I'm mistaken, Wireshark won't capture on that interface.

Any ideas would be most welcome.

Thanks,

Sebastien

[Eric Lawrence]

Hello, Sebastien--

I don't know why Wireshark wouldn't capture there, but you could probably use Microsoft Message Analyzer or Microsoft NetMon to capture the data. You mention that you're watching the SYN/ACK exchanges on the device-- with TCPDump? So why can't you use this to watch what's being sent to Fiddler?

When you say "the other two make a HTTP/1.1 request"-- is this for the CONNECT tunnel, or for something else?

It would be unbelievably fragile to attempt to require TCP timestamps to establish a connection; I've never heard of any device with such a behavior.

Regards,
Eric Lawrence
Telerik

 

 

[Sebastien]

Hi Eric,

In Wireshark the virtual wifi adapter doesn't show up as an available interface, so traffic coming from the device to Fiddler can't be captured.  The only traffic I've been able to capture is between Fiddler and the remote servers to which I'm connecting.  That's where I'm seeing the SYN/ACK exchange with wshark.  There's also no way for me to monitor the traffic on the device itself.

This is what I see in Fiddler when the tunnel is established successfully:

CONNECT www.google.com:443 HTTP/1.0
User-Agent: Opera/9.80 (S60; SymbOS; Opera Mobi/SYB-1106291583; U; en-US) Presto/2.8.149 Version/11.10
Host: www.google.com:443
Connection: Keep-Alive

A SSLv3-compatible ClientHello handshake was found. Fiddler extracted the parameters below.

Version: 3.1 (TLS/1.0)
Random: 53 E1 44 53 99 38 E2 F3 3A 1E 75 51 61 41 21 BC D2 C4 72 64 3C 7F B9 2B 3D AE B0 25 4E B9 46 C6
SessionID: empty
Extensions:
    none
Ciphers:
    [0004]  SSL_RSA_WITH_RC4_128_MD5
    [0005]  SSL_RSA_WITH_RC4_128_SHA
    [000A]  SSL_RSA_WITH_3DES_EDE_SHA
    [002F]  TLS_RSA_AES_128_SHA
    [0035]  TLS_RSA_AES_256_SHA

Compression:
    [00]    NO_COMPRESSION

And when it fails:

CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Symbian/3; Series60/5.2 NokiaN8-00/022.011; Profile/MIDP-2.1 Configuration/CLDC-1.1 ) AppleWebKit/533.4 (KHTML, like Gecko) NokiaBrowser/7.3.1.35 Mobile Safari/533.4 3gpp-gba
x-wap-profile: "http://nds1.nds.nokia.com/uaprof/NN8-00r200-3G.xml"

After the client received notice of the established CONNECT, it failed to send any data.

If I configure the native browser to bypass Fiddler (removing the proxy config on the device), then the connection is established successfully.

This seems to point to a finicky client on the device...

I'm going to look into the tools you suggest to see if they can detect the local interface.

Thanks again,

Sebastien

[Sebastien]

Oh man... MEGA face-palm.  :(

I restarted my local hotspot AFTER having started wireshark so, OF COURSE, it didn't show the local interface!  Really red-faced right now.

I'm going to troubleshoot some more and post my next findings.

Thanks,

Sebastien

[Eric Lawrence]

Hi, Sebastien--

The way HTTPS interception works is the client establishes a TCP/IP connection to Fiddler and says HTTP CONNECT <server>. Fiddler dutifully attempts to connect to the target host:port and if it succeeds, it returns a HTTP/200 to the client. Fiddler then sniffs the client's next bytes and if they're a HTTPS handshake, it will perform the server's half of the HTTPS handshake with that client. If, however, the client does NOT send any bytes (e.g. a HTTPS handshake) then you'll see the "After the client received notice of the established CONNECT, it failed to send any data." message.

Regards,
Eric Lawrence
Telerik

 

 

[Sebastien]

Hi Eric,

Thanks for the explanation.  That's precisely what I've observed yesterday night after sniffing all traffic.  It seems that somehow the native browser doesn't move on with the SSL handshake following the CONNECT response, which is really bizarre. 

Actually, I've been able to reproduce the problem using a two or three year old Gmail application.  The symptoms are the same.  The application receives the HTTP/200 and closes the connection.

I've captured tcpdumps for all three situations (native browser and Gmail app that fail, and Opera which succeeds).  The only differences I have noted are:
1) the two that fail use HTTP/1.1, and the one that succeeds uses HTTP/1.0.  I don't think this is relevant.
2) the two that fail do not echo exactly the CONNECT request's Tsecr value in the response's Tsval, whereas the transaction that succeeds does.

I've read that this can sometimes be a problem but, admittedly, the scenario reported there is when the response's echoed value (in tsval) is older (lower) than the request's tsecr.  In my case, for the failing scenarios, the echoed value is different but is younger (higher) than the request's tsecr.

Nonetheless, I wonder if this could be the problem.  Any idea why there's a variation in the timestamp?

Thanks again for the help,

Sebastien

 

[Sebastien]

I should add that in the linked StackExchange post, it's not entirely clear if the scenario is precisely like mine in terms of timestamp discrepancies, or if it's the opposite.  It depends on one's interpretation of "a few seconds behind". :)

Cheers,

Sebastien

[Sebastien]

Hi Eric,

First, a correction.  In my earlier post where I described my observations about the timestamps in the request/response, I interchanged the terms.  Instead of saying the response's Tsval didn't match the request's Tsecr, I should've said that the response's Tsecr didn't match the request's Tsval.  My observations were good, but I erred on the description. Sorry about that.  I wish there was a way to edit the post.

Having cleared that up, I have another update. 

I wanted to see if the native browser behaved the same way with a different proxy.  I installed Privoxy and used it as the proxy for my device instead of Fiddler.  This time, the native browser continued with the SSL handshake after having received the HTTP/200 for the CONNECT.

I took a look at the timestamps in this case and this time the HTTP/200 response correctly echoed in Tsecr the Tsval that had been provided in the CONNECT request.  It really looks like the device is sensitive to the values.

Coming back to my earlier test where the request from Opera worked, but the one from the native browser failed.  Is it possible that somehow Fiddler is returning the correct value in Tsecr for an HTTP/1.0 request, but not for an HTTP/1.1 request?  It seems preposterous, and yet the only variations are at the HTTP header level as you can see in one of my earlier posts.

I'm both mystified and stuck. :)

Thanks again for your insight and help.  I have a number of tcpdumps if you would like me to check anything in particular.

Cheers,

Sebastien

[Eric Lawrence]

Hello, Sebastien--

Fiddler itself has no awareness or participation in TCP-timestamps; this functionality, if present, would be present in the operating system's socket implementation itself. The OS socket implementation has no facility to behave differently based on the higher level protocol (HTTP1.0 vs 1.1) used on that connection.

If you create the following chain: Device -> Privoxy -> Fiddler -> Website, does everything function?

Regards,
Eric Lawrence
Telerik

 

 

[Sebastien]

Hi Erik,

Sorry for the long silence; I got massively side-tracked.  In the end I found the problem and a solution.  It turns out that when using Fiddler as a proxy, on the CONNECT response a "Connection: close" header is added by Fiddler.  This is regardless of the HTTP version used.  Here are two examples:

CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Symbian/3; Series60/5.2 NokiaN8-00/022.011; Profile/MIDP-2.1 Configuration/CLDC-1.1 ) AppleWebKit/533.4 (KHTML, like Gecko) NokiaBrowser/7.3.1.35 Mobile Safari/533.4 3gpp-gba
x-wap-profile: "http://nds1.nds.nokia.com/uaprof/NN8-00r200-3G.xml"

HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 16:37:05.008
Connection: close

and the second request (which succeeds):

CONNECT www.google.com:443 HTTP/1.0
User-Agent: Opera/9.80 (S60; SymbOS; Opera Mobi/SYB-1106291583; U; en-US) Presto/2.8.149 Version/11.10
Host: www.google.com:443
Proxy-Connection: Keep-Alive

HTTP/1.0 200 Connection Established
FiddlerGateway: Direct
StartTime: 16:38:00.872
Connection: close

My suspicion is that either the two applications on the device are using different HTTP client libraries and one is more stringent than the other, or the client is the same and it ignores the "Connection: close" instruction when negotiating an HTTP/1.0 connection.

The workaround was simple.  I modified the rules to remove the Connection header:

static function OnBeforeResponse(oSession: Session) {
    if (m_Hide304s && oSession.responseCode == 304) {
        oSession["ui-hide"] = "true";
    }
    if (oSession.HTTPMethodIs("CONNECT") && oSession.responseCode == 200)
    {
        if (oSession.oResponse.headers.ExistsAndEquals("Connection", "close"))
        {
            oSession.oResponse.headers.Remove("Connection");
        }
    }
}

But I wonder why Fiddler adds a "Connection: close" header when the next step should be to continue with SSL negotiation?

Thanks,

Sebastien

[Eric Lawrence]

Hello, Sebastien--

Thanks for sharing your finding!

Connection: close means "This response is complete when the connection has closed." 

In HTTP, properly functioning CONNECT tunnels behave in a "Connection: close" manner (see http://tools.ietf.org/html/rfc7231#section-4.3.6 and http://tools.ietf.org/html/rfc7230#section-6.1).

It's certainly possible that a library would have a bug whereby upon seeing "Connection: close" in a response it would itself close the connection instead of properly waiting for the server to close the connection. 

Regards,
Eric Lawrence
Telerik

 

 

[Michael]

Hello, Sebastien

I met the same issue, when try capture ssl traffice between app on mobile and it's web server, 

I use your solution which help solve the problem, just reply here to thank you! :)

and thanks Eric for your great tools

[Kender]

Awesome work Sebastien!

This is a problem on 2 of my samsung android devices and all the android emulators I tried.

Your workaround works like a charm for all of them.