Streams over 2GB

[Graeme]

The app that I am debugging is doing loads of PUT requests over (decrypted) HTTPS and sometimes is PUTting a file that is 4.45GB. Part way through the large PUT request Fiddler shows me this message:

---------------------------
Uncaught Exception in Session #36250
---------------------------
Sorry, the .NET Framework (and Fiddler) cannot handle streams larger than 2 Gigabytes.

Type: System.Exception
Source: Fiddler
   at Fiddler.PipeReadBuffer.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Fiddler.ClientChatter.ReadRequest()
   at Fiddler.Session._executeObtainRequest()
   at Fiddler.Session.InnerExecute()
   at Fiddler.Session.Execute(Object objThreadState)

Fiddler v4.4.9.2 (x64 AMD64) [.NET 4.0.30319.18444 on Microsoft Windows NT 6.1.7601 Service Pack 1] 



Is there a way of working around this 2GB restriction?

[Eric Lawrence]

Hi, Graeme--

Unfortunately, no. Working around this would require Fiddler be able to stream requests (such that it wouldn't have to buffer the whole body). I'm curious about what you're using Fiddler for specifically here... it's possible that I could hack something together for your scenario.


Regards,
Eric Lawrence
Telerik

 

 

[Graeme]

Hi Eric,

One of our applications is for video editing companies and lets people back up their video projects. One option is to backup to OpenStack Swift storage.

In case you aren't familiar with Swift, it uses various HTTP methods (POST/PUT/GET/ETC) over SSL to upload/download files. We're using the OpenStack .Net sdk, so we're not directly making HTTP requests in our own code.

We're getting some strange errors in our app where it appears to get stuck at random points, often after transferring hundreds of GB in both directions so I wanted to log all the decrypted HTTP PUT/GET requests to see if there is something going wrong in the connection (e.g. never getting a reply to a request). To avoid the logs being hundreds of GB, I am not logging the body - only the headers.

The broadcast quality video can be very large; 220Mbps is not a particularly unusual bit rate, so a relatively short (1 minute, 15 second clip) can be over 2GB.

It would be great to be able to stream requests so that I can log the headers of large file PUT/GETs, but if I'm the only one asking for this feature, don't spend too much time on it ;-)

[Eric Lawrence]

Hi, Graeme--

Does your client send a proper Content-Length request header, or does it use the Transfer-Encoding header instead?

Regards,
Eric Lawrence
Telerik

 

 

[Graeme]

The headers look like this:

PUT <ADDRESS> HTTP/1.1
User-Agent: openstack.net/1.1.2.1
X-Auth-Token: <TOKEN>
Host: <HOSTNAME>
Transfer-Encoding: chunked
Expect: 100-continue

[Eric Lawrence]

Figures. :-)

I enabled streaming of requests with a Content-Length header yesterday. I was hoping to be a slacker and not do the work for streaming requests that use Transfer-Encoding: chunked because that's MUCH more complicated.

I'll see what I can do.

Regards,
Eric Lawrence
Telerik

 

 

[Graeme]

Thanks Eric. I must say that this is great support for a product that I'm not even paying for :-D

[Eric Lawrence]

Hello, Graeme--

The "price" for Fiddler is feedback, so you've paid more than most.

Please grab either of the latest Alphas:
http://ericlawrence.com/dl/fiddler2alphaSetup.exe
http://ericlawrence.com/dl/fiddler4alphaSetup.exe

Click Rules > Customize Rules.

Scroll to the OnPeekAtRequestHeaders function (you may need to uncomment it).

Inside that function, add code similar to the following:

if (oSession.host.Contains("whatever.com") &&
    oSession.oRequest.headers.ExistsAndContains("Transfer-Encoding", "chunked"))
{
   oSession["ui-backcolor"] = "orange";
   oSession["log-drop-request-body"] = "streamingThisRequest";
   oSession.oRequest.BufferRequest = false;
}

This will cause the request in question to be streamed and forgotten as it is written.

Please do let me know how this works out. There are probably lots of interesting little corner cases that I missed. 

Thanks!
Eric Lawrence
Telerik

 

 

[Graeme]

Hi Eric,

Thanks, that's great!

The machine I was using is busy doing other tests at the moment, but when it becomes free (probably next week) I'll set this up and let you know how I get on.

[Eric Lawrence]

Great, thanks!


Regards,
Eric Lawrence
Telerik

 

 

[Graeme]

I'm doing a test of it now (exactly the same operation that caused the issue before).

I should have results by this time tomorrow, or Thursday at the latest.

[Graeme]

That worked great. Thanks Eric!

[Eric Lawrence]

Awesome! Thanks for checking and for the nudge to finally get this done.

Please feel free to report any problems you find later using Help > Send Feedback.


Regards,
Eric Lawrence
Telerik