This is a migrated thread and some comments may be shown as answers.

Streams over 2GB

12 Answers 55 Views
Fiddler Classic
This is a migrated thread and some comments may be shown as answers.
Graeme
Top achievements
Rank 1
Graeme asked on 20 Aug 2014, 08:25 AM
The app that I am debugging is doing loads of PUT requests over (decrypted) HTTPS and sometimes is PUTting a file that is 4.45GB. Part way through the large PUT request Fiddler shows me this message:

---------------------------
Uncaught Exception in Session #36250
---------------------------
Sorry, the .NET Framework (and Fiddler) cannot handle streams larger than 2 Gigabytes.

Type: System.Exception
Source: Fiddler
   at Fiddler.PipeReadBuffer.Write(Byte[] buffer, Int32 offset, Int32 count)
   at Fiddler.ClientChatter.ReadRequest()
   at Fiddler.Session._executeObtainRequest()
   at Fiddler.Session.InnerExecute()
   at Fiddler.Session.Execute(Object objThreadState)

Fiddler v4.4.9.2 (x64 AMD64) [.NET 4.0.30319.18444 on Microsoft Windows NT 6.1.7601 Service Pack 1] 



Is there a way of working around this 2GB restriction?

12 Answers, 1 is accepted

Sort by
0
Eric Lawrence
Telerik team
answered on 20 Aug 2014, 06:19 PM
Hi, Graeme--

Unfortunately, no. Working around this would require Fiddler be able to stream requests (such that it wouldn't have to buffer the whole body). I'm curious about what you're using Fiddler for specifically here... it's possible that I could hack something together for your scenario.


Regards,
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
0
Graeme
Top achievements
Rank 1
answered on 22 Aug 2014, 08:58 AM
Hi Eric,

One of our applications is for video editing companies and lets people back up their video projects. One option is to backup to OpenStack Swift storage.

In case you aren't familiar with Swift, it uses various HTTP methods (POST/PUT/GET/ETC) over SSL to upload/download files. We're using the OpenStack .Net sdk, so we're not directly making HTTP requests in our own code.

We're getting some strange errors in our app where it appears to get stuck at random points, often after transferring hundreds of GB in both directions so I wanted to log all the decrypted HTTP PUT/GET requests to see if there is something going wrong in the connection (e.g. never getting a reply to a request). To avoid the logs being hundreds of GB, I am not logging the body - only the headers.

The broadcast quality video can be very large; 220Mbps is not a particularly unusual bit rate, so a relatively short (1 minute, 15 second clip) can be over 2GB.

It would be great to be able to stream requests so that I can log the headers of large file PUT/GETs, but if I'm the only one asking for this feature, don't spend too much time on it ;-)




0
Eric Lawrence
Telerik team
answered on 25 Aug 2014, 09:04 PM
Hi, Graeme--

Does your client send a proper Content-Length request header, or does it use the Transfer-Encoding header instead?

Regards,
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
0
Graeme
Top achievements
Rank 1
answered on 26 Aug 2014, 09:08 AM
The headers look like this:

PUT <ADDRESS> HTTP/1.1
User-Agent: openstack.net/1.1.2.1
X-Auth-Token: <TOKEN>
Host: <HOSTNAME>
Transfer-Encoding: chunked
Expect: 100-continue
0
Eric Lawrence
Telerik team
answered on 26 Aug 2014, 03:28 PM
Figures. :-)

I enabled streaming of requests with a Content-Length header yesterday. I was hoping to be a slacker and not do the work for streaming requests that use Transfer-Encoding: chunked because that's MUCH more complicated.

I'll see what I can do.

Regards,
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
0
Graeme
Top achievements
Rank 1
answered on 26 Aug 2014, 03:40 PM
Thanks Eric. I must say that this is great support for a product that I'm not even paying for :-D
0
Eric Lawrence
Telerik team
answered on 27 Aug 2014, 07:32 PM
Hello, Graeme--

The "price" for Fiddler is feedback, so you've paid more than most.

Please grab either of the latest Alphas:
http://ericlawrence.com/dl/fiddler2alphaSetup.exe
http://ericlawrence.com/dl/fiddler4alphaSetup.exe

Click Rules > Customize Rules.

Scroll to the OnPeekAtRequestHeaders function (you may need to uncomment it).

Inside that function, add code similar to the following:

if (oSession.host.Contains("whatever.com") &&
    oSession.oRequest.headers.ExistsAndContains("Transfer-Encoding", "chunked"))
{
   oSession["ui-backcolor"] = "orange";
   oSession["log-drop-request-body"] = "streamingThisRequest";
   oSession.oRequest.BufferRequest = false;
}

This will cause the request in question to be streamed and forgotten as it is written.

Please do let me know how this works out. There are probably lots of interesting little corner cases that I missed. 

Thanks!
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
0
Graeme
Top achievements
Rank 1
answered on 29 Aug 2014, 12:34 PM
Hi Eric,

Thanks, that's great!

The machine I was using is busy doing other tests at the moment, but when it becomes free (probably next week) I'll set this up and let you know how I get on.


0
Eric Lawrence
Telerik team
answered on 29 Aug 2014, 02:42 PM
Great, thanks!


Regards,
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
0
Graeme
Top achievements
Rank 1
answered on 10 Sep 2014, 01:36 PM
I'm doing a test of it now (exactly the same operation that caused the issue before).

I should have results by this time tomorrow, or Thursday at the latest.
0
Graeme
Top achievements
Rank 1
answered on 11 Sep 2014, 08:05 AM
That worked great. Thanks Eric!
0
Eric Lawrence
Telerik team
answered on 11 Sep 2014, 05:22 PM
Awesome! Thanks for checking and for the nudge to finally get this done.

Please feel free to report any problems you find later using Help > Send Feedback.


Regards,
Eric Lawrence
Telerik
 

Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

 
Tags
Fiddler Classic
Asked by
Graeme
Top achievements
Rank 1
Answers by
Eric Lawrence
Telerik team
Graeme
Top achievements
Rank 1
Share this question
or