Streams over 2GB

13 posts, 0 answers
  1. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 20 Aug 2014 Link to this post

    The app that I am debugging is doing loads of PUT requests over (decrypted) HTTPS and sometimes is PUTting a file that is 4.45GB. Part way through the large PUT request Fiddler shows me this message:

    ---------------------------
    Uncaught Exception in Session #36250
    ---------------------------
    Sorry, the .NET Framework (and Fiddler) cannot handle streams larger than 2 Gigabytes.

    Type: System.Exception
    Source: Fiddler
       at Fiddler.PipeReadBuffer.Write(Byte[] buffer, Int32 offset, Int32 count)
       at Fiddler.ClientChatter.ReadRequest()
       at Fiddler.Session._executeObtainRequest()
       at Fiddler.Session.InnerExecute()
       at Fiddler.Session.Execute(Object objThreadState)

    Fiddler v4.4.9.2 (x64 AMD64) [.NET 4.0.30319.18444 on Microsoft Windows NT 6.1.7601 Service Pack 1] 



    Is there a way of working around this 2GB restriction?

  2. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 20 Aug 2014 Link to this post

    Hi, Graeme--

    Unfortunately, no. Working around this would require Fiddler be able to stream requests (such that it wouldn't have to buffer the whole body). I'm curious about what you're using Fiddler for specifically here... it's possible that I could hack something together for your scenario.


    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  3. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 22 Aug 2014 in reply to Eric Lawrence Link to this post

    Hi Eric,

    One of our applications is for video editing companies and lets people back up their video projects. One option is to backup to OpenStack Swift storage.

    In case you aren't familiar with Swift, it uses various HTTP methods (POST/PUT/GET/ETC) over SSL to upload/download files. We're using the OpenStack .Net sdk, so we're not directly making HTTP requests in our own code.

    We're getting some strange errors in our app where it appears to get stuck at random points, often after transferring hundreds of GB in both directions so I wanted to log all the decrypted HTTP PUT/GET requests to see if there is something going wrong in the connection (e.g. never getting a reply to a request). To avoid the logs being hundreds of GB, I am not logging the body - only the headers.

    The broadcast quality video can be very large; 220Mbps is not a particularly unusual bit rate, so a relatively short (1 minute, 15 second clip) can be over 2GB.

    It would be great to be able to stream requests so that I can log the headers of large file PUT/GETs, but if I'm the only one asking for this feature, don't spend too much time on it ;-)




  4. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 25 Aug 2014 Link to this post

    Hi, Graeme--

    Does your client send a proper Content-Length request header, or does it use the Transfer-Encoding header instead?

    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  5. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 26 Aug 2014 in reply to Eric Lawrence Link to this post

    The headers look like this:

    PUT <ADDRESS> HTTP/1.1
    User-Agent: openstack.net/1.1.2.1
    X-Auth-Token: <TOKEN>
    Host: <HOSTNAME>
    Transfer-Encoding: chunked
    Expect: 100-continue
  6. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 26 Aug 2014 Link to this post

    Figures. :-)

    I enabled streaming of requests with a Content-Length header yesterday. I was hoping to be a slacker and not do the work for streaming requests that use Transfer-Encoding: chunked because that's MUCH more complicated.

    I'll see what I can do.

    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  7. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 26 Aug 2014 in reply to Eric Lawrence Link to this post

    Thanks Eric. I must say that this is great support for a product that I'm not even paying for :-D
  8. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 27 Aug 2014 Link to this post

    Hello, Graeme--

    The "price" for Fiddler is feedback, so you've paid more than most.

    Please grab either of the latest Alphas:
    http://ericlawrence.com/dl/fiddler2alphaSetup.exe
    http://ericlawrence.com/dl/fiddler4alphaSetup.exe

    Click Rules > Customize Rules.

    Scroll to the OnPeekAtRequestHeaders function (you may need to uncomment it).

    Inside that function, add code similar to the following:

    if (oSession.host.Contains("whatever.com") &&
        oSession.oRequest.headers.ExistsAndContains("Transfer-Encoding", "chunked"))
    {
       oSession["ui-backcolor"] = "orange";
       oSession["log-drop-request-body"] = "streamingThisRequest";
       oSession.oRequest.BufferRequest = false;
    }

    This will cause the request in question to be streamed and forgotten as it is written.

    Please do let me know how this works out. There are probably lots of interesting little corner cases that I missed. 

    Thanks!
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  9. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 29 Aug 2014 in reply to Eric Lawrence Link to this post

    Hi Eric,

    Thanks, that's great!

    The machine I was using is busy doing other tests at the moment, but when it becomes free (probably next week) I'll set this up and let you know how I get on.


  10. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 29 Aug 2014 Link to this post

    Great, thanks!


    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  11. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 10 Sep 2014 in reply to Eric Lawrence Link to this post

    I'm doing a test of it now (exactly the same operation that caused the issue before).

    I should have results by this time tomorrow, or Thursday at the latest.
  12. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 11 Sep 2014 in reply to Graeme Link to this post

    That worked great. Thanks Eric!
  13. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 11 Sep 2014 Link to this post

    Awesome! Thanks for checking and for the nudge to finally get this done.

    Please feel free to report any problems you find later using Help > Send Feedback.


    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
Back to Top