Is it possible to not buffer requests

7 posts, 1 answers
  1. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 13 Aug 2014 Link to this post

    I have an app that uses HTTPS and I can successfully use fiddler to view and modify the decrypted traffic. The app often does post requests of very large files (hundreds of MB or more) and fiddler seems to use loads of memory and become slow to respond when these large post requests are happening.

    I'm hoping there is some way to not log the request/response body and just log the headers.

    I have tried using stream mode which seems to make no difference; the body of the large request is still logged

    I also added these two lines:
            oSession["log-drop-request-body"] = "yes";
            oSession["log-drop-response-body"] = "yes";
     to the OnBeforeRequest function. This results in the body not being logged after the request/response has completed, but while it is still going, the body is logged which is what causes the problem if the request is very large.

    Is there anything else that I can try to make it behave better with these large posts?

  2. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 14 Aug 2014 Link to this post

    Hi, Graeme--

    Fiddler inherently buffers request bodies before sending them; if you set log-drop-request-body, it will drop the body as soon as it is transmitted.

    In contrast, if you have Response Streaming enabled and log-drop-response-body set, Fiddler will drop each chunk of the response body as it is read from the server and passed to the client.

    Having said that, I'd like to learn more about your claim that Fiddler "become slow to respond" in this situation. Can you be precise about what specifically is slow? I've not experienced any slowness in this area, and I'm not aware of any technical reason why you'd see a performance difference here. 

    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  3. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 15 Aug 2014 in reply to Eric Lawrence Link to this post

    Hi Eric,

    The slow to respond situation occurs when I click on a session that has a large amount of data in the body and I have the inspector tab in view. I guess it's slow to display the body because it's so large (many megabytes).

    When I'm uploading a 155MB file and and dropping the bodies from the log, the memory usage grows up to about 800MB until the request is finished (while its being buffered). With larger files (500MB or even larger) the memory usage is even bigger; multiple GBs. IT would be great if there was some way of reducing the memory usage when posting very large requests.

    It sounds like I'm just complaining here, but I want to be clear; Fiddler is really, really great. It may just be that I'm trying to use it for a purpose which it wasn't designed...
  4. Answer
    Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 15 Aug 2014 Link to this post

    Hi, Graeme--

    If you have one of the TextView inspectors open to display a huge body, especially if that body isn't text, then yes, Windows will definitely spin a ton of CPU to attempt to render it as text. For huge bodies, especially those that are binary, you're best either limiting yourself to the HEADER inspector, or the HexView inspector (if you want to see the body).

    In terms of memory usage, you should generally see roughly a maximum of 200% overhead during the request read operation, because .NET internally grows its buffers by doubling their size each time the prior allocation was exceeded. However, this is in some respects an accounting trick, because the system's virtual memory manager does work to ensure that these allocations are less of a hit on physical memory than the numbers alone might suggest.

    Seeing overhead as large as you've described is a bit of a surprise, with the caveat that it's entirely possible that the garbage collector simply hasn't yet run due to a lack of memory pressure on the system; clicking Help > About forces a GC and can eliminate that as a culprit.

    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
  5. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 15 Aug 2014 in reply to Eric Lawrence Link to this post

    I'm actually trying to debug a .NET application which can be a bit memory hungry itself. We have a similar little hidden trick for forcing a GC :-)

    I'll see if that makes any difference.

    Thanks
  6. Graeme
    Graeme avatar
    31 posts
    Member since:
    Aug 2014

    Posted 15 Aug 2014 in reply to Eric Lawrence Link to this post

    Looks like you're spot on: I tried it with a 551MB request and it took about 1.2GB (double the request + a bit for the program) then when I forced the garbage collection it went down to around 650MB (551MB + a bit for the program).

    It would be nice if I could choose not to buffer requests like I can for responses, but that's not available by design I guess.

    Thanks for the great support!
  7. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 18 Aug 2014 Link to this post

    Hello, Graeme--

    Yes, the inability to stream requests is a limitation in Fiddler (actually, RPC-over-HTTPS traffic can be streamed, but that's a narrow exception). Enabling request streaming is something that I'd like to enable in the future, but it's quite complex to get it right and it may end up breaking scripts or extensions.

    Regards,
    Eric Lawrence
    Telerik
     

    Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.

     
Back to Top