I have an app that uses HTTPS and I can successfully use fiddler to view and modify the decrypted traffic. The app often does post requests of very large files (hundreds of MB or more) and fiddler seems to use loads of memory and become slow to respond when these large post requests are happening.
I'm hoping there is some way to not log the request/response body and just log the headers.
I have tried using stream mode which seems to make no difference; the body of the large request is still logged
I also added these two lines:
oSession["log-drop-request-body"] = "yes";
oSession["log-drop-response-body"] = "yes";
to the OnBeforeRequest function. This results in the body not being logged after the request/response has completed, but while it is still going, the body is logged which is what causes the problem if the request is very large.
Is there anything else that I can try to make it behave better with these large posts?
I'm hoping there is some way to not log the request/response body and just log the headers.
I have tried using stream mode which seems to make no difference; the body of the large request is still logged
I also added these two lines:
oSession["log-drop-request-body"] = "yes";
oSession["log-drop-response-body"] = "yes";
to the OnBeforeRequest function. This results in the body not being logged after the request/response has completed, but while it is still going, the body is logged which is what causes the problem if the request is very large.
Is there anything else that I can try to make it behave better with these large posts?
6 Answers, 1 is accepted
0
Hi, Graeme--
Fiddler inherently buffers request bodies before sending them; if you set log-drop-request-body, it will drop the body as soon as it is transmitted.
In contrast, if you have Response Streaming enabled and log-drop-response-body set, Fiddler will drop each chunk of the response body as it is read from the server and passed to the client.
Having said that, I'd like to learn more about your claim that Fiddler "become slow to respond" in this situation. Can you be precise about what specifically is slow? I've not experienced any slowness in this area, and I'm not aware of any technical reason why you'd see a performance difference here.
Regards,
Eric Lawrence
Telerik
Fiddler inherently buffers request bodies before sending them; if you set log-drop-request-body, it will drop the body as soon as it is transmitted.
In contrast, if you have Response Streaming enabled and log-drop-response-body set, Fiddler will drop each chunk of the response body as it is read from the server and passed to the client.
Having said that, I'd like to learn more about your claim that Fiddler "become slow to respond" in this situation. Can you be precise about what specifically is slow? I've not experienced any slowness in this area, and I'm not aware of any technical reason why you'd see a performance difference here.
Regards,
Eric Lawrence
Telerik
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.
0
Graeme
Top achievements
Rank 1
Rank 1
answered on 15 Aug 2014, 10:29 AM
Hi Eric,
The slow to respond situation occurs when I click on a session that has a large amount of data in the body and I have the inspector tab in view. I guess it's slow to display the body because it's so large (many megabytes).
When I'm uploading a 155MB file and and dropping the bodies from the log, the memory usage grows up to about 800MB until the request is finished (while its being buffered). With larger files (500MB or even larger) the memory usage is even bigger; multiple GBs. IT would be great if there was some way of reducing the memory usage when posting very large requests.
It sounds like I'm just complaining here, but I want to be clear; Fiddler is really, really great. It may just be that I'm trying to use it for a purpose which it wasn't designed...
The slow to respond situation occurs when I click on a session that has a large amount of data in the body and I have the inspector tab in view. I guess it's slow to display the body because it's so large (many megabytes).
When I'm uploading a 155MB file and and dropping the bodies from the log, the memory usage grows up to about 800MB until the request is finished (while its being buffered). With larger files (500MB or even larger) the memory usage is even bigger; multiple GBs. IT would be great if there was some way of reducing the memory usage when posting very large requests.
It sounds like I'm just complaining here, but I want to be clear; Fiddler is really, really great. It may just be that I'm trying to use it for a purpose which it wasn't designed...
0
Accepted
Hi, Graeme--
If you have one of the TextView inspectors open to display a huge body, especially if that body isn't text, then yes, Windows will definitely spin a ton of CPU to attempt to render it as text. For huge bodies, especially those that are binary, you're best either limiting yourself to the HEADER inspector, or the HexView inspector (if you want to see the body).
In terms of memory usage, you should generally see roughly a maximum of 200% overhead during the request read operation, because .NET internally grows its buffers by doubling their size each time the prior allocation was exceeded. However, this is in some respects an accounting trick, because the system's virtual memory manager does work to ensure that these allocations are less of a hit on physical memory than the numbers alone might suggest.
Seeing overhead as large as you've described is a bit of a surprise, with the caveat that it's entirely possible that the garbage collector simply hasn't yet run due to a lack of memory pressure on the system; clicking Help > About forces a GC and can eliminate that as a culprit.
Regards,
Eric Lawrence
Telerik
If you have one of the TextView inspectors open to display a huge body, especially if that body isn't text, then yes, Windows will definitely spin a ton of CPU to attempt to render it as text. For huge bodies, especially those that are binary, you're best either limiting yourself to the HEADER inspector, or the HexView inspector (if you want to see the body).
In terms of memory usage, you should generally see roughly a maximum of 200% overhead during the request read operation, because .NET internally grows its buffers by doubling their size each time the prior allocation was exceeded. However, this is in some respects an accounting trick, because the system's virtual memory manager does work to ensure that these allocations are less of a hit on physical memory than the numbers alone might suggest.
Seeing overhead as large as you've described is a bit of a surprise, with the caveat that it's entirely possible that the garbage collector simply hasn't yet run due to a lack of memory pressure on the system; clicking Help > About forces a GC and can eliminate that as a culprit.
Regards,
Eric Lawrence
Telerik
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.
0
Graeme
Top achievements
Rank 1
Rank 1
answered on 15 Aug 2014, 02:08 PM
I'm actually trying to debug a .NET application which can be a bit memory hungry itself. We have a similar little hidden trick for forcing a GC :-)
I'll see if that makes any difference.
Thanks
I'll see if that makes any difference.
Thanks
0
Graeme
Top achievements
Rank 1
Rank 1
answered on 15 Aug 2014, 03:32 PM
Looks like you're spot on: I tried it with a 551MB request and it took about 1.2GB (double the request + a bit for the program) then when I forced the garbage collection it went down to around 650MB (551MB + a bit for the program).
It would be nice if I could choose not to buffer requests like I can for responses, but that's not available by design I guess.
Thanks for the great support!
It would be nice if I could choose not to buffer requests like I can for responses, but that's not available by design I guess.
Thanks for the great support!
0
Hello, Graeme--
Yes, the inability to stream requests is a limitation in Fiddler (actually, RPC-over-HTTPS traffic can be streamed, but that's a narrow exception). Enabling request streaming is something that I'd like to enable in the future, but it's quite complex to get it right and it may end up breaking scripts or extensions.
Regards,
Eric Lawrence
Telerik
Yes, the inability to stream requests is a limitation in Fiddler (actually, RPC-over-HTTPS traffic can be streamed, but that's a narrow exception). Enabling request streaming is something that I'd like to enable in the future, but it's quite complex to get it right and it may end up breaking scripts or extensions.
Regards,
Eric Lawrence
Telerik
Check out the Telerik Platform - the only platform that combines a rich set of UI tools with powerful cloud services to develop web, hybrid and native mobile apps.