This is a migrated thread and some comments may be shown as answers.

FiddlerApplication not capturing traffic(sometimes)

18 Answers 1714 Views

FiddlerCore

This is a migrated thread and some comments may be shown as answers.

Bryan asked on 15 Jan 2021, 12:09 AM

Problem: When calling FiddlerApplication.Startup(startupSettings), experiencing an intermittent ~5-10% of all instances of this call, a failure to capture traffic, all other instances it works as expected.

FiddlerApplication instance is not throwing any errors when this happens
FiddlerApplication.Log.OnLogString is also not producing any log lines to indicate why this may be occurring

When it works, we see (brief sample) FiddlerApplication logging like this:

Setting upstream gateway to none
/Fiddler.CertMaker> Invoking CertEnroll for Subject: CN=*.somesite.net, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com; Thread's ApartmentState: MTA
/Fiddler.CertMaker> Finished CertEnroll for 'CN=*.somesite.net, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com'. Returning cert
/Fiddler.CertMaker>31 A racing thread already successfully CreatedCert(*.somesite.net)
/Fiddler.CertMaker> Invoking CertEnroll for Subject: CN=*.www.othersite.com, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com; Thread's ApartmentState: MTA
/Fiddler.CertMaker> Reusing PrivateKey for '*.www.othersite.com'
/Fiddler.CertMaker> Root Certificate located; private key in container 'b4b5bddc0d8ff99a64e1f97df8016840_3049990c-dfd5-4a02-97c7-fb6cb7179000'
[Fiddler] No HTTPS request was received from (msedge:15904) new client socket, port 54026.

When it does not work, these are the only lines that FiddlerApplication logs:

Setting upstream gateway to none
/Fiddler.CertMaker> Root Certificate located; private key in container 'bb04e7ae8f85fb676707737774021be5_67cd9fb4-0771-4c3d-8869-a56e07389aaa'

We only have 1 reliable way of reproducing this, but have found it to occur when the one condition for the reproduction is not met as well

The reproduction involves a Click2Run update that does not work properly having the main application calling a older version of our library that encapsulates all of the FiddlerApplication implementation
We have no indication why/how this is happening currently, nor why executing the older version of the DLL is producing this result with FiddlerApplication
We have seen this repo’d in at least 1 instance (with reports of many others as well) where it appears the properly updated/expected dll was invoked
We have yet to rule out if the issue from the 100% reproducible variant is also manifesting in some way here as well

Consequently we see no traffic captured and logged by FiddlerApplication, and then no SAZ file is produced, which may* be expected since FiddlerApplication did not log CertEnrolls for anything nor throw an error.

We have suspicions we are unable to confirm currently as follows:

Our FiddlerApplication Cert, Proxy, or StartUp settings code is somehow flawed in a way we cant identify that is producing this unexpected transient result
The reproductions by our customers that exist outside of the failed C2R update reproduction has a complex local environment (Certs/Proxy/other) that is fashioning valid start up settings/environment for FiddlerApplication to run without issue, but during runtime it is listening in an unexpected way/domain such as to not actually be able to see/monitor/capture the desired traffic.
The previously mentioned update repro’s root cause existing when it appears the correct dll is executed

Any help identify to the root cause for the issue described here as we have no indication from logging or Exceptions, and make any recommendation we need to take to remediate such issue.

18 Answers, 1 is accepted

answered on 19 Jan 2021, 02:16 PM

Hey Bryan Hunwardsen,

Thanks for reaching out to us and for using FiddlerCore!

It is hard to pinpoint what might be causing the issue, but from what you are describing, it looks like that there might be some racing conditions that are causing the certificate creation sometimes to fail. Could you provide more information about how your application is calling the certificate creation logic?

You could also try to use Bouncy Castle to create and persist your certificate (check this documentation article for more details.). Please let us know if you are experiencing the same issue while using the bouncy castle API.

Regards,
Nick Iliev
Progress Telerik

Virtual Classroom, the free self-paced technical training that gets you up to speed with Telerik and Kendo UI products quickly just got a fresh new look + new and improved content including a brand new Blazor course! Check it out at https://learn.telerik.com/.

answered on 22 Jan 2021, 12:20 AM

Our Cert code boils out to essentially this, it has been minified omitting error checking, telemetry and collapsing nested code for simplicity (I *may have mangled something in the minification but the broad strokes should be there).

This completed our cert work significanlty before FiddlerApplication.StartTracing is called, so Im not sure where/why you think there might be a race condition. In troubleshooting we have put trace logging before/after almost every line of code, as well as a ridiculous amount of nested exception checking/logging. While our telemetery does have instances of this below code reporting exceptions, in all of the instances we have reproduced the above issue, this cert block logs normal, proper operation, although I have not ruled out an issue with this code missing something that is not manifesting to an exception.

01.private void CertSetup()

02.{

03. FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true);

04. FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.CleanupServerCertsOnExit", true);

05. FiddlerApplication.Prefs.SetStringPref("fiddler.certmaker.assembly", Path.Combine("C:\\", Guid.NewGuid().ToString("N") + ".dat"));

06.

07. CertMaker.EnsureReady();

08.

09. bool trustCert = false;

10. while (!certInstalled || !trustCert)

11. {

12. bool certCreated = CertMaker.createRootCert();

13. X509Certificate2 certificate = null;

14. if (certCreated)

15. {

16. trustCert = CertMaker.trustRootCert();

17. certificate = CertMaker.GetRootCertificate();

18. }

19. if (certCreated && certificate != null)

20. {

21. if (!CertMaker.rootCertIsMachineTrusted())

22. {

23. using (X509Store certStore = new X509Store(StoreName.Root, StoreLocation.LocalMachine))

24. {

25. certStore.Open(OpenFlags.ReadWrite);

26. certStore.Add(certificate);

27. }

28. }

29. }

30. certInstalled = this.DetectFiddlerCerts();

31. if (certInstalled || !trustCert)

32. {

33. CertMaker.removeFiddlerGeneratedCerts();

34. }

35. }

36.}

37.

38.private bool DetectFiddlerCerts()

39.{

40. return CertMaker.rootCertExists() &&

41. CertMaker.rootCertIsTrusted() &&

42. CertMaker.rootCertIsMachineTrusted();

43.}

answered on 24 Jan 2021, 06:11 PM

Hello Bryan,

Thank you for the additional details that you have shared. We investigated the code, and we have few things that we need to clarify as follows:

1. Which version of FiddlerCore are you currently using? If you are using version 5.0.0 (of FiddlerCore), please let us know if you are using .NET 4.0, 4.5, or netstandard2?

2. There is a boolean called certInstalled. Where and how is this property set?

3. Is the following code confirmed to work as expected? Is loading a random RandomGuid.dat instead of CertMaker.dll working as expected?

FiddlerApplication.Prefs.SetStringPref("fiddler.certmaker.assembly", Path.Combine("C:\\", Guid.NewGuid().ToString("N") + ".dat"));

4. The boolean set in the code below will return true if there is already added root certificate in the certificate store.

bool certCreated = CertMaker.createRootCert();

That would mean that the following code will be executed:

if (certCreated)
{
    trustCert = CertMaker.trustRootCert();
    certificate = CertMaker.GetRootCertificate();
}

CertMaker.trustRootCert() will add a root certificate but won't check if it exists. There is a possibility that adding the certificate multiple times could manifest as unexpected errors or behavior.

5. It looks like that certInstalled = this.DetectFiddlerCerts(); will return true (after the cycle), so unless there are some errors in the certificate creation and trust processes, then it will always lead to the following code being executed:

if (certInstalled || !trustCert)
{
    CertMaker.removeFiddlerGeneratedCerts();
}

This code should remove all Fiddler certificates. If we understand that part of the code correctly, it looks like you could experience failure even more often than 5-10% of the cases.

Regarding BouncyCastle and CertEnroll usage

The team recommends using BouncyCastle instead of CertEnroll for both installation and persisting your root certificate. If you are interested in implementing BouncyCastle, you could refer to the blog post link below for details on using it in your NET application.

https://weblog.west-wind.com/posts/2014/jul/29/using-fiddlercore-to-capture-http-requests-with-net

Regards,
Nick Iliev
Progress Telerik

answered on 26 Jan 2021, 08:14 PM

1. The following versions are associated to the DLL we consume:

C# Project Target .Net Framework: 4.6.2

Path to dll: ...\FiddlerCore.Licensed.4.6.2.7\lib\net45\

DLL file version 4.6.20191.7809

DLL runtime version 4.0.30319

2. certInstalled is a class variable set on line 30 of the posted code, however, as part of the minification, I omitted the calling code block that calls DetectFiddlerCertificates() and skips over CertSetup() when true.

3. I have inherited this code/problem, so please forgive me as I am reverse engineering, looking up documentation, learning of the fly, and possibly making some mistakes here. Outside of the larger CertReady class, we make no explicit calls to CertMaker, Im assuming that FiddlerApplication is using this path to find/load CertMaker for cert reads on instantiation/StartUp if it is not otherwise loaded/available. The original dev left the following comment on the line in question "// Provide a bogus certmaker path so we fall back to certenroll even if certmaker is present." however I am unsure why this decision was made/necessary, how it materially impacts the issue at hand, and what remediation may be warranted, but I am open to suggestions.

4, Another casualty of minification, We call CertMaker.removeFiddlerGeneratedCerts(true) prior to calling CertSetup(), for which the posted code implicitly leverages the expected state. That is not to say there is no issue here, but it seems the original dev has coded for the scenario you have described and should not manifest as an issue provided CertMaker.removeFiddlerGeneratedCerts(true) works as expected, however, the code is explicitly catching/logging errors from this removeCerts call but is not inspecting the return value of the call. The available documentation is not explicit for condition(s) to return false vs throw exception. I will add in the return value checking immediately, but also need to log cause at runtime, what options are available to capture the cause for a false return value from CertMaker.removeFiddlerGeneratedCerts()?

https://docs.telerik.com/fiddlercore/api/fiddler.certmaker#Fiddler_CertMaker_removeFiddlerGeneratedCerts_System_Boolean_

5. apologies, this was a posting typo, code is actually:

if (!certInstalled || !trustCert) { CertMaker.removeFiddlerGeneratedCerts(); }

RE: BouncyCastle, thank you for the recommendation. As the current implementation is working for 90-95% of our current traffic, the risk/cost to changing to BouncyCastle will not be considered until we have successfully found and understand the current issue cause.

answered on 28 Jan 2021, 11:48 PM

Hi Bryan,

I am Simeon from the development team of Fiddler. Thank you for the information. Hopefully, it will help us solve your issue.

Regarding 2: I understand that somewhere else in the code (not the provided one) DetectFiddlerCerts is called and if it returns false CertSetup called. It is possible that there might be a problem here. DetectFiddlerCerts calls CertMaker.rootCertExists which internally calls CertMaker.EnsureReady. The last one tries to load the ICertificateProvider but at this point it is possible that these Fiddler prefs are not set: FiddlerApplication.Prefs.SetBoolPref("fiddler.certmaker.PreferCertEnroll", true); and FiddlerApplication.Prefs.SetStringPref("fiddler.certmaker.assembly", Path.Combine("C:\\", Guid.NewGuid().ToString("N") + ".dat")); This could mean that FiddlerCore will look for the CertMaker.dll next to the exe file of the application. If it finds it it will load it. Please, note that the Bouncy Castle CertMaker.dll implementation comes along with FiddlerCore so if it is not removed FiddlerCore will find it in this scanario.

FiddlerCore should be writing in the log which certificate provider it is using. You should be able to find the following message:

/Fiddler.CertMaker> Using {0} for certificate generation; UseWildcards={1}.

Regarding 3: If the CertMaker.dll is not present in the directory of the application, then there is no need to set FiddlerApplication.Prefs.SetStringPref("fiddler.certmaker.assembly", Path.Combine("C:\\", Guid.NewGuid().ToString("N") + ".dat")); as FiddlerCore will default to CertEnroll, but still this pref does not hurt.

Regarding 4: CertMaker.removeFiddlerGeneratedCerts tries to remove the Fiddler generated certificates from the certificate store of the user. Please, note that it does not remove the certificate from the store of the machine. It will return false if there is an exception.

Finally, I would like to ask some questions. Is the CertMaker.removeFiddlerGeneratedCerts(true) called on each start/stop of the application? This could mean that a new root certificate is generated each time the application is run. If so, please, consider installing a root certificate once and using it multiple times. Please, note that the Fiddler's implementation of the CertEnroll is not designed to use a different certificate on each start/stop. In the classic Fiddler for Windows (which uses CertEnroll by default) the users are advised to restart Fiddler after trusting the root certificate.

And one more thing - You are telling that FiddlerCore is not producing saz files in some cases - is it possible that FiddlerCore captures the sessions and keeps them in the memory but for some reason it is not writing them in a saz file. Can you confirm that there are no captured sessions in the memory?

Thanks and Regards,
Simeon
Progress Telerik

answered on 29 Jan 2021, 01:28 AM

Thank you for the reply Simeon,

I see you have responded on Ticket ID: 1504286 to Patrick, please note that I do not have access to the support ticket request, and only receive (some) notifications, so should we need to move in any way back through support ticket infrastructure I will be second or third hand removed from those communications.

RE: 2) This is another issue arising from minification to the code, please understand I have hesitancy posting raw, non-obfuscated/minified enterprise code to a public space. Apologies as this has led to some misdiagnosis. In this instance, those preferences are actually set in the default constructor of the CertSetup class prior to any method being called. So I do not believe at this time for this to be an issue. CertMaker.dll is not published to the app executable directly, but I am still looking to see if it is not hidden/re-packed inside another one of our application libraries available in this directory. I do not find the log line you have listed, possibly because we do not register to FiddlerApplication.Log.OnLogString until well after CertSetup has completed, alternatively we are registering for/with any additional logging that would produce this during CertSetup. We register just prior to calling FiddlerApplication.StartUp after which we do see /Fiddler.CertMaker> log lines for each connection request we are listening for.

Re: 4) we where not inspecting the return value, however, our telemetry indicates this line of code is not even being called, so there was no opportunity for it to fail. This leads me to a follow-up question, although we use prefs to remove all certs on exit, knowing CertEnroll stores all site certs in the CertStore where the Root Cert is as well, is it possible the faiures are occuring when a previos run of FiddlerCore was not allowed/ or otherwise failed to cleanup the root cert, and the second execution of our code is somehow installing a second root cert, and/or trusting the wrong cert, and/or miss-mixing the trust chain? I ask because we never callFiddlerApplication.Shutdown(), instead we revert our proxy setting back to their original state then dispose of the CompositionContainer only. I recently tried calling Shutdown, but it would generate a bunch of undesireable errors in our logging I believe to currently open connections being disposed, while I found a place (can't recall) indicating that disposing of the container was the preferred method. I Imagine it is working correctly as most of our we generally have no indication (including our own internal testing) that there is a systemic issue with failure to clean up certs on shutdown impacting our next execution. However, I dont think we are explicity able to rule out some late stage failure that does not get logged as part of shutdown.

Re: addition questions-

We call CertMaker.removeFiddlerGeneratedCerts(true) in 2 scenarios, once if on startup, everything is already installed and trusted, we call it to remove everything and essentially start over from a clean slate, however our telemetry indicates this code is not ever being called. We also call it as part of clean/up shut down of every execution, that would only be omitted by an abnormal execution/termination of our application. We also call CertMaker.removeFiddlerGeneratedCerts() when a failure occurs to install or trust the root cert on startup, we then loop the install code again, which If Im following the logic correctly, could potentially install multiple root certs if there is intermittent/persistent failure to trust a successfully installed root cert and probably should be corrected to uninstall the root certs or omit calling CertMaker.createRootCert(); if the root cert was successfully created. Despite design intentions, we must remove the cert each time knowing this causes overhead and complicates the cert management, it is by policy "from above".

Lastly, the communication driven opening the ticket was somewhat stale, observationally, I can state that I dont believe it is an in/memory write/out to saz issue. The saz always writes out fine when FiddlerCore has captured traffic. This conclusion is driven by our telemetry which consistently shows 2 things: 1) the List<Sessions> _fiddlerSessions object we pass to

Monitor.Enter(_fiddlerSessions );
AllSessions.Add(currentSession);
Monitor.Exit(_fiddlerSessions );

as part FiddlerApplication.BeforeRequest

that after disposing of the CompositionContainer, _fiddlerSessions has no sessions in it

As well as, FiddlerApplication.Log.OnLogString has not logged a single server certificate for the execution when we typically see multiple logs per instance beginning with a line like: /Fiddler.CertMaker> Invoking CertEnroll for Subject: CN=*.somesite.net, O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com; Thread's ApartmentState: MTA

Which both indicate to me it is a capture issue, and nothing more down the line.

It is as if FiddlerCore is successfully configured and running capture, but is listening in an empty room.

Im hoping with our support ticket open we can get a faster response the 24-48 hours, please let me know if I need to do something on my side to enact speedier replies.

Thanks

answered on 29 Jan 2021, 09:38 AM

Hi Bryan,

We noticed that the issue we are discussing in the forum thread is hard to pinpoint based on the different partial code snippets that were provided so far. We are also not entirely sure on what conditions should be met to reproduce it on our side. In order to continue the investigation, we would need to the following:

- A sample POC application that can reproduce your issue. Ideally, that would mean the original application where FiddlerCore is implemented, but if due to some reason that is not an option (e.g., confidentiality), then perhaps you could try to isolate the issue in a basic test application. This sounds like a crucial steps that would help our team to identify how to the issue is manifesting. Use the private ticket thread if you don't want to make a public share of the POC test application.

- Details about the network configuration needed to reproduce the issue. In your initial forum thread, you talked about a specific customer environment (when the problem appears). We would need more instructions and clarification on that part:

... The reproductions by our customers that exist outside of the failed C2R update reproduction has a complex local environment (Certs/Proxy/other) that is fashioning valid startup settings/environment for FiddlerApplication to run without issue, but during runtime it is listening in an unexpected way/domain such as to not actually be able to see/monitor/capture the desired traffic.

- More information about the C2R tool - how it is involved in the issue reproduction, and could we have access to that tool along with the test application that we would use for reproduction. Is the issue only happening when the mentioned C2R tool is involved? Does the C2R tool needs special proxy settings and could it reflect a change that is made dynamically (after the C2R tool is started and running). Any detailed information about how C2R is related to the case would be of great help.

- Post any additional steps that are crucial for reproduction of the issue. For example, is it happening on some runtime network changes or with specific domains only?

Meanwhile, our developers are looking into the last information that came from you and continuing the investigation.

As a side note: I will post the above information in both the public forum post and our private ticket thread. If you wish, please feel free to use the private ticket for sharing any confidential information, libraries, and applications. Meanwhile, we will continue the conversation there (unless your team prefers otherwise).

Regards,
Nick Iliev
Progress Telerik

answered on 29 Jan 2021, 08:30 PM

Hi Nick,

A couple of things:

Please disregard the quote you have posted in the POC request section of your last post, the context was simply reciting a hypothetical possible issue, not a declaration of observed reproduction.
C2R : Click-To-Run is a Microsoft technology for .Net based application packaging/deployment/installation/updates.

We have internally only been able to reproduce this issue under a very specific circumstance as follows:

1) Our application(App v1.0) with supplementary library -containing FiddlerCore- (Lib v1.0) is installed and working as expected.

2) A newer application revision is published to the application (deployment) server (including version revisions to all supplementary libraries even if their code had not changed).

3) When the deployed application launches again, it detects the new version and updates itself.

4) The update happens at two levels: 1)primary application, 2) supplementary libraries

We observed that in this scenario, the primary application was being updated to (App v1.1) and via telemetry we could validate it was running the expected code. We expected the supplemental library, that contains our implementation of FiddlerCore, to be (Lib v1.1), but our telemetry revealed (App 1.1) was calling/executing (Lib 1.0) unexpectedly. (There was no code change between (Lib v1.0) and (Lib v1.1). In this instance our application and FiddlerCore still run as expected, with one exception, when FiddlerCore successfully executes FiddlerApplication.Startup it does not find/capture to memory/log any traffic.

Uninstalling the application and reinstalling it produced expected matching App and Lib versions and everything again worked as expected, which implies exemption to any underlying System level issues as a cause.

We do not find any other issue with the application performance other than FiddlerCore not finding traffic to capture, not logging any issues with its ability to listen/capture or throw any exceptions. My assumption was that during the application upgrade process, the library mismatch was somehow causing FiddlerCore to perform unexpectedly despite running the same library code and having no indications from Telemetry or FiddlerCore logging to the contrary.

What has us more concerned, and the genesis for the ticket, is that we see production telemetry reproducing the FiddlerCore issue when no upgrade has occurred, and bot main Application and Supplementary library are matching/correct. We have been completely unable to reproduce this in-house, which means we cannot provide a POC/way for you to reproduce it either. We have exhaustive telemetry in this code as we have searched for the root cause, there has been no identifiable correlation of any metric that would relate one instance to another, some of the additional information you/we might find useful is only generated locally on client systems due to privacy policies for which we do not have access.

During RCA testing I ran a test whereby I closed all other applications/processes and disconnected from the network, and still capture ~40 background connection requests on my system in about 5 seconds. Another test I tried(repeatedly) was to quit capturing immediately after starting (via our UI) and even a sub 1 second response still would produce several captured connection requests. Both cases I was trying to run our FiddlerCore code in a way that it did not capture (to memory) any traffic. Both of these set-ups failed to reproduce the desired outcome.

Circling back to my root question(s), what could cause FiddlerCore to start tracing and not find any traffic to capture to memory? And/Or, what can be done, to help us find out why/how this is happening?

answered on 02 Feb 2021, 08:48 AM

Hello Bryan,

Thanks for the clarifications!

It is possible that there are two different problems, and it is not clear what the link is, if any, between them. One of the issues is that the App is not updating its libraries correctly, and the other is that sometimes FiddlerCore is not capturing any traffic. We want to focus on the problem with FiddlerCore.

There are two scenarios I can think of when this could happen. The first one is when the application which sends the HTTP/S requests is not aware of the proxy settings pointing to FiddlerCore. One reason for this is if FiddlerCore does not change the system proxy settings for the HTTP and HTTPS protocols to 127.0.0.1:8866 (by default), a.k.a Attach. You can check if FiddlerCore is attached by checking the FiddlerCore.IsAttached property. However, even if FiddlerCore is attached, it is possible that the application does not respect the system proxy settings, a.k.a. WinINET, and the Internet Explorer proxy settings. Or maybe it respects them but checks them only once - on startup of the application. If FiddlerCore is not attached during this time, it will miss the proxy settings update if it is not listening for such an event during runtime of the application.

Another problem might be that FiddlerCore tries to start listening on a port that is already taken by another process. A typical scenario is when the previous process of the application running FiddlerCore does not close properly, and the port is not released. Then when the application is started again, FiddlerCore tries to start listening on the same port. FiddlerCore may bind successfully to the same port, but it is not getting the operating system's requests as they are routed to the first process. If this is the problem, you can try to start FiddlerCore to listen on port 0 - this will take a random free port.

If you don't see any traffic captured by FiddlerCore, even no HTTP CONNECT requests, this could mean this is the case. Otherwise, if there are HTTP CONNECT requests, FiddlerCore is attached and listens properly, but there is a problem with the HTTPS traffic and certificates.

Below I would like to address some of your comments from the previous post.

RE: 2) "...I do not find the logline you have listed, possibly because we do not register to FiddlerApplication.Log.OnLogString until well after CertSetup has completed, alternatively, we are registering for/with any additional logging that would produce this during CertSetup. We register just prior to calling FiddlerApplication.StartUp after which we do see /Fiddler.CertMaker> log lines for each connection request we are listening for."
- I recommend attaching to the FiddlerApplication.Log.OnLogString event to be the first API call you make to FiddlerCore in order to get all the logs that FiddlerCore produces. This could give us some insights into what is happening.

Re: 4) Setting the "fiddler.certmaker.CleanupServerCertsOnExit" pref to true will affect only if you call the CertMaker.DoDispose() at the end. It should be called afterFiddlerApplication.Shutdown() or after you revert the proxy settings and dispose the CompositionContainer.

"I recently tried calling Shutdown, but it would generate a bunch of undesirable errors in our logging I believe to currently open connections being disposed, while I found a place (can't recall) indicating that disposing of the container was the preferred method."

- Yes, the FiddlerApplication.Shutdown documentation mentions that if there's any traffic in progress while you're calling this method, your background threads are likely to blow up with ObjectDisposedExceptions or NullReferenceExceptions.

Regards,
Nick Iliev
Progress Telerik

answered on 09 Feb 2021, 05:46 PM

@Nick,

Thanks for the continued support, I have taken your feedback a made several revisions to our code to hopefully get us closer to finding/fixing the issue. This code should be deploying to production this week and I should have telemetry by the end of the week from the new code.

Notably, I moved the logging setup to be the first API call as suggested, and I can now see the indicated log line that was mentioned, however it reports out as:

/Fiddler.CertMaker> Using .+ for certificate generation; UseWildcards=True.

I am unsure yet what this means, but at least we are capturing it now.

From all of the initial feedback you have given, my first goal of the refactored code was to better ensure the Fiddler certificate state is as expected. Our intention is to leave the client executing environment without artifacts upon completion, and we do a best effort to clean up cert, proxy and firewall changes before our application exits. But of course, there are still circumstances whereby termination of our application will leave these artifacts in place. As well as the possibility of such certs existing from another Fiddler implementation. I know you have previously stated that in general the idea is to leave the trusted root cert in place, but that CertMaker.createRootCert() will create a second root cert even if one already exists.

Would you kindly please clarify/correct the following:

-When FiddlerApplication.Startup is called, if there are multiple root certs available (possibly in various states of trust/not trusted), then the output of FiddlerApplication may fail to capture sessions to memory. => We should ensure only one root cert exists(purge and recreate) in the store with all trust validated before calling Startup. We should also purge all Fiddler server certs before each call to FiddlerApplication.Startup as it may impact FiddlerApplication from collecting sessions to memory.

Once we get a look at the next set of telemetry I may have more data to help up with. Our ticket has been closed due to inactivity as we have been working through this forum. Do I need to have it reopened in order to ensure proper allocation of continued support here/or there?

The next thing I would like to address is the stale process holding onto the default port and receiving traffic instead of the newly bound process. Given some time I may be able to validate this if we roll back our ClickOnce update changes in a DEV environment to enable me to use execute our 1 reproduction. However, even during our reproduction scenario, the update to the application, or the application re-start (under admin credentials) all happens before CertMaker or Fiddler application is called, so they *should all be running under the same/current process. Aside from implementing the random port suggestion, does FiddlerApplication have any ability to detect existing bindings to the port, or would I have write some custom code for that. (We would like to detect and classify the issue before making changes like the random port change)?

I will await a reply, and post further updates here as I have them.

Thanks

answered on 10 Feb 2021, 12:28 PM

Hi Bryan,

There's no need to reopen the ticket - we are monitoring this thread closely and we investigate it with high priority. Of course, if you want to share private information, you can do this in the ticket instead of here.

Please find below our answers of your questions:

- The "/Fiddler.CertMaker> Using .‰+˜ for certificate generation; UseWildcards=True." log looks okay, the strange `. +` symbols are logged because of FiddlerCore obfuscation (it should be the type of the certificates creator instance).

- Regarding the certificates clean up before the FiddlerApplication.Startup, yes, we may fail to capture sessions if there are outdated certificates and it’s a good idea to ensure the clean state before the Startup call. Reviewing your code snippets, we saw that you are setting both user and machine level certificates but bear in mind that CertMaker.removeFiddlerGeneratedCerts is cleaning only the user-level certificates. I suggest you check if you really need the machine-level trusting and ensure you are properly cleaning it, or keep only the user-level certificate trusting.

- Regarding the stale processes holding the default port, you can either pass 0 for a random free port or write a custom logic for killing such processes, there’s nothing out of the box in FiddlerCore for detecting and cleaning such processes.

- Regarding the IsAttached deprecation, as far as I see you are attaching the Fiddler proxy during FiddlerApplication.Startup and this property will work fine for you, you could just ignore or suppress the warning.
The Telerik.NetworkConnections.NetworkConnectionsManager.GetCurrentProxySettingsForConnection replacement of the IsAttached property is meant to be used when the proxy is manually attached by the users (i.e. manually set the OS proxy settings or manually call the Telerik.NetworkConnections internal APIs for managing the proxy settings). Its idea is that you can get the currently set proxy settings using the above-mentioned GetCurrentProxySettingsForConnection method and manually check if the proxy is attached by comparing the proxy value to the currently used FiddlerCore port.

Hope this helps!

Regards,
Rosen Vladimirov
Progress Telerik

answered on 03 Mar 2021, 12:56 AM

Hello Telerik Team!

I have an issue reproduction update/wrinkle to supplement the prior conversation.

We have been able to partially reproduce our issue again in a new manner but with a further unexpected outcome we could use your insight on.

As a refresher, our initial reproduction, involved our (ClickOnce deployed) main application was executing an incorrect/older version of our library that wraps FiddlerCore. Ultimately this led to no SAZ file being produced. Based on continued analysis, I suspect that for some reason due to calling the wrong version of the package, Fiddler was listening, but did not find traffic to capture, which distally resulted in no saz file being produced.

In this instance, we have a user that does not have admin privilege's running our application. The application determines it must be restarted with admin privilege's (to perform pre-req tasks like Certs, Proxy, etc). Via code, we initiate shutdown of our (non-admin instance) application as well as initiate the (re)start of our application with the "run as admin flag". The user is first presented with the Windows UAC dialog to authenticate admin access.

In this reproduction, the user(user's account) has no admin privilege, but the user did have a secondary set of credentials on the machine that does have admin privilege. Entering the alternate credentials caused the application/process to assign to the (admin) user's user space. The user started traffic capture, then browsed a few websites, then stopped traffic capture.

We see in our application telemetry a few lines of (fiddlercore forwarded) traffic logged at the start of tracing which looks like traffic coming only from our application/process, but none of the web traffic the user browsed while capturing. When the user stopped traffic capture, ]the expected saz was produced, but without any system or application traffic generated in/from his active userspace.

We know we need to fix the dll update issue so that it uses the correct/expected library, however, even in this instance running the unexpected dll, fiddlercore did find and capture traffic.

To me this seems to be some cross userspace issue. We set up all the certs, ports, and proxy under the admin account/userspace, and all the traffic capturing immediately follows (in code) from the same process in that user space. Off the top of my head, I could only suspect the proxy we apply might be limiting to traffic from the (admin) users userspace (which nothing else is happening in on that machine) and cannot see traffic from the active user and all system processes under that user, or some other Windows/Fiddler limitation/side-effect that is not proxy dependent, but manifests in essentially the same way.

Please provide any feedback, insight, follow-up questions you can to help us understand what/how this happens, and how we migfht fix it.

Thanks

answered on 04 Mar 2021, 07:00 AM

Hello Bryan,

We have few questions related to the issue you are describing:

1. What is the OS version where the issue happens? There is a difference in handling the userspace settings depending on whether you are using Windows 10, Windows Server, etc.

2. When setting up the proxy in the admin userspace, is the system proxy also set in the UI of the user's non-admin userspace?

Regards,
Nick Iliev
Progress Telerik

answered on 04 Mar 2021, 06:03 PM

Thanks for the reply Nick,

The users system was running Windows 10 Enterprise. Although I expect we still have users on Windows 7. While I cant currently confirm a reproduction on Windows Server (in progress), those would be a small subset of overall occurrences, but still may occur. I would prefer to focus first on Windows 10 (non-server) and we can enhance/address Win 7 and Server as needed.

I am not sure how to answer question number 2.

Much of our application is relies on Windows %userprofile% pathing for the installation of the application, runtime local logging, etc. When our application is restarted to enable admin elevation, and an alternate credential is provided to the UAC dialog, we see our application process running under the credential provided (essential a run-as with an admin privileged account) although the application itself is still running in the primary users UI session. We can also validate the our library wrapping fiddlercore is running from an installation path under the (admin) user account and not from the primary active users account.

All of the FiddlerCore related code, including proxy code, is running under the process credentials of admin elevated user account, but inside the primary users Window session.

This snippet is of our proxy code setup, I just dont understand the impacts of the proxy/setup when running in this situation.

Again, fiddlercore captured traffic originating from this process but nothing from processes running under the original user even though the original user account was the active windows UI session, and there was no (logged in but not active) session for the admin elevatable account (that I was aware of).

ProxySettings fiddlerProxySettings = new ProxySettings(true, "127.0.0.1", 9999, "*.sts.microsoft.com,<-loopback>");
connectionsManager.SetProxySettingsForConnections(fiddlerProxySettings, connectionsManager.GetAllConnectionFullNames().ToList());

answered on 05 Mar 2021, 01:21 PM

Hey Bryan,

We tested the scenario where FIddler is installed on a VM with two users - an admin and a user with limited privileges (we are calling this one the LimitedUser). The installation for FiddlerEverywhere (a UI tool that is using the FiddlerCore library) was made for the LimitedUser. Then we tried to run Fiddler with the LimitedUser and with the admin user. Here are our observations:

-Normal startup scenario: FIddler is started for the LimitedUser, and the proxy settings are being set as expected. Capturing traffic from MS Edge is working as expected. Interestingly, when running Edge as an admin, it request are still being captured (although at this point we are using it with the admin account)

- Run Fiddler as an admin scenario: In this case, the OS proxy is not being set, and this leads to no traffic being captured.
- Running Edge without an admin account is not capturing any traffic (this is expected as no proxy is being set for the LimitedUser).
- Running Edge as admin captures only one request (unexpected behavior from Edge).
- Running Chrome without an admin account is not recording any traffic (this is expected as no proxy is being set for the LimitedUser).
- Running Chrome as an admin - everything is being captured as expected (due to the Fiddler proxy being set for the admin account)

The above results are demonstrating the following:
- When we run an application that needs to set the proxy with another user, it won't set it for the machine, and that would lead to the proxy not being set for the active user (nothing being captured).
- Edge has strange misbehavior, where requests are always executed through the active user and not through the one who has started the app (run as admin scenarios...). Even when we explicitly ran Edge as admin, the browser is still executing another process that has the LimitedUser as an owner.

Here is a demonstration video of the above behavior.

All of the above said, the default behavior on Windows is that the proxy settings are being set per user and not per machine. If you want to change it, you will need to explore the Windows options to change that behavior so that the proxy settings are set per machine (and this for all users). I guess you would need something similar to the solution discussed in this thread.

Regards,
Nick Iliev
Progress Telerik