Python HTTPS requests do not work while Fiddler is running

3 Answers 6880 Views
Fiddler Classic Fiddler Everywhere
IamMusavaRibica
Top achievements
Rank 1
IamMusavaRibica asked on 12 Jun 2022, 08:58 AM | edited on 12 Jun 2022, 09:09 AM

I tried running some requests in Python to see if they will appear in Fiddler

For a HTTP request, it worked with no problem:

However, trying to send a request over HTTPS results in long blocking and eventually fails. Nothing is logged to Fiddler, because no actual request was made. After 20-30 seconds, a long error is raised in Python:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /?id=1234567890 (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1129)'))))

What is causing Python to stop working (only while Fiddler is running)?  My current Fiddler version is v5.0.20204.45441 which is, I know, an old version. I trusted the root certificate, even installed it manually, and the HTTPS traffic from my browser is displayed normally as it should.

3 Answers, 1 is accepted

Sort by
0
Lini
Telerik team
answered on 13 Jun 2022, 04:42 PM

Hi,

I noticed that a similar was reported recently on the urllib3 issue tracker and that a fix was merged for 1.26.x about 10 days ago. The next update of the library should work in your case or display a more informative error message.

In any case, please check your environment variables and make sure that HTTP_PROXY and HTTPS_PROXY (or ALL_PROXY) are set to the same value - e.g. "http://127.0.0.1:8888" if you are using the default Fiddler settings. If the environment variables look OK or your proxy is configured in another way, you need to modify the settings there.

Alternatively, you can try setting your proxy manually with code before the tests request:

import requests
...
proxies = {
   'http': 'http://127.0.0.1:8888',
   'https': 'http://127.0.0.1:8888'
}

response = requests.post(url, proxies=proxies)

Regards,


Lini
Progress Telerik

Love the Telerik and Kendo UI products and believe more people should try them? Invite a fellow developer to become a Progress customer and each of you can get a $50 Amazon gift voucher.

IamMusavaRibica
Top achievements
Rank 1
commented on 13 Jun 2022, 09:28 PM

Thanks for your response! However, I find this problem too generic, and even if I manage to send a request and it being captured, there would probably be countless other apps and software where requests would just be blocked
Lini
Telerik team
commented on 16 Jun 2022, 07:32 AM

If an app supports using a proxy for network traffic, then Fiddler can capture it. Almost all UI applications will just work out of the box with the system proxy settings, so the only thing you need to do is turn on capturing in Fiddler. For console applications, most will work with the "HTTP_PROXY/HTTPS_PROXY/ALL_PROXY" environment variables. You just need to set these before you execute the shell command. Finally, only if you write your own network requests using some API, then you will need to take into account adding the Fiddler proxy settings manually.
0
Hiroshi
Top achievements
Rank 1
Iron
answered on 09 Jul 2022, 03:02 PM

I got a same problem and arrived at this discussion.

What Lini said sounds like true and I found a workaround to install older version of urllib3.

pip install --upgrade urllib3==1.25.11


Lini,

I tried the latest urllib3 1.26.10 which was released on July 7, 2022, but the problem still persists. Can you tell me an exact post of urllib3 issue tracker? I was not able to find it.

Nick Iliev
Telerik team
commented on 10 Jul 2022, 08:58 PM

Lini probably referenced this issue:

https://github.com/urllib3/urllib3/issues/2564 and the linked fix https://github.com/urllib3/urllib3/pull/2613 

Hiroshi
Top achievements
Rank 1
Iron
commented on 10 Jul 2022, 10:33 PM

Thank you, Nick. I see that change is already included in 1.26.10, but I still cannot make HTTPS call via Fiddler HTTPS proxy. This might be a different issue (of urllib3).
Hiroshi
Top achievements
Rank 1
Iron
commented on 30 Oct 2022, 06:03 PM

I tested the scenario today and it works! I have no idea why it is recovered now.

My test environment today:

  • Windows 10 (21H2)
  • Fiddler Classic v5.0.20211.51073 for .NET 4.6.1
  • Python 3.7.5
  • requests module 2.28.1
  • urllib3 module 1.26.12 (latest) and also 1.26.10 (force downgraded)

In case anyone interested in, below is the exact code I tested. I used virtualenv to install only minimum modules (requests and its dependencies). I added verify=False in order to separate the discussion of SSL cert, which is unrelated to this issue.

import requests
url = 'https://cnn.com/'
proxies = {
   'http': 'http://127.0.0.1:8888',
   'https': 'http://127.0.0.1:8888'
}
response = requests.post(url, proxies=proxies, verify=False)
print(response)

0
Noob Saibot
Top achievements
Rank 1
Iron
answered on 30 Oct 2022, 06:18 AM | edited on 06 Nov 2022, 04:20 PM

It took me like whole day to figure everything out to be able to sniff Python apps with Fiddler. It was hard mostly because i don't have any background on programming.

1. Use following package to allow python to use windows certificates store
https://pypi.org/project/pip-system-certs/

2. Launch Fiddler and uncheck Capture - Very important because i think this is where the problem lies. Python just doesn't work while WinINET proxy is active. I was curious and updated my urllib3 to 1.26.12 because of the comments above mentioning fix . Still didn't work.

Though i still think problem lies in Python because Charles proxy/Python also doesn't work while proxy is active.

3. Open command prompt and set proxy environment variables. Don't close the window. We're going to use this window to launch python app.

set http_proxy=127.0.0.1:8888
set https_proxy=127.0.0.1:8888

4. Launch python app via the command prompt window above

C:\Users\Admin\TEMP\Python2ExeApp.exe

py C:\temp\pythonvirus.pyc

python C:\Users\Admin\TEMP\HelloWorld.py

etc


Tags
Fiddler Classic Fiddler Everywhere
Asked by
IamMusavaRibica
Top achievements
Rank 1
Answers by
Lini
Telerik team
Hiroshi
Top achievements
Rank 1
Iron
Noob Saibot
Top achievements
Rank 1
Iron
Share this question
or