I want to capture and inspect API traffic. I've already set up a Fiddler reverse proxy that decrypts HTTPS. What is the best way to deal with the many client side devices? I'll commonly generate traffic from Android, iOS and PC browsers but also dedicated devices such as Amazon Echo and IoT.
Here's my current thinking for a simple automated approach. I'll set up a network with two WiFi APs named "DIRECT" (normal) and "PROXY" (Fiddler reverse proxy). For Android and iOS devices, I can create "DIRECT" and "PROXY" settings. Done.
For PC browsers, how can I configure them so when I'm on "PROXY", I use proxy settings but when on any other AP, I DIRECT without tripping over any proxy existing settings? Should I use NetSetMan to change proxy settings based on network?
What about dedicated devices that may not have a proxy setting? Should I deal with them by creating some kind of "PROXY" router settings?
Is there some universal client side solution available such as described by Lawrence in http://www.telerik.com/forums/fiddlercore-as-transparent-proxy
23 Answers, 1 is accepted
Yes, a tool like NetSetMan seems like a fine approach to conditionally setting the system proxy based on which network connection you're using.
Fiddler doesn't support any sort of "transparent proxy" mode; it's probably possible to hack something together using IPTables or the like, but I expect this would be fairly fragile.
One approach you *might* try is using "Automatically Detect Proxy" on the clients, and then install the WPAD/DHCP Fiddler extension: http://deletethis.net/dave/wpadserverfiddlerextension/ or setting the appropriate setting in your WiFi router's configuration.
Regards,
Eric Lawrence
Telerik
Rank 1
Rank 1
In such a configuration, HTTP traffic should work fine, because it will contain a HOST header that tells Fiddler where to send the traffic.
HTTPS traffic gets a little more tricky, because transparent proxying of HTTPS traffic requires that the proxy "somehow" know where the traffic was meant to be sent so it can return a proper certificate. For instance, the setting the fiddler.network.https.SetCNFromSNI preference to true will help, but only works if the client uses SNI in its HTTPS handshake, and it also requires that the client (in this case, probably MITM proxy) accept Fiddler's root certificate.
Stepping back a bit though-- what specifically are you hoping to do with Fiddler in your scenario(s)?
Regards,
Eric Lawrence
Telerik
Rank 1
The big picture is that I want to inspect web API requests and responses. I'm a developer with many different devices. I want to inspect traffic from any of the devices. I don't want to have to configure and unconfigure proxy settings on a host of devices.
My solution is to configure a router, call it "PROXY", that routes device traffic to a transparent proxy. This eliminates the need for proxy changes. In fact, I've been able to do all this with a dd-wrt router and mitmproxy (OS X). However, I much prefer Fiddler over mitmproxy because of Fiddler's superior UI and my personal preference for C# and .Net. So I've been trying to get mitmproxy's system to redirect HTTP(S) to a Windows 10 system running Fiddler. That way I can have the best of all worlds. So far I've been unable to figure out the ip commands to get the traffic from mitmproxy's system to Fiddler system.
The ideal solution is to configure the router to directly send HTTP traffic to Fiddler. However, I haven't been able to figure out how to do that. The fallback is to figure out how to get OS X to redirect to Fiddler. I haven't been able to figure that out either. So I'm stuck using mitmproxy, which while working, is a less than ideal solution.
Today I tried inspecting traffic from an Amazon Echo using mitmproxy. I've been unable to get that combo working. The Echo doesn't have proxy settings so I can't use Fiddler either. I guess I need to look into Wireshark.
Interesting! Can you share information (or a link) about how you configured dd-wrt to do this, and what, if any, special settings you put in MITMProxy? In principle, Fiddler should be able to do anything MITMProxy can do.
You will need to ensure that Fiddler is configured to accept requests on the public interface, using Tools > Fiddler Options > Connections > Allow Remote Computers to Connect, and ensure that the system firewall allows inbound connections to Fiddler's port.
Intercepting HTTPS traffic from an Echo is unlikely to ever work, using any tool (unless you've figured out how to jailbreak it) because it seems very unlikely that Echo exposes any way to trust an interception root certificate.
Regards,
Eric Lawrence
Telerik
Rank 1
Yes, interesting what happens when a noobie assumes something can be done, has time available, and is motivated. Usually the interesting thing turns out badly but in this rare exception, all the experiments have lead to a good payoff.
===
Here's a really easy way to inspect realtime web API traffic. It works for both HTTP and HTTPS (decrypted) traffic. There's no hassling with device proxies. It works with all sorts of client devices including Android, iOS. Linux and Windows. You'll need to setup a router and install transparent proxy software. Specifically, I use dd-wrt router software, and installed mitmproxy.org's transparent proxy on a Mac OS/X system. Probably any router's software will do. There's other choices for transparent proxy software (squid), and other hosting system choices such as Linux. There's articles on the internet which explain, in uber-guru terms, how to install mitmproxy on a dd-wrt router so a host system isn't needed. There's also articles about using a Raspberry PI to as both a router and mitmproxy host.
Client Device Enabling
First, connect your client device to your private proxy network (192.168.3.x) -- just select your wireless SSID. Second, to enable HTTPS decryption, follow your proxy provider's instructions for storing their SSL certificate.
Router Configuration
Let's assume your LAN is 192.168.1.x (usually having a gateway of 192.168.1.1), your transparent proxy host is 192.168.1.2, and you've chosen to create a new LAN (named PROXY) of 192.168.3.x for your client devices. Note that the attached image uses a somewhat unusual gateway of 192.168.1.254 because that's my internet provider's default.
You need to configure the router so that the gateway value points to the transparent router host ip (192.168.1.2). To do so, assign a static IP address for the WAN. Use an ip which is outside of the LAN's DHCP range, often 192.168.1.201 is good. Router's WAN automatic DHCP assignment can't be used because it will also assign the default gateway (usually 192.168.1.1), which in this case won't be the desired proxy host address (192.168.1.2). Hence we are using the router's static IP WAN option to assign a WAN address. Attached is a screen shot image of my dd-wrt setup. The image shows a static IP of 192.168.1.201, a gateway of 192.168.1.2, some google DNS addresses. The router will use a private network of 192.168.3.1.
Transparent Proxy Installation
I chose to install mitmproxy, a transparent proxy, on an OS/X system. There are other choices for proxy and system. You must use a proxy that has a transparent mode, otherwise this configuration won't work. Charles and Fiddler offer reverse proxy but NOT transparent proxy. Follow your proxy provider's installation instructions.
How to Execute mitmproxy on OS/X
#!/bin/sh# Mac OS/X shell script to start up mitmproxy transparent proxy.# sudo enables the subsequent command to run in root. You may need to enter the root password.# Enable packet forwardingsudo sysctl -w net.inet.ip.forwarding=1# create a file, pf.conf, to contain firewall rules. Be sure the interface name is the correct one (en0? en2?). My en0 interface handles the 192.168.1.x network.echo 'rdr on en0 inet proto tcp to any port 80 -> 127.0.0.1 port 8080' > pf.confecho 'rdr on en0 inet proto tcp to any port 443 -> 127.0.0.1 port 8080' >> pf.conf# Execute commands to setup mitmproxy execution environmentsudo pfctl -f pf.confsudo pfctl -e# Specify the path to mitmproxy. In my case the path is ~/Downloads/osx-mitmproxy-0.12.1# Note Bene: On first execution of mitmproxy, it will create a .mitmproxy directory containing configuration settings.~/Downloads/osx-mitmproxy-0.12.1/mitmproxy -T --host
Rank 1
Fiddler itself should work similarly to mitmproxy for this scenario, with the caveat that to get HTTPS interception working you need to instruct Fiddler to start a new secure listener on a different port (e.g. 8889).
The more challenging bit is this script you write:
echo 'rdr on en0 inet proto tcp to any port 80 -> 127.0.0.1 port 8080' > pf.confecho 'rdr on en0 inet proto tcp to any port 443 -> 127.0.0.1 port 8080' >> pf.confI don't know how you'd do the equivalent of that on Windows, although I'm sure there's some mechanism to do something similar.
Regards,
Eric Lawrence
Telerik
Rank 1
Regards,
Eric Lawrence
Telerik
Rank 1
I tried sending an email using "Send Feedback" in Fiddler's Help menu. It launched Outlook and wanted me to setup an email account. I really don't want to do that and have always avoided using Outlook. Is there an alternate method of contacting you, an email address?
I've spent a few more hours trying to get this to work. No progress. The issue may be more of a Windows issue than Fiddler, I just don't know. If you'd like to help, here's a simple setup which isn't working. If you can get it to work, perhaps that's all that's needed. It works great with OS/X and mitmproxy.
1. Setup a router whose gateway points to a Windows system.
2. Launch some listener program on the Windows system such as Fiddler. Listen on port 8888.
3. Connect a computer to the router. In the computer's browser type a url using port 8888 (e.g. www.example.com:8888). The HTTP GET request should be heard and displayed by the listener (Fiddler) on the gateway system.
4. Now try again using port 80, then port 443. If it works, all's done.
I've been unable to get this simple setup to work. Windows doesn't seem to pass on the GET to the listener.
The link in question simply launches the hyperlink mailto:fiddler@telerik.com, which means that your computer, for whatever reason, is not configured to support mailto links. You can use whatever mail program you use to send us mail instead.
With regard to your question below: *without* doing the trick with the router, is the client computer able to send traffic to http://FiddlerPC:8888/ directly?
Regards,
Eric Lawrence
Telerik
Rank 1
In the client browser, when I enter http://FiddlerPC:8888/, Fiddler reports empty GETs, which I assume is the correct.
This happens when the client is on the same subnet AND also when the client is on the router's own subnet.
What's the next thing to try?
That's a good sign, as it means there's not a problem with the Windows firewall or the port configuration on the windows machine.
One question: Have you tried requesting a URL with an IP address instead of a hostname? If the client is attempting to use the Windows machine as a gateway for DNS, that will fail (unless you find some means to get that machine to serve DNS)
I'll give this a try on my router at home this weekend and see if I can figure out what's afoot.
Regards,
Eric Lawrence
Telerik
Rank 1
For systems connected to the proxy router, where the router uses the Fiddler system as the gateway, DNS will work if you enable IP forwarding on the gateway/Fiddler system following a reboot. Follow this registry change:
https://support.microsoft.com/en-us/kb/99686
As you surmised above, the trick is to get the gateway/Fiddler system to respond to traffic forwarded to it from the router.
I'm not sure how to interpret your message though-- are you saying that you performed the steps in the KB article to ensure DNS flowed correctly and now the transparent proxy functions properly, or are you saying that even after doing the routing for DNS the transparent configuration still is not working?
thanks,
Eric Lawrence
Telerik
Rank 1
The latter. While DNS is working, transparent is not. Fiddler is not displaying any HTTP traffic gateway-ed from the router. From the client, connected to the router, requesting http://example.com displays (in browser) the correct response but nothing is ever displayed by Fiddler (although http://example.com:8888 works, 80 does not). It appears that Fiddler never hears a HTTP port 80 request. I've blindly tried various NETSH commands, trying to redirect from port 80 to 8888, similar to OS/X, but nothing's working. I really don't understand what NETSH commands are required to get Fiddler to see the HTTP traffic or even if NETSH is the correct way to redirect. I'm hoping you can figure out what possibly is the last hurdle.
Something like NetSh's addv4tov4 command seems like it could mimic that, but I don't know that it would help if the machine already has IIS or whatever listening on port 80. https://technet.microsoft.com/en-us/library/cc776297(WS.10).aspx#BKMK_1
The same goes for NetCat: http://serverfault.com/questions/332217/using-netcat-in-windows-to-forward-a-tcp-door-to-another-machine
Alternatively, it seems like DD-WRT might be able to do this itself if you use Port Forwarding and turn off the Filter WAN NAT Redirection option. But this is all pretty far outside my area of expertise. I wonder if someone on ServerFault.com knows an easy way to accomplish this task, as it doesn't really seem that crazy.
Regards,
Eric Lawrence
Telerik
Rank 1
I've spent more hours but haven't made any progress. Here's where I'm stuck. In a proxy router's client's browser, if I type "http://fiddlersystem" I see the request coming into the gateway/Fiddler(proxy set to port 80) system. If I type a website, e.g. "http://httpbin.org", I don't see anything coming into Fiddler, Charles, or any other listener however the proper response is received. So it seems that Windows is not passing the http request through to Fiddler.
Note that the above works perfectly using OS/X with mitmproxy.
Rank 1
Not really sure what to tell you-- Fiddler simply shows the traffic that's sent to it.
If it is running on port 80 (and nothing else is) and the firewall allows inbound connections, there's no explanation as to why you wouldn't see the traffic, or, at worst, an error message of some sort.
There's nothing in Windows itself that would forward your traffic to a target website ("the proper response is received") in the absence of a proxy like Charles or Fiddler.
Regards,
Eric Lawrence
Telerik
Rank 1