Best Way to Inspect HTTP(S) APIs of Many Devices

24 posts, 0 answers
  1. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 08 Jul 2015 Link to this post

    I want to capture and inspect API traffic. I've already set up a Fiddler reverse proxy that decrypts HTTPS. What is the best way to deal with the many client side devices? I'll commonly generate traffic from Android, iOS and PC browsers but also dedicated devices such as Amazon Echo and IoT.

    Here's my current thinking for a simple automated approach. I'll set up a network with two WiFi APs named "DIRECT" (normal) and "PROXY" (Fiddler reverse proxy). For Android and iOS devices, I can create "DIRECT" and "PROXY" settings. Done.

    For PC browsers, how can I configure them so when I'm on "PROXY", I use proxy settings but when on any other AP, I DIRECT without tripping over any proxy existing settings? Should I use NetSetMan to change proxy settings based on network?

    What about dedicated devices that may not have a proxy setting? Should I deal with them by creating some kind of "PROXY" router settings?

    Is there some universal client side solution available such as described by Lawrence in http://www.telerik.com/forums/fiddlercore-as-transparent-proxy

     

     

  2. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 13 Jul 2015 Link to this post

    Hi, Robert--

    Yes, a tool like NetSetMan seems like a fine approach to conditionally setting the system proxy based on which network connection you're using.

    Fiddler doesn't support any sort of "transparent proxy" mode; it's probably possible to hack something together using IPTables or the like, but I expect this would be fairly fragile.

    One approach you *might* try is using "Automatically Detect Proxy" on the clients, and then install the WPAD/DHCP Fiddler extension: http://deletethis.net/dave/wpadserverfiddlerextension/ or setting the appropriate setting in your WiFi router's configuration.

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  3. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 14 Jul 2015 in reply to Eric Lawrence Link to this post

    I have a transparent proxy, mitmproxy, working sufficient well for API inspection. I would like to be able to use Fiddler's superior UI and my .Net skills. Would fiddler work if I redirected http(s) traffic to it from the transparent proxy. I'm assuming I can figure out the iptables issue. Will Fiddler care if the http(s) traffic is marshaled from a transparent proxy or would something be missing?
  4. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 14 Jul 2015 in reply to Robert Link to this post

    Restating above, assuming I can find a method to change iptables on the transparent proxy's system to redirect http(s) traffic to the fiddler system, will fiddler work by simply having redirecting traffic or is there some other magic needed?

  5. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 14 Jul 2015 Link to this post

    Hi, Robert--

    In such a configuration, HTTP traffic should work fine, because it will contain a HOST header that tells Fiddler where to send the traffic. 

    HTTPS traffic gets a little more tricky, because transparent proxying of HTTPS traffic requires that the proxy "somehow" know where the traffic was meant to be sent so it can return a proper certificate.  For instance, the setting the fiddler.network.https.SetCNFromSNI preference to true will help, but only works if the client uses SNI in its HTTPS handshake, and it also requires that the client (in this case, probably MITM proxy) accept Fiddler's root certificate.

    Stepping back a bit though-- what specifically are you hoping to do with Fiddler in your scenario(s)?

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  6. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 14 Jul 2015 in reply to Eric Lawrence Link to this post

    The big picture is that I want to inspect web API requests and responses. I'm a developer with many different devices. I want to inspect traffic from any of the devices. I don't want to have to configure and unconfigure proxy settings on a host of devices.

    My solution is to ​configure a router, call it "PROXY", that routes device traffic to a transparent proxy. This eliminates the need for proxy changes. In fact, I've been able to do all this with a dd-wrt router and mitmproxy (OS X). However, I ​much prefer Fiddler over mitmproxy because of Fiddler's superior UI and my personal preference for C# and .Net. So I've been trying to get mitmproxy's system to redirect HTTP(S) to a Windows 10 system running Fiddler. That way I can have the best of all worlds. So far I've been unable to figure out the ip commands to get the traffic from mitmproxy's system to Fiddler system.

    The ideal solution is to configure the router to directly send HTTP traffic to Fiddler. However, I haven't been able to figure out how to do that. The fallback is to figure out how to ​get OS X to redirect to Fiddler. I haven't been able to figure that out either. So I'm stuck using mitmproxy, which while working, is a less than ideal solution.

    Today I tried inspecting traffic from an Amazon Echo using mitmproxy. I've been unable to get that combo working. The Echo doesn't have proxy settings so I can't use Fiddler either. I guess I need to look into Wireshark.

  7. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 15 Jul 2015 Link to this post

    Hi, Robert--

    Interesting!  Can you share information (or a link) about how you configured dd-wrt to do this, and what, if any, special settings you put in MITMProxy? In principle, Fiddler should be able to do anything MITMProxy can do.

    You will need to ensure that Fiddler is configured to accept requests on the public interface, using Tools > Fiddler Options > Connections > Allow Remote Computers to Connect, and ensure that the system firewall allows inbound connections to Fiddler's port.

    Intercepting HTTPS traffic from an Echo is unlikely to ever work, using any tool (unless you've figured out how to jailbreak it) because it seems very unlikely that Echo exposes any way to trust an interception root certificate. 

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  8. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 16 Jul 2015 in reply to Eric Lawrence Link to this post

    Yes, interesting what happens when a noobie assumes something can be done, has time available, and ​is motivated. Usually the interesting thing turns out badly but in this rare ​exception, all the experiments have ​lead to a good payoff.

    ===

    Here's a really easy way to inspect realtime web API traffic. It works for both HTTP and HTTPS (decrypted) traffic. There's no hassling with device proxies. It works with all sorts of client devices including Android, iOS. Linux and Windows. You'll need to setup a router and install transparent proxy software. Specifically, I use dd-wrt router software, and installed mitmproxy.org's transparent proxy on a Mac OS/X system. Probably any router's software will do. There's other choices for transparent proxy software (squid), and other hosting system choices such as Linux. There's articles on the internet which explain, in uber-guru terms, how to install mitmproxy on a dd-wrt router so a host system isn't needed. There's also articles about using a Raspberry PI to as both a router and mitmproxy host.

    Client Device Enabling

    First, connect your client device to your private proxy network (192.168.3.x) -- just select your wireless SSID. Second, to enable HTTPS decryption, follow your proxy provider's instructions for storing their SSL certificate.

    Router Configuration

    Let's assume your LAN is 192.168.1.x (usually having a gateway ​of 192.168.1.1), your transparent proxy host is 192.168.1.2, and you've chosen to create a new LAN (named PROXY) of 192.168.3.x for your client devices. Note that the attached image uses a somewhat unusual gateway of 192.168.1.254 because that's my internet provider's default.

    You need to configure the router so that the gateway value points to the transparent router host ip (192.168.1.2). To do so, assign a static IP address for the WAN. Use an ip which is outside of the LAN's DHCP range, often 192.168.1.201 is good. Router's WAN automatic DHCP assignment can't be used because it will also assign the ​default gateway (usually 192.168.1.1), which in this case won't be the desired proxy host address (192.168.1.2). Hence we are using the router's static IP WAN option to assign a WAN address. Attached is a screen shot image of my dd-wrt setup. The image shows a static IP of 192.168.1.201, a gateway of 192.168.1.2, some google DNS addresses. The router will use a private network of 192.168.3.1.

    Transparent Proxy Installation

    I chose to install mitmproxy, a transparent proxy, on an OS/X system. There are other choices for proxy and system. You must use a proxy that has a transparent mode, otherwise this configuration won't work. Charles and Fiddler offer reverse proxy but NOT transparent proxy. Follow your proxy provider's installation instructions.

    How to Execute mitmproxy on OS/X

     

    #!/bin/sh
    # Mac OS/X shell script to start up mitmproxy transparent proxy.
    # sudo enables the subsequent command to run in root. You may need to enter the root password.
    # Enable packet forwarding
    sudo sysctl -w net.inet.ip.forwarding=1
    # create a file, pf.conf, to contain firewall rules. Be sure the interface name is the correct one (en0? en2?). My en0 interface handles the 192.168.1.x network.
    echo 'rdr on en0 inet proto tcp to any port 80 -> 127.0.0.1 port 8080' > pf.conf
    echo 'rdr on en0 inet proto tcp to any port 443 -> 127.0.0.1 port 8080' >> pf.conf
    # Execute commands to setup mitmproxy execution environment
    sudo pfctl -f pf.conf
    sudo pfctl -e
    # Specify the path to mitmproxy. In my case the path is ~/Downloads/osx-mitmproxy-0.12.1
    # Note Bene: On first execution of mitmproxy, it will create a .mitmproxy directory containing configuration settings.
    ~/Downloads/osx-mitmproxy-0.12.1/mitmproxy -T --host

  9. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 16 Jul 2015 in reply to Robert Link to this post

    Hope my description of router configuration and mitmproxy setup leads to an equivalent feature for Fiddler on Windows. As you can see, it's dead simple to ​ready new devices ​for inspection. The missing link is to be able to use Fiddler, Windows and leverage .Net skills.
  10. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 17 Jul 2015 Link to this post

    Interesting, thanks for sharing these instructions.

    Fiddler itself should work similarly to mitmproxy for this scenario, with the caveat that to get HTTPS interception working you need to instruct Fiddler to start a new secure listener on a different port (e.g. 8889).

    The more challenging bit is this script you write:

    echo 'rdr on en0 inet proto tcp to any port 80 -> 127.0.0.1 port 8080' > pf.conf
    echo 'rdr on en0 inet proto tcp to any port 443 -> 127.0.0.1 port 8080' >> pf.conf

    I don't know how you'd do the equivalent of that on Windows, although I'm sure there's some mechanism to do something similar. 

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  11. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 20 Jul 2015 in reply to Eric Lawrence Link to this post

    I've spent hours on trying to get transparent proxy to work on Fiddler but to no avail. Maybe there's someone out there who's more clever and can figure it out. I think it's a pretty significant capability. Until then, I'll just have to use mitmproxy in most cases but can use Fiddler for Windows apps.
  12. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 20 Jul 2015 Link to this post

    Hello, Robert-- Please do feel free to send me mail using the "Send Feedback" link in Fiddler's help menu. If you can explain more about what you've tried thus far and what results you got, I may be able to help get this going for you.

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  13. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 23 Jul 2015 in reply to Eric Lawrence Link to this post

     I tried sending an email using "Send Feedback" in Fiddler's Help menu. It ​launched Outlook and wanted me to setup an email account. I really don't want to do that and have always avoided using Outlook. Is there an alternate method of contacting you, an email address?

     

    I've spent a few more hours trying to get this to work. No progress. The issue may be more of a Windows issue than Fiddler, I just don't know. If you'd like to help, here's a simple setup which isn't working. If you can get it to work, perhaps that's all that's needed. It works great with OS/X and mitmproxy.

    1. Setup a router whose gateway points to a Windows system.

    2. Launch some listener program on the Windows system such as Fiddler. Listen on port 8888.

    3. Connect a ​computer to the router. In the computer's browser type a url using port 8888 (e.g. www.example.com:8888). The HTTP GET request should be heard and displayed by the listener (Fiddler) on the gateway system.

    4. Now try again using port 80, then port 443. If it works, all's done.

    I've been unable to get this simple setup to work. Windows doesn't seem to pass on the GET to the listener.

  14. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 23 Jul 2015 Link to this post

    Hi, Bob--

    The link in question simply launches the hyperlink mailto:fiddler@telerik.com, which means that your computer, for whatever reason, is not configured to support mailto links. You can use whatever mail program you use to send us mail instead.

    With regard to your question below: *without* doing the trick with the router, is the client computer able to send traffic to http://FiddlerPC:8888/ directly?

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  15. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 23 Jul 2015 in reply to Eric Lawrence Link to this post

    In the client browser, when I enter http://FiddlerPC:8888/, Fiddler reports empty GETs, which I assume is the correct.

    This happens when the client is on the same subnet ​AND also when the client is on the router's own subnet.

    What's the next thing to try?

  16. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 24 Jul 2015 Link to this post

    Hi, Bob--

    That's a good sign, as it means there's not a problem with the Windows firewall or the port configuration on the windows machine.

    One question: Have you tried requesting a URL with an IP address instead of a hostname? If the client is attempting to use the Windows machine as a gateway for DNS, that will fail (unless you find some means to get that machine to serve DNS)

    I'll give this a try on my router at home this weekend and see if I can figure out what's afoot.


    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  17. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 25 Jul 2015 in reply to Eric Lawrence Link to this post

    For systems connected to the proxy router, where the router uses the Fiddler system as the gateway, DNS will work if you enable IP forwarding on the gateway/Fiddler system following a reboot. Follow this registry change:

    https://support.microsoft.com/en-us/kb/99686

    As you surmised above, the trick is to get the gateway/Fiddler system to respond to traffic forwarded to it from the router.

  18. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 27 Jul 2015 Link to this post

    Thanks for the great link!

    I'm not sure how to interpret your message though-- are you saying that you performed the steps in the KB article to ensure DNS flowed correctly and now the transparent proxy functions properly, or are you saying that even after doing the routing for DNS the transparent configuration still is not working?

    thanks,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  19. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 27 Jul 2015 in reply to Eric Lawrence Link to this post

    The latter. While DNS is working, transparent is not. Fiddler is not displaying any HTTP traffic gateway-ed from the router. ​From the client, connected to the router, requesting http://example.com ​displays (in browser) the correct response but nothing is ever displayed by Fiddler (although http://example.com:8888 works, 80 does not). It appears that Fiddler never hears a ​HTTP port 80 request. I've blindly tried various NETSH commands, trying to redirect from port 80 to 8888, ​similar to OS/X, but nothing's working. I really don't understand what NETSH commands are required to get Fiddler to see the HTTP traffic or even if NETSH is the correct way to redirect. I'm hoping you can figure out what possibly is the last hurdle.

  20. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 28 Jul 2015 Link to this post

    Ah. Would I be correct in guessing that you've already got another web server listening on port 80 and thus can't simply move Fiddler over to that port? That seems like the easiest approach.

    Something like NetSh's addv4tov4 command seems like it could mimic that, but I don't know that it would help if the machine already has IIS or whatever listening on port 80. https://technet.microsoft.com/en-us/library/cc776297(WS.10).aspx#BKMK_1

    The same goes for NetCat: http://serverfault.com/questions/332217/using-netcat-in-windows-to-forward-a-tcp-door-to-another-machine

    Alternatively, it seems like DD-WRT might be able to do this itself if you use Port Forwarding and turn off the Filter WAN NAT Redirection option. But this is all pretty far outside my area of expertise. I wonder if someone on ServerFault.com knows an easy way to accomplish this task, as it doesn't really seem that crazy.

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  21. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 01 Aug 2015 in reply to Eric Lawrence Link to this post

    I've spent more hours but haven't made any progress. Here's where I'm stuck. In a proxy router's client's browser, if I type "http://fiddlersystem" I see the request coming into the gateway/Fiddler(proxy set to port 80) system. If I type a website, e.g. "http://httpbin.org", I don't see anything coming into Fiddler, Charles, or any other listener however the proper response is received. So it seems that Windows ​is not passing the http request through to Fiddler.

     Note that the above works perfectly using OS/X with mitmproxy.

  22. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 02 Aug 2015 in reply to Robert Link to this post

    Eureka moment. I decided to ​try mitmdump on Windows to see if it works in transparent mode. Indeed it does, just as expected. So it seems that all the Fiddler/Charles transparent proxy issues I'm having are not OS or hardware config issues. This is definitely an issue with Fiddler/Charles simply not working in transparent mode, at least not as well as mitmdump.
  23. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 03 Aug 2015 Link to this post

    Hello, Bob--

    Not really sure what to tell you-- Fiddler simply shows the traffic that's sent to it.

    If it is running on port 80 (and nothing else is) and the firewall allows inbound connections, there's no explanation as to why you wouldn't see the traffic, or, at worst, an error message of some sort.

    There's nothing in Windows itself that would forward your traffic to a target website ("the proper response is received") in the absence of a proxy like Charles or Fiddler.

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
  24. Robert
    Robert avatar
    23 posts
    Member since:
    Dec 2011

    Posted 04 Aug 2015 in reply to Eric Lawrence Link to this post

    I can't read this any other way. mitmdump sees the traffic and Charles and Fiddler do not. I can only conclude that transparent proxy is a feature that's not supported by Charles or Fiddler and is not any kind of config issue. This is too bad as Fiddler is the most powerful of the lot. It just lacks this one feature, transparent proxy, ​which is way cool and useful. I've spent over 20 hours trying to get transparent to work. I have to see it end this way.
Back to Top