Home » Python » Capturing HTTP requests using BrowserMobProxy

About Soumyajit Basu

Soumyajit Basu
Soumyajit is a QA/DevOps engineer by profession and a technology enthusiast by passion. He loves communicating about technology and is an author at his own blog platform as well as in Dzone and Web Code Geeks.

Capturing HTTP requests using BrowserMobProxy

1. BrowserMob Proxy

Browsermob proxy is a developers utility that helps to monitor network traffic of a web application. It is a technology that is developed under the Selenium open source project. It is originally called andBrowser Mob is now a part of Neustar.

A proxy is an (Apache 2.0 license) utility that can be used with Selenium or otherwise can be used independently as well.

Browsermob allows you to manipulate HTTP requests and response, capture the HTTP content and export the data as a HAR object, also known as HTTP archive.

2. Configuring BrowserMob proxy with selenium

Configuring BrowserMob proxy for the automation process will run your selenium scripts behind a proxy server. To configure browser-mob proxy you would need to do the following:

2.1 Browsermob Proxy library

Get the browser-mob proxy library from here. You are going to need this to create the proxy server. Download it and keep a copy of the browser-mob proxy in the project directory.

2.2 Python dependency for BrowserMob proxy

Use pip to install python dependency for the browser-mob proxy. To install the dependency just type in pip install browsermob-proxy.

2.3 Creating a server and instantiating a proxy instance

First of all, you need to create a server. Browsermob already has the configuration for Jetty server. You have to instantiate the server by giving the path for browser mob-proxy inside the browser mob-proxy/bin directory. By default, the Jetty server would be running on port 8080. The jetty server is a Java based Web Server with a servlet container. It helps in addressing requests and responses through the servlet container.

Instantiating a browser-mob server

# Configure proxy server and return server object
def configure_proxy_server(self):
    server = Server("browsermob-proxy/bin/browsermob-proxy")
    return server

2.4 Instantiating a proxy from the server

Once the Jetty server is up and running, next a proxy needs to be created using the server instance of Jetty.

Creating a proxy

# Configure proxy using the server object and return proxy object
def configure_proxy(self, server):
  server.start()
  proxy = server.create_proxy()
  return proxy

2.4.1 Configuring browser driver with proxy server configurations

Pass on the proxy configuration to the respective web driver object, so that it starts using the proxy of the jetty server of Browsermob rather than the proxy of the selenium server.

Configuring Chrome driver with proxy server configuration

# Configure selenium driver and return driver object
def configure_driver(self, proxy_data):
  try:
    co = webdriver.ChromeOptions()
    co.add_argument('--proxy-server={host}:{port}'.format(host='localhost', 
                                                          port=proxy_data.port))
    driver = webdriver.Chrome(executable_path = "/usr/local/bin/chromedriver", 
                                                          chrome_options=co)
    return driver
  except WebDriverException as e:
    print(">>>WebDriver exception: {}".format(e))

2.5 Storing the HTTP requests as HAR object

The outgoing HTTP requests from the browser can be stored in the HAR or Http Archive format. At the end of the execution, you can iterate the HAR object to get the HTTP requests. Also, you can also store the respective data in a .har file. Once the file.har is created you can use Google GSuite Analyzer to track the respective performance or monitor the site.

Assembling the HTTP requests and storing it as har object

# Track network traffic
def get_networkTraffic(self, server, proxy_data, driver):
    try:
        input_url = raw_input(">>>Enter the url to be checked for network traffic: ")
        request = requests.get(input_url)
        response_code = request.status_code
        # Store the http requests as har object
        if response_code == 200:
            proxy_data.new_har("google")
            driver.get(input_url)
            proxy_data.har
            for ent in proxy_data.har['log']['entries']:
                print(ent['request']['url'])
        else:
            print(">>>Status code: "+str(response_code))
        driver.quit()
        server.stop()
    except requests.exceptions.RequestException as e:
        print(">>>Request Exception: {}".format(e))

3. SSL Support

You have to explicitly set up the ca-certificate-rsa.cer for your respective browser. I have set up the certificate for Chrome browser. Without this the requests cannot be made over secure socket layer. To set up the certificate just open chrome browser goto settings > advanced > Privacy & Security > Manage Certificates. If you are working on OSX platform, you will get the KeyChain Access window. Just add the ca-certificate-rsa.cer. Once added, just click on ‘Trust’ and select the option ‘Always Trust’ and save the configuration by giving the root password. You will be able to find the ca-certificate-rsa.cer within the repository itself.

4. Download the Source Code

This was an example of Capturing HTTP requests using BrowserMobProxy.

Download
You can download the full source code of this example here : Using BrowserMobProxy with Selenium

Do you want to know how to develop your skillset to become a Web Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

 

1. Building web apps with Node.js

2. HTML5 Programming Cookbook

3. CSS Programming Cookbook

4. AngularJS Programming Cookbook

5. jQuery Programming Cookbook

6. Bootstrap Programming Cookbook

 

and many more ....

 

 

One comment

  1. Why do you not use fiddler.?

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Want to take your WEB dev skills to the next level?

Grab our programming books for FREE!

Here are some of the eBooks you will get:

  • PHP Programming Cookbook
  • jQuery Programming Cookbook
  • Bootstrap Programming Cookbook
  • Building WEB Apps with Node.js
  • CSS Programming Cookbook
  • HTML5 Programming Cookbook
  • AngularJS Programming Cookbook