Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

How do I configure python to proxy through BurpSuite for https?

Oscar | Last updated: Feb 03, 2017 07:17PM UTC

I have BurpSuite configured as a proxy. I have generated and installed a certificate for Burp in my Mac's keychain. On a Kali box I have also configured Iceweasel to proxy through my Burp Proxy running on my Mac. I can can browse from Kali to both http and https sites and burp intercepts the requests. I forward them through and confirm the security exceptions and the browser eventually displays web pages. From the Kali command line I exported the following two variables export http_proxy=http://172.xx.yy.z:8081 export https_proxy=http://172.xx.yy.z:8081 I wrote one python script: (env) ojblass@kali:~/effective_python$ cat ojblass_urllib_http.py import urllib url = urllib.urlopen("http://www.hotmail.com") data = url.read() print data This correctly hits the BurpSuite proxy; however, when I attempt to run https traffic through the request is not intercepted: (env) ojblass@kali:~/effective_python$ cat ojblass_urllib_https.py import urllib url = urllib.urlopen("https://mail.live.com/default.aspx") data = url.read() print data Instead the following error is printed: <html><head><title>Burp Suite Professional</title> <style type="text/css"> body { background: #dedede; font-family: Arial, sans-serif; color: #404042; -webkit-font-smoothing: antialiased; } #container { padding: 0 15px; margin: 10px auto; background-color: #ffffff; } a { word-wrap: break-word; } a:link, a:visited { color: #e06228; text-decoration: none; } a:hover, a:active { color: #404042; text-decoration: underline; } h1 { font-size: 1.6em; line-height: 1.2em; font-weight: normal; color: #404042; } h2 { font-size: 1.3em; line-height: 1.2em; padding: 0; margin: 0.8em 0 0.3em 0; font-weight: normal; color: #404042;} .title, .navbar { color: #ffffff; background: #e06228; padding: 10px 15px; margin: 0 -15px 10px -15px; overflow: hidden; } .title h1 { color: #ffffff; padding: 0; margin: 0; font-size: 1.8em; } div.navbar {position: absolute; top: 18px; right: 25px;}div.navbar ul {list-style-type: none; margin: 0; padding: 0;} div.navbar li {display: inline; margi-left: 20px;} div.navbar a {color: white; padding: 10px} div.navbar a:hover, div.navbar a:active {text-decoration: none; background: #404042;} </style> </head> <body> <div id="container"> <div class="title"><h1>Burp Suite Professional</h1></div> <h1>Error</h1><p>Invalid&#32;client&#32;request&#32;received&#58;&#32;First&#32;line&#32;of&#32;request&#32;did&#32;not&#32;contain&#32;an&#32;absolute&#32;URL&#32;&#45;&#32;try&#32;enabling&#32;invisible&#32;proxy&#32;support&#46;</p> <div class="request">GET&nbsp;https://mail.live.com/default.aspx&nbsp;HTTP/1.0<br> User-Agent:&nbsp;Python-urllib/1.17<br> Accept:&nbsp;*/*<br> <br> </div><p>&nbsp;</p> </div> </body> </html> The main content of the error states 'Invalid client request received. First line of request did not contain an absolute URL. I have googled this error but everything I seem to come does not appear relevant. I am new to this and would appreciate any insight you might have.

Liam, PortSwigger Agent | Last updated: Feb 06, 2017 09:46AM UTC

Hi Oscar Thanks for your message. I noticed that you found a workaround and posted it on stack exchange: - http://security.stackexchange.com/questions/150329/proxying-requests-through-python-and-burpsuite-not-working Are you now able to use Burp in the required manner? Please let us know if you need any further assistance.

Burp User | Last updated: Mar 27, 2019 01:06PM UTC

https://www.th3r3p0.com/random/python-requests-and-burp-suite.html Home Python Requests and Burp Suite Problem: When I am conducting a pentest, I commonly write python scripts to use the requests module and need to proxy them through Burp. I have been using the "Easy way out," but there are problems with doing this and there is a much more efficient way in handling this. Easy way out: I can proxy requests through Burp Suite fairly easily by creating a proxies dictionary and assigning that dictionary to the proxies argument. I then have to set the verify argument to False because Burp's certificate is not trusted by the requests library's certificate bundle. Example code: import requests proxies = {"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"} r = requests.get("https://www.google.com/", proxies=proxies, verify=False) Problem with easy way out: What happens if you have many calls to the requests library and you don't want to set the proxies and verify arguments for each request. Or possibly you have been given a test harness that utilizes the requests library and you don't want to modify each and every call to the library. I have always searched for this answer and only found that I can export two environment variables HTTP_PROXY and HTTPS_PROXY. However, this does not fix the fact that I have to set the verify argument to False on every single request. Solution: In addition to the HTTP_PROXY and HTTPS_PROXY environment variables, there is also a REQUESTS_CA_BUNDLE which can be set to specify the location of a certificate. However, the documentation is not very clear about the certificate format required. After some basic troubleshooting, I was able to determine the encoding needed for the REQUESTS_CA_BUNDLE file is PEM. After you have downloaded your certificate from Burp (either through the browser or directly from the application's GUI), it is DER formatted. In order to convert it to the needed PEM encoded format, run the following command: openssl x509 -inform der -in certificate.cer -out certificate.pem You are now ready to export your environment variables and use requests with Burp. export REQUESTS_CA_BUNDLE="/path/to/pem/encoded/cert" export HTTP_PROXY="http://127.0.0.1:8080" export HTTPS_PROXY="http://127.0.0.1:8080" Now all of your HTTP requests made through the requests library without the proxies argument configured will be routed through Burp. In order to remove these environment variables, run the following commands: unset REQUESTS_CA_BUNDLE unset HTTP_PROXY unset HTTPS_PROXY

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.