Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

addToSiteMap(IHttpRequestResponse item) generates redundant requests

Yaroslav | Last updated: May 27, 2020 05:34PM UTC

Hello, Could you advise how to fix the the problem of the duplicate requests in the site map? While adding of the 1 request "GET /fatiz1997/site-map-extension?_pjax=%23js-repo-pjax-container HTTP/1.1" Burp generates automatically 3 additional requests, so in the result there are 4 requests in the site map: 1) GET / HTTP/1.1 2) GET /fatiz1997 HTTP/1.1 3) GET /fatiz1997/site-map-extension HTTP/1.1 4) GET /fatiz1997/site-map-extension?_pjax=%23js-repo-pjax-container HTTP/1.1 But, this is not exactly my problem. The problem begins when multiple requests are adding. Auto-generated additional requests duplicate. So, I have that getSiteMap(String prefix) method returns array IHttpRequestResponse[] with duplicates. For example, array contains 2 or more instances of the "GET /fatiz1997 HTTP/1.1" request/" Please, advice how to avoid duplicates

Uthman, PortSwigger Agent | Last updated: May 28, 2020 12:54PM UTC

Can you share the full code for context, please?

Yaroslav | Last updated: May 28, 2020 05:21PM UTC

Hi Uthman, please refer to the GitHub for code. (https://github.com/fatiz1997/burpbugaddsitemap) the output after setup of the shared extension is below: After 1st request amount of IHttpRequestResponse objects in site map for github.com is: 2 -------------0------------- GET / HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== -------------1------------- GET /notifications HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== After 2nd request amount of IHttpRequestResponse objects in site map for github.com is: 4 -------------0------------- GET / HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== -------------1------------- GET /notifications HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== -------------2------------- GET /notifications HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== -------------3------------- GET /notifications/beta HTTP/1.1 Host: github.com Accept-Encoding: gzip, deflate Accept: */* Accept-Language: en User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36 Connection: close =========================== Please, find the duplicates after 2nd request has been added (duplicates have 1, 2 indexes) in above output . Advise how to avoid such duplicates in the sitemap

Uthman, PortSwigger Agent | Last updated: May 29, 2020 11:18AM UTC

Thanks. You probably do not want to define the requests as static strings. You should ideally use makeHttpRequest to make HTTP requests and buildHttpMessage to create the HTTP request. Have you taken a look at any examples on our Github? https://github.com/PortSwigger What are you trying to do? Make a request and then retrieve it from the site map?

Yaroslav | Last updated: May 29, 2020 01:53PM UTC

I do not want to use makeHttpRequest() method, because I work with big amount of requests(~50 000) and making request for each will take time especially for HTML responses. So, my purpose is to use stub for IHttpRequestResponse to avoid making HTTP requests. This leads to duplicates as I said. Could you please advice how to fix this without making HTTP requests?

Hannah, PortSwigger Agent | Last updated: Jun 03, 2020 12:04PM UTC

Hi Have you tried having a look at some other extensions that implement the same functionality that you are trying to implement? You can find the code for all of the extensions that are available on the BApp Store on our public GitHub: https://github.com/PortSwigger

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.