Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Problems with Crawler browser

Raul | Last updated: Aug 21, 2020 10:26PM UTC

Hi, good afternoon, I have been trying to test the latest crawler functionality and scan with the browser that has burp suite but I have had a problem when I crawl without the browser, it adds all the content to my sitemap but when I do it with the browser from burp suite only adds some links to me but I can visualize with the flow plugin that actually makes more requests to the page and finds many more links but does not add them. What is this problem due to? Why do I see more requests than links added to my sitemap?

Liam, PortSwigger Agent | Last updated: Aug 24, 2020 10:50AM UTC

Do you have any scope / filter rules set? Would it be possible to provide us with screenshots or screen records demonstrating your issue?

Raul | Last updated: Aug 24, 2020 12:28PM UTC

I'm not using any filter, I just start the program and then enter a site to do the crawl, with the flow plugin that allows you to see the requests that go out and come in, I can see that several requests are made to the site that I enter for it crawl but many links are not added. https://i.ibb.co/c6FS2tD/Sin-t-tulo.png Of the people 220 requests that were made 100 requests are to the page that you enter for the crawl but it does not add those links, only 3 links

Liam, PortSwigger Agent | Last updated: Aug 25, 2020 09:35AM UTC

Is the site publicly accessible? Would we be able to take a look?

Ilya | Last updated: Sep 03, 2020 10:47AM UTC

I have the same problem. I can see scanner requests in flow extension, but endpoints not added to sitemap in target tab after end of crawling. No any scope / filter rules set. https://imgur.com/a/A0YBVRO

Liam, PortSwigger Agent | Last updated: Sep 04, 2020 08:29AM UTC

Is it possible to provide us with permission to test this site? If so, could you provide us with steps to reproduce the issue?

Ilya | Last updated: Feb 20, 2021 03:03PM UTC

> Is it possible to provide us with permission to test this site? Yes, use max 5 threads please. > If so, could you provide us with steps to reproduce the issue? 1. Start Burp v2021.2.1 2. Install flow extension from BApp Store 3. Start New Scan -> Crawl -> Enter "https://travel.yandex.ru" from post below 4. You can see /api/... requests in Flow, but not /api section in Target tab

Uthman, PortSwigger Agent | Last updated: Feb 22, 2021 02:18PM UTC

Thanks, Ilya. We will do some testing and get back to you.

Uthman, PortSwigger Agent | Last updated: Feb 24, 2021 04:53PM UTC

Ilya, please review the feedback from our development team: All the API requests return JSON files, which are not added to the sitemap (we only add HTML and JS files).

Ilya | Last updated: Mar 02, 2021 01:07PM UTC

> All the API requests return JSON files, which are not added to the sitemap (we only add HTML and JS files). Thanks for testing. But when I use proxy with browser and load "travel.yandex.ru", API requests with json response correctly adds to sitemap. It is intended behaviour? It looks a bit weird.

Hannah, PortSwigger Agent | Last updated: Mar 05, 2021 04:33PM UTC

If you manually browse to the site through a proxied browser, more requests will be added to the sitemap. When using the automated crawler, there is more logic applied to what should be added into the sitemap and so it is more selective about what is added. If you're interested in finding out more about the Scanner works, you can check out our documentation and brand new blog post: - https://portswigger.net/burp/documentation/scanner - https://portswigger.net/blog/web-application-cartography-mapping-out-burp-suites-crawler

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.