Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Crawler not crawling thorough enough

floyd | Last updated: Jan 14, 2021 10:34AM UTC

Hi there, Burp Pro v2020.12.1. I have an application and I managed to make the headed crawler login properly. After the login credentials were entered the crawler needs to figure out it can supply *any* 2fa code in the next form (test web application where everything is accepted). That works as well. However, afterwards the page needs a couple of seconds to load. The first problem seems to be that the crawler is not patient enough and moves away too quickly, in most of the cases not getting to the post-login website. However, sometimes it is able to login correctly and see the post-login page. But then it usually only clicks one or two links in total and nothing else. A lot of links in the page are of the form: <a ui-sref="test.faq" href="#/faq"><span translate="" class="ng-scope">Help</span></a> But they are never clicked (although 2 or 3 of them are). But there are also <button> that are never clicked. I've tried all the crawler settings: - Maximum link depth: 25 - Crawl strategy: Most complete - Read timeout for site resources: 9999 But in the end it only says it crawled 4 unique locations, which is not a lot. However, I did not dare yet to change the Crawl Strategy Tuning settings. Any hints on what I could try in there?

Uthman, PortSwigger Agent | Last updated: Jan 14, 2021 01:35PM UTC

Hi Floyd, You mentioned that "afterwards the page needs a couple of seconds to load". Can you elaborate on this? What resources take long to load? JS files? Something else? What are the sizes of these files?

floyd | Last updated: Jan 15, 2021 11:58AM UTC

Hi Uthman, The couple of seconds is the bundle.js that is a 5MB file that has to be loaded. At first it took 15 seconds to load through Burp, but now I've optimized that part and it only needs 5 seconds now. Three more ttf fonts that have 170KB take another 1.2 seconds each to load, but that does not seem to be a big issue (a lot of it is in parallel). The Burp crawler is now able to login properly each time it wants because it only takes 5s to load the bundle.js. There would be 3 links on the top of the site, but they never get clicked. It only clicks the 6 main menu links on the left side and these load a lot of different functionality into the website (single site though, so all on the same URL via JavaScript). However, the crawler also doesn't click the loaded links. Any ideas on how to crawl such an Angular app more thorough? If necessary I'm willing to wait for a long time and do a very thorough crawl.

Uthman, PortSwigger Agent | Last updated: Jan 15, 2021 12:12PM UTC

Thanks for that information. Can you please send us an email (support@portswigger.net) with diagnostics and a screen recording of this behavior in a headed crawl? You can enable the headed crawl option under Crawling > Miscellaneous > Embedded Browser Options in your scan configuration. Is it a single-page application that you are testing? Do you notice different results with browser-powered scanning off vs on? (Crawling > Miscellaneous > Use embedded browser for crawl and audit)

floyd | Last updated: Jan 15, 2021 01:56PM UTC

Hi Uthman, Sorry, I can't do the email and screen recording at the moment. I'm already using the headed crawl, that's why I know it doesn't click the links (and I also checked Logger++). Non-headed crawl didn't give any different results. Yes, it is a single-page application I am testing. Yes, when turning browser-powered scanning off, there are even less results :)

Uthman, PortSwigger Agent | Last updated: Jan 19, 2021 09:22AM UTC

Thanks! Unfortunately, the crawler cannot handle SPAs well at the moment. We are working on incremental updates to improve this drastically. We will update this thread when an improvement to the scanner has been released.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.