Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Force spider engine to wait for page to load (Automated spider)

Sebastien | Last updated: May 07, 2018 10:14AM UTC

Hello, I was testing an intensive application this week and noticed that the spider tool wasn't finding a lot of the content on the site. The spidering was done through a scheduled task, so there was no manual browsing involved. I believe the cause of this is that the site is very JavaScript intensive, and it takes about 5 seconds before all the content is actually loaded on the site. (The site itself loads pretty fast, it's just the JavaScript content that's slow) Therefore I was wondering if it was possible to force the spider engine to wait x amount of seconds before it starts looking for other URL's. I already tried setting the delay between spider requests to 20 seconds, but this still leaves me with the problem that the spider engine only lists the URL's that are available when the page loads (without any of the JavaScript content), and just delays the requests between these found URL's. There also doesn't seem to be any extensions that provide this sort of functionality. Thanks

PortSwigger Agent | Last updated: May 08, 2018 08:52AM UTC

Hi Sebastian, Thanks for your message. You can configure Spider to pause between requests; this is in Spider > Options > Spider Engine > Throttle between requests. However, I expect this will not help your situation. Spider only has very basic support for JavaScript links. For JavaScript heavy apps you are normally best doing a manual crawl. You can still use Scanner effectively. We are aware this is a limitation and there is a major program of work ongoing to improve Burp's capability in this area. Please let us know if you need any further assistance.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.