Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Best way to crawl (traditional web sites as well as heavy javascript and AJAX using websites)

vivek | Last updated: Oct 05, 2020 07:57AM UTC

Hi, I want to know how I can use burp to best crawl and auto submit forms on a website as an unauthenticated user for both traditional web sites as well as heavy javascript and AJAX using websites. I tried a test case yesterday where I gave burp a page to crawl (https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_form_method) and it was unable to go to the iframe and submit the simple GET based form. Can you please tell me how it can be done and how can burp be made to do the best crawling and auto form submission in general.

Hannah, PortSwigger Agent | Last updated: Oct 06, 2020 08:36AM UTC

Hi Due to this example being in an iframe, for this case, you will need to disable browser-driven scanning to crawl properly and submit the form. Iframe support for browser-driven scanning is an improvement on our roadmap. Generally speaking though, for crawling more modern websites we would recommend ensuring that browser-driven scanning is enabled. You can find this option in your scan configuration, under "Crawling > Miscellaneous > Use embedded browser for crawl and audit".

vivek | Last updated: Oct 06, 2020 01:50PM UTC

Thanks for the clarification :)

vivek | Last updated: Oct 07, 2020 02:12PM UTC

Hi there, I disabled the browser-driven scanning but the crawler was still not able to submit the form in the iframe. Any suggestions? And in your knowledge is there any crawler better for this job which I can use with burp or stand alone? Best Regards, Vivek

Hannah, PortSwigger Agent | Last updated: Oct 08, 2020 07:28AM UTC

We've just been testing using a crawl only (as an audit is too intrusive). Try with the start URL of https://www.w3schools.com/tags/tryit.asp?filename=tryhtml_form_method with https://www.w3schools.com/tags/ and https://www.w3schools.com/action_page.php as the scope, and the embedded browser disabled. With those settings, we submitted the form. Please also make sure that you are looking under both HTTP and HTTPS.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.