Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Scanner is crawling and auditing out of scope items.

Johnathan | Last updated: Aug 13, 2020 12:25AM UTC

Hello, I am attempting to automate some tests with crawl and audit. I have defined my scope to exclude *.css files. When I use scan to crawl and audit, the crawl will find the *.css files and audit will start auditing those files. Is there a possible work around for this issue? I have attempted to define my scope in scan as well as target. It seems to be working in the site map, but when running the scan it is still picking up the *.css files. The regex I am using to exclude the .css files is '\.css($|\?)' (Please disregard the single quotes, they are not part of the regex)

Uthman, PortSwigger Agent | Last updated: Aug 13, 2020 09:13AM UTC

Hi Jonathan, You have a few options to exclude file extensions depending on how you launch a scan: 1. If you are using the Auditing selected items function (Select an item in the Target > Sitemap > Scan > Open scan launcher), you can use the consolidated items function to remove the appropriate extensions. 2. To exclude the extensions in a Crawl and audit, you can use the advanced scope controls in the New Scan wizard. E.g. .*\.css$ 3. For a live task, you need to set up your scope under Target > Scope first. You can then select Suite scope as the scope for the task. E.g. Advanced scope and an excluded from scope rule similar to: .*\.(css|js|jpg) If you have any issues, please send us an email on support@portswigger.net with screenshots.

David | Last updated: Dec 21, 2020 09:21PM UTC

Since the target scope is used in most parts of Burp Suite, it would make more sense if the Crawl and Audit also honoured the scope settings by default (that's what I would expect). Before running a scan, a tester may browse an application manually to identify paths that may be problematic or should not be touched in a scan and remove them from scope. It's very unfortunate that this isn't also automatically restricting the scope of a crawl or audit. At the very least, please give us an easy option to import current target scope into the scan scope configuration (and perhaps display a warning when the scan scope includes items marked as out-of-scope for the rest of Burp's tools).

Uthman, PortSwigger Agent | Last updated: Dec 22, 2020 09:48AM UTC

Hi David, If you set the Target > Scope > Include in scope prior to launching a scan, all the in-scope URLs should be automatically added to the URLs to scan. Would you like me to raise a feature request to add the Excluded from scope URLs to Excluded URL prefixes in the scan task?

David | Last updated: Dec 29, 2020 03:29PM UTC

Hi Uthman, Yes, I would like to raise feature request(s) for the following to fix current issues I see with Burp's Scanner: 1. By default, make the scanner (for both crawl and audits) to respect the general scope settings defined in Target (this is what we expect from all tools within Burp). If a custom scope is required for the scan that differs from the general Target scope, then that should be configured as a detailed scope configuration. If the scan is not made to respect Target scope by default, the very least I would like to see is an option (preferably enabled by default) that imports the current Target scope into the detailed scan scope configuration. 2. When I right-click a target in the site map and select "Scan", I would like the scanner to automatically include *everything known* for that host (apart from items explicitly marked as out of scope, see #1 above). I might have done manual discovery of content and used other tools to uncover paths that are not directly or indirectly linked from the base URL, however these will not be covered by default when Burp is scanning the host (unless explicitly added to the scan-specific scope). Burp should already know these when they are present in the proxy or target site map, and include them by default. Now the scanner will miss anything that's not directly found by the crawler. The assessor may already have spent a lot of time uncovering additional content, which currently is not leveraged when doing a Crawl and audit. 3. Retain the directed graph between crawl and audit. Currently, Burp will only leverage the directed graph built up in the crawl phase during audits when the "Crawl and audit" option is selected. As a user, I would expect that doing a "Crawl" and then "Audit selected items" (covering all items discovered in the crawl) would yield the same results as doing a "Crawl and audit". It does not, since the relation between items obtained during the crawl is not preserved for the audit in the second step. This is problematic, because I want to know what the crawler will touch and uncover before I give the scanner the go ahead to run a full audit. I want the ability to fine-tune the scope between the crawl/discovery and active audit (e.g., remove some additional pages discovered during crawl from the scope of audit phase). With the way Burp currently works, I have to chose between either re-run a time-consuming crawl by doing a "Crawl and audit" after the initial "Crawl", or just go with "Audit selected items" and instead miss out on all the intelligence of the scanner to handle multi-step forms, dynamic CSRF token updates, etc. (unless I manually create those macros & rules for the scanner). In other words, I'm left with two bad options, which should be easy to solve if only Burp had the sense to retain and re-use the directed graph built during a Crawl in a subsequent "Audit selected items". This would give testers fine-grained control over scope while retaining the intelligence of the scanner. Many thanks!

Nicolas | Last updated: Dec 29, 2020 05:47PM UTC

Thanks David for this great post! Some quick notes... #1 I found troubling that the Scanner doesn't respect the Suite scope. Especially when the documentation of the global scope states that "the suite-wide scope definition provides a quick and easy way to tell Burp what is fair game and what is off limits". I'd have expect Scanner to respect the Suite scope, with the possibility (like in session handling rules) to expand or narrow it locally. I can easily imagine situations where going out-of-scope would lead to serious troubles, from bans (bug bounties) to legal risks (pentests) #2 From my understanding, a scan (crawl or audit) simply defines the entry points. And considering all resources as possible entry points would be wasteful. If a manual investigation/recon identified additional items, adding them manually to the scan definition (or running separated scans) makes sense to me #3 Ah, the directed graph... I full agree, it's a pity to lose it when running a crawl, then later an audit. I'd like to see persisted, and possibly even viewable (I mean, in an easier way than the current debug settings dumping to a text file). In a perfect world, it could even be re-used from manuals tools like Repeater and Intruder, but I'm dreaming out loud, sorry. In short, the crawler form v2 is awesome, but please let us limit it (scope) and re-use the incredibly useful knowledge it produces (graph).

PortSwigger | Last updated: Dec 29, 2020 05:58PM UTC

Point #2 is mainly an issue because of #3, if the directed graph was saved it would be sufficient to just use "Audit selected items" to include all additional items not found by the crawler in a "smart" scan.

Michelle, PortSwigger Agent | Last updated: Jan 04, 2021 02:46PM UTC

Thanks, both of you, for the feedback. We've raised a feature request to look at improvements for how the Scan Configuration Scope and Target Scope work. I've linked this thread so we can let you know when there is an update. For the directed graph for a crawl, do you have my further feedback on how you would ideally like to see this information presented, please, so I can also raise this with our developers?

David | Last updated: Feb 18, 2021 05:43PM UTC

Hi PortSwigger, any update yet on when we can expect target scope to be used in scans (or at least a default option to import that scope)?

Michelle, PortSwigger Agent | Last updated: Feb 19, 2021 12:02PM UTC

Thanks for getting in touch. We don't have any timescales as yet I'm afraid, but this is something that's on our radar. We'll post back here when there's an update.

onX | Last updated: Dec 17, 2021 09:44PM UTC

Even when I specify advanced scopes the scan logger will follow links to resources referenced in the page. Is this normal? I even went so far as to exclude domains starting with any letter except my domain but I still catch google cloud resources, ad tracker links etc in my net. I really want the scanner to only hit URLs that I specify. Even when I have it in my target scopes and my scanner scopes, it doesn't obey. v2021.10.3

Michelle, PortSwigger Agent | Last updated: Dec 20, 2021 08:42AM UTC

Thanks for your message. Are you seeing this during the crawl? If so, in your crawl configuration, do you have Miscellaneous -> Embedded Browser Options -> 'Load site resources from out-of-scope hosts' enabled?

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.