Research Academy
My account Customers About Blog Careers Legal Contact Resellers
The Burp Suite User Forum was discontinued on the 1st November 2024.

Burp Suite User Forum

For support requests, go to the Support Center. To discuss with other Burp users, head to our Discord page.

SUPPORT CENTER DISCORD

Modify Content Discovery task queue

Isaac | Last updated: Jul 17, 2020 10:12PM UTC

Current behavior: Content discovery session configuration, specifically task queue, is static after session start. It updates dynamically according to discovered content, with no option to remove items from queue. Desired behavior: Ability to remove items from queue while session is running.

Liam, PortSwigger Agent | Last updated: Jul 20, 2020 02:19PM UTC

Thanks for your message. To help us properly record this request, could you provide a use case for this feature?

Isaac | Last updated: Jul 22, 2020 12:20AM UTC

Initial content discovery/enumeration necessarily involves not knowing what content is on the target webserver. As such, it is difficult to provide a configuration at the outset that achieves optimal thoroughness without having overly long-running discovery sessions. Oftentimes there are directories that are discovered (and are added to the task queue) that are unlikely to contain relevant content. For example, running content discovery on http://host.com/ may find http://host.com/css http://host.com/js http://host.com/img etc. On an initial scan, I dont usually care to fully search these common directories. To my knowledge there is not even a way to exclude directories from consideration. This leads to a chain of subsequent tasks that are equally irrelevant (e.g. all combinations of: /imgs/{directory}/{filename}.{extension}, /js/{directory}/{filename}.{extension} /css/{directory}/{filename}.{extension} The problem compounds with each additional level of Max Depth setting. So, in the event that these were discovered, I would like to be able to manually remove them from the queue, thereby shortening the duration of my discovery session. Sort of human-in-the-loop automation. Happy to provide more input if it's still not clear.

Michelle, PortSwigger Agent | Last updated: Jul 23, 2020 08:17AM UTC

Thanks for the explanation of your use case. This is currently not on our roadmap or backlog, but we’re keeping a record of it to see if there is further demand. We prioritize our ideas based on value to all of our customers. If the demand grows it will be added to our roadmap or our backlog of small improvements. In the meantime c=would the option to provide a custom directory list instead of using the built-in ones help you at all in this situation?

Isaac | Last updated: Aug 12, 2020 11:31PM UTC

Thanks. No, that option already exists (if I understand what you are suggesting) and it doesnt produce the behavior I'm hoping for. I already use custom lists for most things, but the real issue I'm having is that there is no way to change the settings on the fly. So I am forced to either let the discovery process run to completion (and scan parts of the application that I probably dont care about), or stop the scan and refine my configuration before running again. From a workflow perspective this is pretty inefficient. This leads me to run multiple discovery sessions, starting with very restrictive configuration (e.g. a single custom wordlist, and only directories), then incrementally increasing the depth of discovery on results of previous discovery session. Since I can do this already using tools like gobuster, there is little benefit to me doing this in Burp (except for it being added to my Target tab). This is one of those mundane tasks that I do for every application, and I'm sure I'm not the only one. Adding this feature would accelerate the recon phase for users, and would differentiate Burp from existing tools even further.

Michelle, PortSwigger Agent | Last updated: Aug 13, 2020 07:48AM UTC

Hi Thanks for the additional information, we appreciate you taking the time to explain the background. We've passed this all on to our product team and we'll post back on this thread when there is an update.

Joshua | Last updated: Nov 05, 2020 03:55PM UTC

Just came across this thread while searching for this exact thing. It really would make discovering content a lot easier and take less time. Depending on the response from the server when an extra '/' is appended to the path, the discovery session may think the new path is a discovered directory. It will then get stuck in this "loop" recursively searching that "discovered" path. In a discovery session I am running right now, the '///////////////////////' path is being tested for content. My session is pretty much useless because of this. If I could remove queued tasks, then the session could move on to discover actual content.

Michelle, PortSwigger Agent | Last updated: Nov 05, 2020 04:04PM UTC

Thanks for the feedback, we'll add your vote to the feature request

Andrew | Last updated: Aug 03, 2021 07:48AM UTC

Me too. The Content discovery out of the box is extremely difficult to use and seems to generate a large amount of junk duplicate folders. This actually may be a bug as i can see why it has created hundreds of paths such as: /add-account/static/css /add-account//static/css /add-account//static/ /add-account//static/// /add-account////static/css /add-account//static////css etc Giving the user the ability just to junk stuff out of the list would make it much easier than trying to figure out why all these duplicate folders have been created. Also another idea would be to provide a list of what is being pulled in from the site map with a remove duplicate content like in the Active scan dialog.

Michelle, PortSwigger Agent | Last updated: Aug 03, 2021 09:50AM UTC

Thanks for the feedback. We've added your vote for the ability to cancel/re-order tasks in Content Discovery. As regards the junk duplicate folders, can I check if you have any extensions enabled? If you do, do you see the same behavior with extensions disabled (in case there's some specific combination that's causing this)?

Andrew | Last updated: Aug 05, 2021 03:38AM UTC

Thanks.Appreciate it. I have numerous extensions i enable and disable from time to time to test things. Any one of them could result in junk content in the site map and cause subsequent problem. Rather than try and figure out the extension causing the problem its just easier to to be able to modify discovery on the fly. I did find that disabling the 'Names observed i use on target site' stopped importing the addition slash directories i described. That might be good work around for people with the same issues but i assume this also leads to lowers the efficacy of the scan also.

Andrew | Last updated: Sep 09, 2021 12:16AM UTC

Another reason why this requirement is important is that you can tune what is considered a valid finding in Content Discovery (e.g. exclude 400 and 406 as valid findings), the scan is flagging lots of invalid 'found' content and then feeding that back into later discovery tasks. This basically creates snowball of discovered content which doesn't actually exist. If we could just delete items from the Queued tasks we could get rid of these false positives before they get fed into other Discovery tasks. The alternatively is adding the ability to filter out particular response codes as indicative of a successful discovery like other standalone discovery tools allow. Thanks

Michelle, PortSwigger Agent | Last updated: Sep 09, 2021 07:55AM UTC