Burp Suite User Forum

Login to post

Scanned Urls are showing only two url Parent url and Parent url/robots.txt

Paresh | Last updated: Feb 09, 2021 07:36AM UTC

After scanning by Web app i can see only two url Parent url and Parent url/robots.txt in scanned urls . i have added parent url in the site url and none in the included urls. Do i need to add manually all the internal urls like Parent url/Home, Parent url/Show etc ? If the burp scan does all the internal scan automatically where i can see all the internal urls scanned details to be sure ? Regards, Paresh Badgujar

Ben, PortSwigger Agent | Last updated: Feb 09, 2021 09:45AM UTC

Hi Paresh, I have just responded to the email that you sent in on this matter. It might be easier to continue this discussion via the email thread.

David | Last updated: Mar 15, 2021 02:03AM UTC

Hi Guys! I am trialing the Enterprise version and I have teh exact same issue. I have also unchecked get robots.txt but still just get stuck on the beginning. Other tools I use and have tried just bypass this. How do I do that here? Please add me to this thread. Thanks /Dave

Ben, PortSwigger Agent | Last updated: Mar 15, 2021 03:17PM UTC

Hi Dave, I replied to the email that you also sent into us about this - it might be easier to continue troubleshooting your issue via email so we will await your response there.

Rui | Last updated: Sep 14, 2021 10:14AM UTC

I'm also facing this issue on one application, how can the coverage/crawling can be improved? I've checked the event log and there is an error event there saying "Communication error: host"

Ben, PortSwigger Agent | Last updated: Sep 14, 2021 10:30AM UTC

Hi Rui, To confirm, are you experiencing this issue with Burp Professional or Burp Enterprise?

Rui | Last updated: Sep 14, 2021 12:21PM UTC

Hi Ben, I'm experiencing this issue with Burp Suite Enterprise Edition v2021.8.1

Ben, PortSwigger Agent | Last updated: Sep 14, 2021 01:20PM UTC

Hi Rui, I can see that you have also sent us an email about this issue - in terms of information sharing it is probably going to be easier to troubleshoot this via the email thread that you have with my colleague (who has just replied to you). We will await your response there.

Rui | Last updated: Sep 14, 2021 03:25PM UTC

Hi Ben, the issue is solved now! Basically the first time that the app is accessed there is a redirect to an Identity provider, then another redirect, then a last one to the app again after getting cookies. The scanner does not follow those redirects apparently. I've added a recorded login sequence that just goes to that first redirect location and retrieves the needed cookies and now it is working and getting to all of the expected locations. Thank you anyway for the help. You email from support helped me understanding were the mistake could be.

Ben, PortSwigger Agent | Last updated: Sep 15, 2021 07:26AM UTC

Hi Rui, Glad to hear that you were able to solve this one!

You need to Log in to post a reply. Or register here, for free.