Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Make Burp Pro crawl actually discover anything from an SPA app using OIDC?

Kamil | Last updated: May 21, 2021 04:35PM UTC

We are struggling with the Burp Enterprise trial actually discovering anything useful about our web app, and as the Enterprise version seems to offer barely any logs, I went for the Burp Pro trial, to see what's happening. Our web app is a Single Page Application (SPA) and uses OpenID Connect (OIDC) for authentication (and authorization). It also offers OpenAPI (Swagger) documents for quite a lot integration REST API-s. I recorder the logon process, to use it with an automated scan. The process goes through communication with an Identity Provider (IDP) service, which is not in the scope of the scan (a set of separate domain addresses). The process ends with the browser loading the application home page, which in turn causes a couple of UI-dedicated REST API-s being called (being part of the application and the scan scope) and loading some external components (e.g. a spatial map, like Google Maps, OpenStreet Map or Bing Maps), which are not in the scan scope. I explicitly included in the scan scope URL-s of the OpenAPI documents - those are only accessible to authenticated users, including the one used in the recorded logon process. The problem is that the Burp Pro crawl only discovers what it was given on input (home page address, OpenAPI documents and the logon URL) and a couple of static files (e.g. referenced from the home page images, fonts and scripts and files like "robots.txt", "favicon.ico"). It does not include anything documented in the OpenAPI documents nor anything (API-s) referenced by the authenticated home page. The crawl config is configured to use the Embedded Browser and to parse API definitions. The crawl strategy is set to "Most complete" with basically no crawl limits. The crawl and audit log shows the initial unauthenticated calls to URL-s listed in the scope, then the recorded logon process is executed a couple of times (with visible calls to the IDP addresses and then - authenticated - to the home page and the REST API-s it refers to) and there's basically nothing (in the crawl or audit part), which would suggest, that it noted the API-s called from the authenticated homepage. There's basically 2 paths it does see: 1 API which starts the logon process and 1 API, which is normally also called from unauthenticated home page and the crawl/scan seems to call the latter one only as unauthenticated (no authentication cookies present) so it gets 401 response and probably that's why it does not add it to the site map from the crawl. It does pull the OpenAPI (there are authenticated and unauthenticated calls to it) but it does nothing with it (other than using random-generated variations of the URL-s I've given to it or of the static files). I don't know what I'm doing wrong, but I can't make Burp notice all that the SPA code calls nor make it navigate the page (which would lead to discovering many more UI-dedicated REST API-s, than those called by just loading the home page) or use the OpenAPI document contents. The result is 16 audit items found, where 6 are the URL-s I've given to it, 9 are the static content it found and the last is the logon URL. The issues reported afterwards are basically meaningless (e.g. all the OpenAPI documents are reported - multiple times - as "Backup file" and the static files as cacheable). Please advise.

Michelle, PortSwigger Agent | Last updated: May 26, 2021 11:10AM UTC

Thanks for your message. Which version of Burp are you currently using? We are currently working on improvements to support SPAs so it would be worth making sure you are on the latest version and also keeping a close eye out for new releases to make sure you get further improvements. One option to find out how the scan progresses and where it has problems would be to watch the crawl in a headed browser (this can be set in the Crawl configuration under Miscellaneous -> Show the crawl in headed browser). Is the OpenAPI resource explicitly referenced in a page?

Kamil | Last updated: May 26, 2021 04:50PM UTC

I'm using Burp Suite Professional v2021.5.1, so the latest stable available. As for the Burp Suite Enterprise, it has the same version of the scanner (2021.5.1). As for the SPA improvements in the making, I doubt we'll see them. Our trial expires in a couple of days, so the result of the evaluation will probably be that Burp Enterprise is useless for our case, and as result our organization will probably not buy any licenses (as Burp Pro licensing decision seems to be also bound to this evaluation). If you have any immediate hints on how to make the Burp Enterprise to work for us, I'm all ears. As for watching the progress, locally (Burp Pro) it's quite OK - I'm looking at the logger in the crawl/audit task details and it's enough. Unfortunately there's no access to such a thing in the Enterprise edition. I don't think that headed browser option is available for Enterprise (and even if it would, it'd probably happen on the agent machines, where I have no access). During the scan the Enterprise edition does not even show you the event log - it's available only after the scan is finished and the event log is so vague that's basically useless for identifying the causes of scan issues. (That's why I'm trying to recreate the Enterprise approach and configuration locally under Pro.) As for OpenAPI resource, it's not explicitly referenced in the page, but - as I wrote - the URL to the OpenAPI resource is explicitly given on the list of URL-s to scan and it is visited (and the "backup file" issue is reported against variants of this URL) and from looking at the logger, at least some of requests against it are authenticated, so they do receive the contents of the OpenAPI document. Still, the information from the document is not used. Personally I do see value of Burp Pro, which - if given a chance to work as a proxy to get to know the application - could probably do a lot, but as licensing for Pro seems to exclude use in automation (i.e. licenses not bound to users, but machine/process/organization) that value for the organization is minimal. As a suggestion I would say, that the Enterprise could benefit much from having the proxy scanning capabilities of the Pro version (i.e. where you could plug it in as proxy into our automated UI and API tests as input for subsequent crawl and audit) and other features of it or you could simply offer a licensing option for Burp Pro, which would allow use in automation scenarios (e.g. licensing based on count of machines/agents it will run on).

Michelle, PortSwigger Agent | Last updated: May 27, 2021 02:47PM UTC

For the SPA, if this is a publicly accessible site or you happy to send us the HTML the page uses we'd be happy to take a closer look so we can let you know how our planned improvements will cope with the site for future reference. You can create a scan configuration for Burp Suite Enterprise to use so that the crawl is shown in a headed browser, but you are correct in thinking that this would be on the Agent Server itself so if your access to the Agent Server is limited this may be difficult for you. For the issues you're describing with the OpenAPI resource, it sounds like something we would expect the Scanner to cope with during an automated scan (which would then mean you would be able to scan it using Enterprise). Again, we'd be happy to speak to the developers and take a closer look at this, either by performing a test crawls or taking a look through the copy of the OpenAPI spec. If you're happy to share this with us you can email us via support@portswigger.net. As Burp Suite Enterprise is designed for automation it does not include the same manual tools as are available in Burp Suite Professional, but I will pass on your feedback about how you are able to manually crawl your sites and use Pro to audit them but then cannot pass this information from the manual crawl of the site into Enterprise.

Kamil | Last updated: May 28, 2021 04:26PM UTC

We can't share the application access/code without explicit written agreement (NDA, etc.). As for headed browser, we were already asked to do the scans "with the embedded browser disabled and dynamic analysis techniques disabled" as part of diagnosing a case of Enterprise scan apparently getting stuck. So it would seem, that our chances to see something of value from the scan is getting lower and lower. As for OpenAPI, unfortunately we also can't share the exact OpenAPI files. I can say, that the documents use OpenAPI v.3.0.1 specifications, are served in Json format and are automatically generated (this is an ASP.Net Core 3.1 web application with Swashbuckle used and includes Swagger UI). Application offers multiple OpenAPI files (divided functionally and by API version). We have verified that the files are structurally valid. (Incidentally this proves that my previous statement, that OpenAPI documents are not referenced in an application page was not quite correct, as Swagger UI pages do reference those, but Swagger UI is a set of pages separate from the actual product pages.) Is there any indication in the event logs I should look for, to know if Burp has recognized a document as an OpenAPI specification? I have not found anything like that, and - as I wrote - I found no evidence of using its contents from the requests visible in Bur Pro logger (I made sure everything is captured), but maybe I'm missing something obvious there. As for Enterprise being for automation, there are many Bur Pro features, which would be beneficial in automation scenarios. Some are offered by free products, which we already use in our automation (CI/CD), e.g. ability to work as a proxy during automated functional scans, to have subsequent security scan leverage the information gathered during functional scans. The ability to pass manual crawl results from Pro to automated scans of Enterprise could also be an improvement.

Uthman, PortSwigger Agent | Last updated: Jun 01, 2021 10:44AM UTC

Thanks for the feedback. Can you try this again in 2021.6 (the latest early adopter release of Burp Pro)? In terms of the indication in the event log if Burp has found an OpenAPI definition, you should see something like the below: 1622543852968 Debug Task 4 Found API definition at location https://petstore3.swagger.io:443/api/v3/openapi.json. Are you adding the location of the definition file as a seed URL? Or just the paths to the API endpoints? Importing manual crawl results from Pro to Enterprise is a popular feature request. We have registered your interest and will let you know when that gets implemented.

Kamil | Last updated: Jul 28, 2021 04:28PM UTC

This were a busy couple of months and I was unable to get back to this topic during that time. Now I see we've got the Enterprise trial extension some time ago... which ends in 2 days. My Pro license trial has expired, so I only have the 2 days of Enterprise license to work with. :) I also see, that we have an NDA in place now, so feel free to contact me directly for application access. The Enterprise version we now have is 2021.6 with Burp Scanner version 2021.6.2. I see, that this version has some UI issues on Firefox and my first attempt to do a re-scan of the application has failed with some cryptic "Lost communication with BurpSuite" error message after working for 20 minutes. (I think it was already past crawl, but there's nowhere to verify this.) No logs are available to me, just a scan report with what it was able to find before it crashed (10 information issues, showing that it was able to discover some static content). (Not sure why would Burp not make the event log available for failed scans - that's when it's most useful.) I've ordered a new scan - we'll see how it goes. Last one failed after about 20 minutes, previous non-failed scans took about 1.5h, this one is spinning for about 30 minutes now and seems to be in auditing phase. (The information that it's auditing is only visible on the sites list, or at least I have not found any other place it would show up.) As for the "event log" entry I should see, I'm sure I did not see that one previously. As for locations of the definition files, as I wrote before "I explicitly included in the scan scope URL-s of the OpenAPI documents" (I listed them in the "Included URLs"). (I also tried pointing it to Swagger UI pages, where the OpenAPI document URL-s can be easily discovered.) I have not provided it with any API endpoint URL-s, as it's the role of the OpenAPI documents to show what is available (and also discover undocumented endpoints, which are used by the application SPA) and we don't want to tweak the Burp configuration each time we change something in the application API-s. As for "Importing manual crawl results from Pro to Enterprise", this would be a nice feature, but the more valuable for us would be to have that proxy feature in the Enterprise (so it can learn/map the application from our automated functional tests without a human / manual process involved).

Uthman, PortSwigger Agent | Last updated: Jul 29, 2021 08:27AM UTC

Hi Kamil, Thanks for that information! The improvements to SPA crawling were added to the 2021.7 release (this is an 'early adopter' release of Burp Pro). It will not be available in Enterprise until that release goes to the stable channel so we will keep you updated on that. If you could email support@portswigger.net with some URLs and screenshots, that would be great.

Kamil | Last updated: Jul 29, 2021 07:28PM UTC

As I wrote, my Burp Pro trial license has expired, so if the early adopter version is not available for Enterprise, I have no way of testing the early adopter 2021.7 version. As I mentioned, I requested a new scan from the Enterprise app (2021.6 / 2021.6.2). It did not fail this time. - It failed to authenticate, so it was not able to access all the OpenAPI documents available, but for those it did, the result is still the same: there's no trace of it understanding, that those are OpenAPI documents. (The authentication failure is probably due to a change in the Identity Provider service UI we use, so the recorded logon sequence could not be executed in full.) - As this did not fail, so I do have access to the event log. Not much of value in there though. It has nothing on the OpenAPI. It's not even a valid CSV, as it does not properly quote the values if there are quotes in them (and there are this time, as the RECORDED_SEQUENCE_ERROR entries contain quoted parts). It's not even properly ordered (it's sort-of in reverse chronological order, but entries happening on the same second are not). - I only know it did visit the OpenAPI documents from the "Scanned URLs" list (where it shows those, which are accessible without logon) and from the issues list (where - as before - it raises false-positive "Backup file" entries due to how file extensions are handled for those by the server). There's no other log which would show what were the hundreds of requests it did for those and why it did not consider as scanned the other ones listed in site configuration as "Included URLs". I'm not sure what URL-s and screenshots you'd like me to send, but I'll come up with something tomorrow. (Along - hopefully - with results of an authenticated scan, as I recorded a new logon sequence.)

Uthman, PortSwigger Agent | Last updated: Jul 30, 2021 09:49AM UTC

Hi Kamil, Thank you for the feedback! It may be better to discuss this over email so that we can fully explain any limitations and gather more information from you e.g. the API definition file, screenshots of your scan, a scan report, etc... The forum is public so it's better to communicate this information over email. I will arrange an extension for your trial to help us with some further debugging.

Liam, PortSwigger Agent | Last updated: Nov 04, 2021 02:10PM UTC

After careful consideration, we've decided against providing a feature to import Burp Pro sitemaps into Burp Enterprise. Please let us know if you need any further assistance.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.