Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Create new post

Possible Bug in scanner processes not scanning APIs that end with file extensions.

Hellquist, | Last updated: Sep 28, 2020 07:43PM UTC

During a recent web app scan, someone on the team noticed a particular API wasn't being scanned at all. The API appeared to be taking a PDF as an argument (like https://blah.foo/mysite/myapi.aspx?ID=XXXXXXX-YYYYY-ZZZZZ-1111-222222222.pdf) The working theory is that Burp is classifying this as a PDF, and not an API hit that takes a file as an argument. Obviously, this isn't just a miss in Burps part; it's a fairly bad one. APIs that take files as arguments should be given extra attention. Not bypassed.

Hellquist, | Last updated: Sep 28, 2020 07:55PM UTC

v2.1.06. Im on an old version, so there is that.

Uthman, PortSwigger Agent | Last updated: Sep 29, 2020 08:42AM UTC

Hi Cody, Can you see if this issue persists in the latest version? (2020.9.1) We are working on improving API scanning capabilities (e.g. allowing users to define endpoints using OpenAPI definition files). Have you tried running an active scan on the endpoint? Or are you performing a crawl & audit?

Hellquist, | Last updated: Sep 29, 2020 06:55PM UTC

Yes, I tried with 2020.9.1 and the issue is still there. One way to see the issue is after surfing to the api containing page and then kicking off an audit: Audit -> Audit Selected items -> Consolidate Items If you select "Remove items with No parameters" it will drop the https://blah.foo/mysite/myapi.aspx?ID=XXXXXXX-YYYYY-ZZZZZ-1111-222222222.pdf (its a "get" by the way). If you scan it, it will register as an item with no parameters. You can kind of trick it to scanning it again by manually hitting the API hit in the browser without the extension. The scan will then correctly see this as an API with a single parameter. So back to my posting, it appears that this is registering as a binary file, and not an API hit that takes a file name as a single parameter.

Uthman, PortSwigger Agent | Last updated: Sep 30, 2020 09:26AM UTC

Hi Cody, Thanks a lot for that information. Can you please send us an email with further information? You can reach us at support@portswigger.net

Hellquist, | Last updated: Sep 30, 2020 01:44PM UTC

Uthman, I believe the information we've provided should be sufficient for PortSwigger to investigate, and I'm unsure what further information you expect us to provide. Keep in mind, based on the nature of this work, we cannot provide actual API hits or dumps of actual Burp interactions with a site being tested. This seems to be a fairly big issue with the tool. Please let us know once you've validated the issue, and what release you expect it to be fixed in.

Uthman, PortSwigger Agent | Last updated: Oct 01, 2020 08:48AM UTC

Hi Cody, Sorry about that. We have investigated the issue by creating a simple web app with a page with a link to a page with a pdf filename query parameter. We have run a crawl and audit of this, and also followed your steps of auditing selected items and consolidating items. We have found that Burp behaves correctly when crawling and auditing this item. Note that if you look in the Target > Site map, you will see the item with the full query parameter. But in the Audit items view, the parameter is deliberately not shown because we are sending multiple different attacks via query parameters. So even though the link with the query parameter is not visible, it is in fact being crawled and audited correctly. If you double-click the audit item, you should see the full query parameter. Please let us know if there are any further questions about this. For example, if you find that the crawl/scan is not finding links beyond the page with the PDF filename query parameter then we can investigate further.

Hellquist, | Last updated: Oct 02, 2020 07:36PM UTC

Uthman, Retested to follow what you were saying above. "Target > Sitemap" shows the base API hit (without the parameter), however, it appears to be greyed out. In the "Contents" panel, the "Params" for this item has no count associated with it. I tested something interesting out just a minute ago that might shed some more light on this issue. Some background; the API hit actually returns the file (in this case a PDF). If the file isn’t found, an error page is returned. When calling the API with a valid PDF name (but where the file doesn’t exist, and HTML is returned) Burp seems to detect it as an API. However, when the file is actually returned in binary form, Burp doesn’t seem to detect it’s an API (IE its grayed out). So while your test might have matched what I said higher up in the chain, you might not have seen the issue I am seeing unless your test API hit actually returns a binary file instead of just a junk return.

Uthman, PortSwigger Agent | Last updated: Oct 05, 2020 02:41PM UTC

Hi Cody, I have been discussing this further with our development team. Please review the feedback below: When an item in the site map is greyed out, it means that we have spotted the link in a response but we have not yet seen a request which matches that link. However, we should still be able to parse parameters on links we have not yet requested. Can you please provide additional information on the below? - What does the page which links to the pdf look like. Is it a form, an anchor, etc? Does the page have the correct 'real' pdf prefilled in the link or is it left to the user to browse to it knowing the real pdf? - What is the content type on the pdf, application/pdf or application/octet-stream? - If you install Logger++ and do an audit, can you see that parameter being used in the requests or not?

Hellquist, | Last updated: Oct 06, 2020 08:45PM UTC

1. The link presented to the user is a JavaScript function call. Something like: javascript:__doXXXX('YYYYY','ZZZZZ') Clicking this link opens a new window. This window's URL is the API call. Something like: https://blah.foo/mysite/myapi.aspx?ID=XXXXXXX-YYYYY-ZZZZZ-1111-222222222.pdf It should be noted I am clicking on this link and not relying on 2. Content time is "application/pdf". 3. I’m having trouble right now rescanning the site (as my initial assessment is done). However, I can tell you this: a. In the sitemap, the API is listed greyed out with no params. b. In proxy -> Http history, the item is listed in the "Other binary" category. c. If I "Audit Selected items" then "Consolidate Items...", the API hit is removed when I "Remove items with no parameters". I’ll try to get things set up for a rescan, bit it might take some time.

Uthman, PortSwigger Agent | Last updated: Oct 07, 2020 12:57PM UTC

Hi Cody, Thanks a lot for that information. When you say 'It should be noted I am clicking on this link and not relying on ', do you mean to say that you are clicking on the link manually in the browser and not relying on burp crawl/audit/scanner to do so itself? If you enable the 'Other binary' filter in the site map, do you see the item with the parameter? If you 'Audit selected items' on requests with parameters, they will be truncated in the scan wizard in the UI (under Items to scan). If you double-click the audit item in the task, you should see the query parameter.

Hellquist, | Last updated: Oct 07, 2020 03:54PM UTC

1. Correct. 2. Yes, when I select "Other Binary" I see the API hit as well as the parameter. 3. I’m not sure I am following this 100%. When trying to follow this item, I noticed the following: If I do the "Other Binary" thing you listed above, and then generically scan the API (Audit Selected Items), it seems to pull in the API plus filename parameter. If I don’t; the same scan will be limited to the URL hit without any parameters. So the Audit Selected Items seems to be based on what the Site Map filter is set to. It seems Burp is acting under its own rules consistently, so it might be tempting to say this working as designed. However, I still feel this is an issue. Lumping a Binary returning API with "Other Binary" makes sense, but it’s really something different from a static binary file on a web server. Also having scans limited based off of the filter on the site map is unexpected (by me, maybe not other people).

Uthman, PortSwigger Agent | Last updated: Oct 09, 2020 09:49AM UTC

Thanks a lot. We will take your feedback on board. When you are 'auditing selected items' and a filter is set on the sitemap, there are no items with parameters selected. You have hit an API where a certain parameter causes the response to be binary, so when the filter is hiding binary content, we still show the API path but hide the specific parameters which resulted in binary. If you change the parameter to cause HTML to be sent back (as you said you did) we continue to hide the binary returning parameter but show the one which returned HTML. So we do have some degree of granularity.

You must be an existing, logged-in customer to reply to a thread. Please email us for additional support.