Research Academy
My account Customers About Blog Careers Legal Contact Resellers

Burp Suite User Forum

Login to post

[Enterprise] Random scanning machine errors

JAVIER | Last updated: Feb 03, 2023 08:06AM UTC

Hi, We have recently deployed BurpEnterprise and I've been seen random errors in scanning machines. Our setup es the following: - BE server + posrgresql database is running in its own server (8 cores, 32GB ram) - 20 scanning machines, with 2 licenses each. Each machine has 8 cores and 32GB ram. They are all hosted in AWS, connected via an internal network. I've seen two kind of errors: - Agent not responding - "Scanning machine has been deleted" When ssh-ing to the affected machine it looks like the BE agent processes are still running. Inspecting enterpriseAgent.log there are a few entries like this, and no activity for a few hours: 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 122] 2023-02-03 12:04:50.845 914235348172412, net.portswigger.qm INFO - connectedSocket, opened new socket: 1894934965 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 27] 2023-02-03 12:04:53.843 914238346350190, net.portswigger.t6 INFO - Closing socket due to timeout: 1894934965 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 27] 2023-02-03 12:04:53.844 914238346626672, net.portswigger.t6 INFO - Closing socket: 1894934965 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 27] 2023-02-03 12:04:53.844 914238346823134, net.portswigger.t6 INFO - Socket closed: 1894934965 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 122] 2023-02-03 12:04:53.844 914238346982136, net.portswigger.t6 INFO - Closing socket: 1894934965 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - 2023-02-03 12:04:53: Failed to send READY heartbeat to http://127.0.0.1:8095/5ee44c04-e014-4bd2-a335-bf4c68f874b0/api/heartbeat [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - java.net.SocketTimeoutException: Communication timed out [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.s(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.A(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.T(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.qm.i(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.je.t(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at burp.zpk.k(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at burp.zpk.lambda$new$0(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.FutureTask.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.lang.Thread.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - Caused by: net.portswigger.gh: Socket closed [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.hy.Q(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.ty.n(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.g(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - ... 1 more [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - Caused by: java.net.SocketException: Socket closed [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.endRead(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - ... 5 more [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - 2023-02-03 12:04:53: Rescheduling heartbeat more promptly due to failure [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - 2023-02-03 12:04:53: Using longer timeout due to failure [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 125] 2023-02-03 12:04:53.846 914238348545781, net.portswigger.qm INFO - connectedSocket, opened new socket: 1799760419 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - Deleting temporary files - please wait ... done. [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 125] 2023-02-03 12:04:56.846 914241349313283, net.portswigger.t6 INFO - Closing socket: 1799760419 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - [Thread: 125] 2023-02-03 12:04:56.847 914241349740037, net.portswigger.t6 INFO - Socket closed: 1799760419 [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - 2023-02-03 12:04:56: Failed to send READY heartbeat to http://127.0.0.1:8095/5ee44c04-e014-4bd2-a335-bf4c68f874b0/api/heartbeat [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - java.io.IOException: Read timed out [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.s(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.A(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.T(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.qm.i(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.je.t(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at burp.zpk.k(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at burp.zpk.lambda$new$0(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.FutureTask.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.lang.Thread.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - Caused by: java.net.SocketTimeoutException: Read timed out [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.timedRead(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at java.base/java.net.Socket$SocketInputStream.read(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.hy.Q(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.ty.n(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.g(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - at net.portswigger.t6.run(Unknown Source) [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - ... 1 more [l] 2023-02-03 00:05:03 INFO b.BurpProcess.license.license-bsp - 2023-02-03 12:04:56: Rescheduling heartbeat more promptly due to failure [l] The time those entries start to show matches with a peak of CPU usage, according to our metric collectors. CPU usage goes back to normal after a minute or so, but that log does not show activity anymore. I've tried to diagnose potential network errors between agents and the server, but it seems to be working fine. They only way I've found to make the scanning machine responsive again is to kill all running chromium processes, kill all java processes, and restart the service. Kind regards, Javi

Maia, PortSwigger Agent | Last updated: Feb 03, 2023 06:25PM UTC

Thank you for your message. So that we can investigate your issues further, can you please email support@portswigger.net with a full copy of the enterpriseAgent.log along with a copy of the Support Pack please? Please include all logs in the support pack for the past ten days, so we have as much history as possible. Can you also let us know the operating system you are using for the Enterprise server and the scanning machines please?

JAVIER | Last updated: Feb 06, 2023 10:47AM UTC

Hi, I have just sent the support pack and log files from all crashed machines. All machines in our deployment are Debian 11, fully upgraded. It looks like more agents have had some trouble during the weekend. They show as "disconnected", but the agent seems to be running in the machine, there are lots of chromium processes, and connectivity towards the BE server looks good. The stack trace in enterpriseAgent.log looks like this: 2023-02-04 19:49:14 WARN o.g.jersey.jetty.JettyHttpContainer - Unable to close response output. [qtp1491243406-5862] org.eclipse.jetty.io.EofException: null at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277) at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381) at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:828) at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241) at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223) at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:555) at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:1009) at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:1086) at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285) at org.eclipse.jetty.server.HttpOutput.close(HttpOutput.java:638) at org.eclipse.jetty.server.Response.closeOutput(Response.java:909) at org.eclipse.jetty.server.Response.completeOutput(Response.java:920) at org.glassfish.jersey.jetty.JettyHttpContainer$ResponseWriter.closeOutput(JettyHttpContainer.java:352) at org.glassfish.jersey.jetty.JettyHttpContainer$ResponseWriter.commit(JettyHttpContainer.java:339) at org.glassfish.jersey.server.ContainerResponse.close(ContainerResponse.java:390) at org.glassfish.jersey.server.ServerRuntime$Responder.release(ServerRuntime.java:740) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:364) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:258) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) at org.glassfish.jersey.jetty.JettyHttpContainer.handle(JettyHttpContainer.java:175) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:190) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at net.portswigger.enterprise.r.o.p.handle(Unknown Source) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Broken pipe at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at java.base/sun.nio.ch.SocketDispatcher.writev(Unknown Source) at java.base/sun.nio.ch.IOUtil.write(Unknown Source) at java.base/sun.nio.ch.IOUtil.write(Unknown Source) at java.base/sun.nio.ch.SocketChannelImpl.write(Unknown Source) at java.base/java.nio.channels.SocketChannel.write(Unknown Source) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:274) ... 48 common frames omitted Kind regards, Javi

Maia, PortSwigger Agent | Last updated: Feb 06, 2023 02:35PM UTC

Thank you for the details. I can confirm that we have received your logs and will respond to your email once we have reviewed them.

You need to Log in to post a reply. Or register here, for free.