Category Archives: CSP8

Watchguard XTM Firewall and UTM Appliance – High CPU Usage in the GAV (gateway anti-virus) scand process causes lag and typing delay in Remote Desktop Sessions (RDP) and SIP or VoIP latency issues

Watchguard XTM Firewall and UTM Appliance – High CPU Usage in scand process causes lag and typing delay in Remote Desktop Sessions (RDP).  You may find that remote users report a lag with Remote Desktop Sessions, freezing sessions, black screen and random disconnections.  At around the same time users report these issues you may find that the CPU usage of the scand process on your Watchguard has increased to 100% and the majority of the activity is attributed to the scand process.  You may be able to recreate this issue by browsing websites that utilise lots of Adobe Flash or Media Content as GAV will need to scan all these elements of the web page.  Login to the Watchguard System Manager and then open Firebox System Manager click on Status Report and scroll down the report until you find the Process List (Screenshot Below).  This information will automatically update every 30 seconds so you can see the %CPU column will change and update every 30 seconds.  The top value system shows the overall CPU utilisation and if you look further down you can see which sub processes are actually occupying the CPU time and making up the overall system usage.  In the screenshot below we can see that system is showing 100 % CPU Usage and then further down we can see that the scand process is accounting for 90.99% of this.  When the CPU Usage reaches 100% on the Watchguard unit it may stop forwarding other traffic and this accounts for the lag and jitter we see within the Remote Desktop Session.  Other time sensitive traffic such as VoIP or SIP traffic may also be affected by this issue as the packets are delayed whilst the Firewall recovers from the resource exhaustion.  Users may also report that web pages are slow to load at the time these issues occur where the GAV process is still dealing with the other requests.

Resolution/Workaround:

You can try disabling the GAV (gateway antivirus) for the HTTP and FTP Proxy to ensure that this is the actual cause of your issues, if the problem subsides then you may need to consider updating the XTM OS to the latest release i.e. 11.5.2 and/or adjusting the GAV policy so that it does not scan some content i.e. Images/Text within websites.  You may also need to consider opening a support case with Watchguard to make them aware of this issue, if you have a large number of users then you may even need to consider upgrading your XTM appliance to a larger unit i.e. XTM 23 to XTM 505 or XTM 22 to XTM330 to provide additional processing power (CPU) and system resources to cope with the additional anti-virus scanning requirements.

Watchguard XTM High CPU Usage scand
Watchguard XTM High CPU Usage scand

Watchguard XTM Firewall UTM Device – Cannot browse some sites and logs report GAV job open failed (failed to connect to scand at scand)

Cannot browse some sites and logs report GAV job open failed (failed to connect to scand at scand)

You may find that you cannot access or browse some websites when you are using a Watchguard XTM Firewall or UTM device and the GAV (gateway antivirus) is enabled.  When you review the appliance logs you see the following event logged GAV job open failed (failed to connect to scand at scand).  In this instance the anti-virus proces or component of the XTM device has probably crashed or stopped responding.

Resolution/Workaround:

You might be able to permantently resolve this issue by upgrading to a newer XTM OS i.e. 11.4.2 to 11.5.2 or you may simply need to apply the latest CSP release for the XTM OS build you are using i.e. 11.4.2 CSP9 (Service Pack).  Newer OS releases and Service Packs often included fixes for these sorts of GAV issue.

A workaround would be to schedule a reboot of your Watchguard XTM appliance, this will reset the GAV (gateway antivirus) and should allow pages to load correctly again.

HTTP and HTTPS requests or traffic to a Windows Vista, Windows 7, Windows Server 2008, SBS 2008, Windows Server 2008 R2 or SBS 2011 machine may exhibit increased latency if the connection is through a network load balancer

If you utilise Microsoft Internet Information Services IIS or an application that uses the System.Net.HttpListener class is installed or running on one the operating systems below, and you have a Network Load Balancer then you may find that Increased latency occurs on HTTP and HTTPS requests and traffic.

This issue occurs because the HTTP and HTTPS requests from clients can include zero length data in the SSL records, certain server-side variables do not update correctly in this instance and Http.sys leaves the connection in the CLOSE_WAIT state.  This intern exhausts the open connection limit can introduce latency, timeouts and connection problems.

Affected Operating Systems:

Microsoft Windows Vista

Microsoft Windows 7

Microsoft Windows Server 2008

Microsoft Small Business Server 2008 – SBS 2008

Microsoft Windows Server 2008 R2

Microsoft Small Business Server 2011 – SBS 2011

The Microsoft Knowledge Base Article KB 2634328 includes further information on this issue and provides an updated version of Http.sys that corrects the issue http://support.microsoft.com/kb/2634328

Watchguard XTM 2 Series, XTM 5 Series, XTM 8 Series – Fireware XTM OS 11.4.2 – CSP8 Build # 328728

Watchguard XTM 2 Series, XTM 5 Series, XTM 8 Series – Fireware XTM OS 11.4.2 – CSP8 Build # 328728

11.4.2 – CSP8 Build # 328728 Resolves the following issues:

BUG60697: Import of customer CA Root causing FSM to show java error message

BUG63136; Improved proxy connection handling to prevent file transfers from stalling during periods of high proxy

connections.

BUG63222, BUG63468,BUG62925,BUG62934: Resolved issue resulting in proxy processes becoming stuck. As a result of

the stuck proxy process a low memory state will occur causing the Firebox to stop passing all traffic.

BUG62358: Resolved issue causing throughput degradation due to excessive ARPing from the Firebox when using

FireCluster Active/Active.

BUG63702: Firebox will not failover to secondary log server when log collector’s cache is full.

BUG62024,BUG63035: Resolved issue which caused a PPPoE process to stop responding resulting in the PPPoE connection

not returning after down event from ISP.

BUG56483,BUG62518: FC has to be rebooted every time for 3rd party cert to take effect after a failover event.

You can request 11.4.2 – CSP8 Build # 328728 from Watchguard Support by logging a support case online, they should then be able to provide an ftp download link and appropriate credentials.

Please note that Watchguard CSP releases are cumulative so you should only need to apply the latest to ensure that you also have any previous fixes.