Watchguard XTM Firewall and UTM Appliance – High CPU Usage in scand process causes lag and typing delay in Remote Desktop Sessions (RDP). You may find that remote users report a lag with Remote Desktop Sessions, freezing sessions, black screen and random disconnections. At around the same time users report these issues you may find that the CPU usage of the scand process on your Watchguard has increased to 100% and the majority of the activity is attributed to the scand process. You may be able to recreate this issue by browsing websites that utilise lots of Adobe Flash or Media Content as GAV will need to scan all these elements of the web page. Login to the Watchguard System Manager and then open Firebox System Manager click on Status Report and scroll down the report until you find the Process List (Screenshot Below). This information will automatically update every 30 seconds so you can see the %CPU column will change and update every 30 seconds. The top value system shows the overall CPU utilisation and if you look further down you can see which sub processes are actually occupying the CPU time and making up the overall system usage. In the screenshot below we can see that system is showing 100 % CPU Usage and then further down we can see that the scand process is accounting for 90.99% of this. When the CPU Usage reaches 100% on the Watchguard unit it may stop forwarding other traffic and this accounts for the lag and jitter we see within the Remote Desktop Session. Other time sensitive traffic such as VoIP or SIP traffic may also be affected by this issue as the packets are delayed whilst the Firewall recovers from the resource exhaustion. Users may also report that web pages are slow to load at the time these issues occur where the GAV process is still dealing with the other requests.
Resolution/Workaround:
You can try disabling the GAV (gateway antivirus) for the HTTP and FTP Proxy to ensure that this is the actual cause of your issues, if the problem subsides then you may need to consider updating the XTM OS to the latest release i.e. 11.5.2 and/or adjusting the GAV policy so that it does not scan some content i.e. Images/Text within websites. You may also need to consider opening a support case with Watchguard to make them aware of this issue, if you have a large number of users then you may even need to consider upgrading your XTM appliance to a larger unit i.e. XTM 23 to XTM 505 or XTM 22 to XTM330 to provide additional processing power (CPU) and system resources to cope with the additional anti-virus scanning requirements.
Cannot browse some sites and logs report GAV job open failed (failed to connect to scand at scand)
You may find that you cannot access or browse some websites when you are using a Watchguard XTM Firewall or UTM device and the GAV (gateway antivirus) is enabled. When you review the appliance logs you see the following event logged GAV job open failed (failed to connect to scand at scand). In this instance the anti-virus proces or component of the XTM device has probably crashed or stopped responding.
Resolution/Workaround:
You might be able to permantently resolve this issue by upgrading to a newer XTM OS i.e. 11.4.2 to 11.5.2 or you may simply need to apply the latest CSP release for the XTM OS build you are using i.e. 11.4.2 CSP9 (Service Pack). Newer OS releases and Service Packs often included fixes for these sorts of GAV issue.
A workaround would be to schedule a reboot of your Watchguard XTM appliance, this will reset the GAV (gateway antivirus) and should allow pages to load correctly again.
Watchguard XTM 2 Series, XTM 3 Series, XTM 5 Series, XTM 8 Series – Fireware XTM OS 11.5.2 Build # 337008 has been release and can be downloaded via the Watchguard website http://www.watchguard.com/
This major XTM OS Release introduces the following changes/improvements:
Significant enhancements introduced in this release include:
Application Control now offers a deny message to users whose HTTP requests are blocked
New Advanced Search for log messages added to Log and Report Manager
Mobile VPN with SSL now supports multiple authentication servers and Active Directory authentication domains
Management Server device configuration template improvements
Ability to create a device configuration template from an existing configuration file
Inclusion of hosted WebBlocker Server settings in template for XTM 2 Series and XTM 33 devices
Policy order in a template is preserved when you apply the template to an XTM device
Support for FireCluster for XTM 330 appliances
Minor enhancements include:
The ability to select the port used to send email notifications from the Log Server
An updated list of trusted Certificate Authorities for proxies
Diagnostic log messages for the Terminal Services Agent and TO Set Tool
SMTP-proxy TLS encryption rules now limited to a maximum of 200 bytes
The following bugs/issues have also been resolved in this release
General
After you reboot your XTM device, you no longer see the warning “Cannot create directory `/etc/wg/tmp’”. [62883]
A problem that caused some XTM 8 Series devices to reboot because of a kernel crash has been resolved in this release. [64465]
ICMP redirects are now handled correctly after you upgrade from Fireware XTM v11.4.2 to v11.5.2. [64589]
Several problems that resulted in device configuration changes requiring a reboot to take effect have been resolved in this release. [64201, 64763]
A memory leak in the SNMP daemon has been corrected. [63860]
This release resolves problems that occurred during an upgrade from Fireware XTM OS v11.3.x. [64815]
Networking
When you configure policy-based Dynamic NAT with the Source IP option, you no longer need to add the Source IP address as a secondary address on the external network. [64292]
A default route is now correctly added when you enable Eth0 as an external interface. [63588]
This release resolves a problem with server connection stability when you use an SNAT load balancing action in your configuration. [64280]
It is no longer necessary to reboot your XTM device after you change the wireless guest network. [64415]
Dynamic NAT now supports nested aliases. [58991]
Dynamic NAT now works correctly when configured with an IP address range. [45005]
Branch office VPN tunnels can now successfully be created to use 1-to-1 NAT configured with an IP address range. [65105, 64480]
The multi-WAN Link Monitor now works correctly. [62535, 64817, 61487]
Multi-WAN now works correctly when configured in Routing Table mode, with the Link Monitor configured to use both ping and TCP using domain name. [61564]
XTM 2 Series Gigabit interfaces are now correctly labeled in the Firebox System Manager Status Report. [60621]
QoS marking now works correctly in proxy policies. [63518]
Multicast traffic can now pass through a branch office VPN on an XTM device configured in drop-in mode. [62234]
Proxies
The default proxy trusted CA list has been updated to match the ones in most major popular browsers. [64438]
A problem that caused some HTTP downloads to stall when you use the HTTP proxy with IPS enabled has been resolved. [63136]
This release includes improvements to the clean-up of stale proxied connections that could prevent new connections from being allowed. [63574, 64519]
FireCluster
A problem that caused the master XTM 8 Series device in a FireCluster to reboot because of a kernel crash has been resolved. [63683]
High UDP traffic levels through a branch office VPN tunnel that terminates to a FireCluster no longer cause the FireCluster to reboot. [64251]
The FireCluster synchronization timeout has been increased to prevent synchronization failures. [63231]
A problem with the maintenance of the static ARP list for an active/active FireCluster has been resolved to improve FireCluster stability. [64440]
This release includes improved proxy debug log messages for FireCluster users. [63793]
A problem that caused a FireCluster to crash and reboot has been resolved. [64669]
A problem has been resolved that sometimes caused the FireCluster master to reboot when you used the CLI to remove a member. [64518]
The Status Report cluster load balancing algorithm label has been corrected from “Lease Connections” to “Least-connections”. [64684]
A problem that caused DHCP leases to fail to synchronized with the backup member has been fixed. [64148]
The backup member no longer crashes when the FireCluster is under heavy proxy traffic load and a manual failover is initiated. [64226]
A problem that caused some active/active FireClusters to fail when configured with branch office VPN has been resolved. [64909]
In a FireCluster environment, log messages are now consistently and correctly sent to the Log Server. [64983]
Mobile VPN with SSL
The Mobile VPN with SSL Mac client now correctly removes the previously assigned DNS server addresses after it is disconnected. [64418]
Mobile VPN with IPSec
A problem that resulted in a stack trace with the error message “Iked stack trcace eip=0x080c4013” has been resolved. [65026]
Branch Office VPN
Packets with a total data size value between 1507 and 1538 now correctly pass through a branch office VPN tunnel. [63764]
You can now correctly configure 1-to-1 NAT with an IP range for a branch office VPN tunnel. [65105]
Authentication
A problem that caused the SSO Event Log Monitor to crash has been fixed. [64824]
WSM & Management Server
The Apache httpd server bundled with WSM has been upgraded to version 2.2.21. [58220, 64556]
IPS is now correctly shown as enabled/disabled within an XTM template. [64124]
The “Add Device” wizard now correctly detects and configures the gateway device with the private IP address of the Management Server. [64498]
Policy Manager
The backup archive password is no longer recorded in the log file. [64088]
Attempts to save configuration changes for new features to an XTM device running Fireware XTM v11.4 are now detected and handled correctly. [64609]
Web UI
The WatchGuard Authentication policy is no longer deleted when you disable Mobile VPN with SSL from the Web UI or CLI. [65234]
Gateway AV can now be enabled in an SMTP proxy action from the Subscription Services > Gateway AV > Configure page. [62464]
Security Services
A problem that caused Gateway AV to fail when performing archive file scanning has been resolved. [64898, 65114]
Logging & Reporting
The log level setting now works correctly for DHCP log messages. [57096]
Corrupt Log or Report Server database tables no longer cause an upgrade to the new v11.5.x database schema to abort. [64726]
This release corrects multiple issues with restoring log files from backup archives. [64141, 65173]
You can now export log search results from the Log and Report Manager. [63198, 42489, 62608]
The Log and Report Manager Logs > Devices list now includes the serial number for each device. [63786]
WatchGuard Server Center no longer resets the start time to a previous value when a report schedule is edited. [64980]
Users with permission to view only reports can now see all the reports they have access to. [65127]
The Alarms Summary report is now available in the Log and Report Manager Dashboard and Devices sections. [64420]
The default pivot (Users or Hosts) can now be set for the graphs available on the Log and Report Manager Dashboard tab. [65098]
Percentage and Total count statistics have been added to a number of summary reports in Report Manager. [62981, 63019]
WatchGuard System Manager Version 11.5.1 – Update 1 was released recently to correct some XSS Flaws that were reported to WatchGuard, it also corrects two internally identified vulnerabilities. You can obtain the latest version via the WatchGuard support portal.
