Most modern secure web gateways do a great job of web filtering and web content sanitizing. When a user attempts to browse to a site that is not allowed, they are typically presented with a block screen letting them know that the site is blocked, and/or it is suspect and that they should not proceed.
Modern websites are comprised many elements from many different domains, so blocked items are not always rendered and presented to the user. For instance, if an embedded javascript is not allowed to download and execute, there is no visible indicator to the user that something is wrong, other than the web page not rendering correctly or behaving in an undesirable way.
However, these ‘behind the scenes’ block actions are logged by your secure web gateway, and can therefore be reported on with WebSpy Vantage.
Reporting on Blocked Traffic for Troubleshooting
Visibility into what is transparently being blocked can help troubleshoot why a particular site is not working for a user. For example, I had a line of business application that refused to load, and it turns out the that application needed to retrieve fonts from the Internet to render the pages. These font files we’re being blocked at the firewall, so the application did not work correctly.
Seeing all the network requests made by a web browser along with what is allowed or denied can be achieved using your browser’s developer tools, but this often requires you to be on the user’s machine and logged in with their authentication details to observe and troubleshoot the problem. Although it may tell you what network requests are failing, it will not necessarily tell you why.
Log file analysis and reporting lets you investigate these issues in bulk (for all users) and offline (without relying on live tests).
Reporting on Blocked Traffic for Security
One of the concepts in physical security is to look for evidence of tampering or attempted tampering. This can indicate that someone is ‘casing the joint’ and looking for vulnerabilities. Seeing evidence that someone tried to pry open a window lets you know that something is up and that you need to strengthen your defenses.
Reporting on unexpected blocked traffic provides the same type of insight on your digital perimeter. There are various reasons a web gateway or firewall may block traffic and knowing the context around a denied request can provide you with insight to help strengthen your security policies.
Visualize the following two examples:
- A few connection attempts from a single IP to known Apple servers from a user-agent that contains ‘itunes’ that was not able to authenticate against the proxy. Traffic was blocked because authentication was required.
- A few connections from multiple internal IP addresses to an unknown and unclassified IP address with a blank user agent string. Traffic was blocked because authentication was required.
Even though the reason for the blocked traffic in both examples is the same (failed authentication), you would not classify these two as posing the same amount of risk. The first example is most likely the iTunes application communicating with Apple, but the second example is much more suspicious.
Analyzing Blocked Traffic in Log Files for Suspicious Activity
One fundamental weakness in today’s malware and botnets is that they need to connect back to a command and control server to retrieve instructions and/or payloads. This action typically leaves a log trail on firewalls and proxies that you can investigate.
I am going to step through analyzing a log file with WebSpy Vantage, and zoom in on traffic that raised some flags and warranted further investigation.
After importing my firewall log files into a WebSpy Vantage Storage (in my case, I am using the web filter logs from Sophos UTM), I ran an ad-hoc analysis on the Storage, navigated to the Action summary, and drilled down into block events.
Here, I looked for traffic that did not attempt to authenticate (identified with a blank Active Directory Domain), and drilled down into the blank entry to look further into the Anonymous traffic.
Next, I selected the Reputation summary and drilled down into traffic destined for locations with a known malicious reputation. This revealed traffic that had been categorized as both Malicious Sites and Hacking/Computer Crime by the Sophos UTM’s web filter.
Drilling down into the Hacking/Computer Crime category revealed two source IPs that each made exactly 36 requests totaling exactly 111KB. This was either an amazing coincidence or they were programmatically generated requests.
The User Agent string is also suspicious or fraudulent as it only showed a value of “Mozilla/4.0”. All the commonly used modern web browsers have a much more comprehensive user agent string.
The Hour summary also indicated odd behavior, as this happened between midnight and 1AM.
The Site URLs are the final definitive piece of the puzzle as the URLs are linked to known malware.
The blocked traffic did not trigger any of the native alerts on Sophos UTM because the traffic was blocked and did not contain any malware. In essence, it is ‘business as usual’ for the UTM, however you may want to look at the two computers that tried to make these requests in more detail, and perhaps investigate if any other machines have tried to access these URLs.
Conclusion
This is a straightforward example that relied on the gateway’s ability to flag blocked and suspicious traffic. But sometimes, you may be need to look a little broader and/or deeper to convict the traffic as having malicious intent. For example, some firewalls may only log HTTPS status codes and source/destination IP addresses, in which case WebSpy Vantage can be an invaluable tool to resolve status codes into meaningful names, and resolve IP addresses to hostnames.
Having a picture of not only the allowed traffic leaving your network but also the denied or blocked traffic can help you identify if and when a compromise is brewing. Being able to interactively drill down into a massive log file with a comprehensive log analysis and reporting framework such as WebSpy Vantage is an essential part of any network security professional’s toolkit.
WebSpy Vantage supports over 200 log file types and you can get started easily by downloading the free 30 day trial.
