Patching and Testing
Testing and Patching: The Advantages and Risks of Both
Security vulnerabilities will eventually impact every publicly accessible system given enough time. It is unreasonable to expect over a thousand programmers to perfectly cooperate with each other and manage a code base of gigabytes of code written in different languages so perfectly that no vulnerabilities are ever created. Luckily, most software engineering firms are willing and capable of fixing exploitable software after release. These fixes are often released as patches, but occasionally they will just release an entire replacement version of their software which doesn’t contain the vulnerability. In this article we will cover the practical advantages, risks, and methods of patching and testing as it relates to critical systems. We will also be going over a couple alternatives to path testing that are in common use.
This process does have some limitations, it both requires that a vulnerability be both discovered and shared with the company that is responsible for the code base. It also takes companies some time to provide patches, as not all vulnerabilities can be fixed with a simple solution. As such, many vulnerabilities exist which are waiting to be discovered, and some have already been discovered and have not been made public yet. For this reason, it is never safe to assume that patching is all that is required to properly secure a system, but good patching will provide a massive head start for the security of your computer systems and network infrastructure.
Patching has always been the primary method for removing security vulnerabilities from your system. But patching does come with some risk to the availability and stability of the software it patches. Luckily this risk can be completely mitigated through a simple process: Testing. Testing new updates and patches prior to installation will allow you to avoid exposing your primary infrastructure to potentially harmful or undesirable changes. A patch which fixes one bug by creating a worse bug, isn’t really all that useful.
Luckily, testing is a fairly straightforward process. You just need three things: a test network, a test environment, and a testing protocol. A test network is just a simplified version of your primary network infrastructure. A test environment is an exact copy of all the important systems and services on your primary network but without any of the personal or critical information included. And a testing protocol is a list of important tests which you should perform before validating any new patches for installation.
The first two, the test network and testing environment, are fairly easy to create. However, the testing protocol will be a little harder to design, as each system will require a different set of tests based upon your network and system layout. You will likely need to design a testing method for each of the primary and secondary services and functions that your infrastructure provides. Even though this is a lot of additional workload, it can mean the difference between normal operation and a complete catastrophe. A testing environment also allows you to provide important configuration details in a bug-report to the developers of whichever software has a malfunctioning patch, without requiring you to give away any critical security or client information to them.
The other primary consideration is the amount of time to spend testing. This is often determined by your, or your company’s, perceived importance of the system being patched. This could be as simple as a test install to make sure a system isn’t crippled, followed up by an immediate patch. Or it could involve a week of testing dozens of inter-system functions to make sure that a patch doesn’t cause a system failure. Patching nuclear reactor control software should probably be a slow and meticulous process, but patching a website that just serves a music playlist can probably be done in minutes.
These methods for testing and patching are meant as the ideal. But unfortunately, even the best plans won’t survive contact with the enemy. In most cases, the enemies of any systems or network engineer are budget and time. You can’t always afford to spend enough time testing each possible scenario, as such there are a couple less optimal methods to mitigate some of the risks involved with patching. These two methods are delayed patching and revertible patching.
Delayed patching is exactly what it sounds like, setting a delay before installing new patches with the hopes that others will find any problems and report them to the developers before your preset delay expires. By delaying your patches by a week, you can prevent a lot of small errors from making it though the software development cycle. This does come with its own risk, as you will be running a vulnerable system for the full duration of the delay cycle. You can monitor for exploitation and make regular backups to help reduce the risk, but it will remain a potential hazard for as long as it takes you to patch.
Revertible patching is the high-risk alternative to delayed patching. You simply patch immediately and hope that nothing fails. If something does fail, you revert the patch as quickly as possible. This can be done through system and database backups, but it does risk service outages as well as system integrity problems. This is by far the cheapest and easiest solution, but it does provide a much higher risk of failure and it requires you to store far more backups than normal in order to be a reliable alternative.
