Creating Disaster Recovery and Incident Response Plans

GPS

If your organization needs to create a DR plan from scratch, there are a lot of great resources that can guide you. Ready.gov has a document that describes how to build out an IT DR plan. For a more comprehensive look at DR, take a look at the book, "Modern Data Protection: Ensuring Recoverability of All Modern Workloads," by W. Curtis Preston.

Why is that? Most DR plans are written around having to restore a single server or cluster of servers. One scenario might be that a Microsoft Exchange server crashes and is unrecoverable. The DR plan says to take the most recent backup and restore from that point. Recovery from backup takes three hours and the last backup was completed 30 minutes before the server crashed, so the RTA is 3.5 hours. An example of RPO and RTO is shown in the diagram to the right.

Deep dive

There’s a quote from Homer Simpson that’s often overused in IT security circles, “Oh, they have the Internet on computers now.” The quote wonderfully captures how surprised people are by things that seem like a natural progression to people who understand a topic deeply. In this case, more and more ransomware groups are creating versions of their ransomware specifically designed to encrypt VMware ESXi systems.

Why? Because if a ransomware actor can encrypt an ESXi server, they can instantly remove dozens or hundreds of machines from the network, creating significantly more chaos. Being able to knock an ESXi server offline allows the attacker to do a lot of damage in a shorter period of time, not just because of the number systems, but also because of the type of data stored on ESXi servers. ESXi systems usually store backups, file storage, code repositories, databases, and other critical files making their encryption a serious business disruption.

But there’s another advantage: Many organizations have virtualized their DR environments. Whether it’s a hosted environment or a Disaster Recovery as a Service (DRaaS), organizations can save a lot of money by going virtual and can restore servers very quickly after a ransomware attack. However, if the DR site is reachable from the network, the ransomware attacker can use that connectivity to access and encrypt the DR servers. This isn't a hypothetical scenario. Unfortunately, it has happened to several ransomware victims.

Organizations relying on virtual servers for DR should ensure those servers are fully segmented from the live network, to avoid encryption by a ransomware group. In addition, these systems should have the same security systems installed and monitoring that are applied to live servers. DR servers are critical to ransomware recovery and should be monitored as such.

... in the DR plan is that the decision requires leadership input, and the time to ask this question is not after a catastrophic ransomware event. There will be a lot of different groups making demands of the DR team in the aftermath of a ransomware attack, and every group will think their systems are top priority. Having a clear, prioritized list of systems that need to be restored and in what order allows the DR team to get to work without having to deal with the natural chaos that’s part of any recovery from a ransomware attack.

GOOD TO KNOW

Cyber insurance companies are getting more selective about who they cover and whether they pay a ransom in the event of a ransomware attack. Most policies renew annually. Part of the cyber insurance policy renewal process should involve updating the DR plan to confirm that cyber insurance will still pay the ransom in the event of a ransomware attack.

If they successfully strike a victim network, a lot of the questions that incident responders try to answer are immediately known. An incident responder walking into a ransomware attack might know the organization that infiltrated and what they want, but what the incident responder might not know is:

GOOD TO KNOW

Some ransomware groups are better at branding than others. That sounds like a silly statement, but it’s true. Although most of the time, incident responders can look at a ransom note and know which ransomware group encrypted a network, that’s not always the case. Some ransomware groups simply steal the text of ransom notes from other groups and don’t include a name or anything else that would help the incident responder identify which ransomware was used in the attack. Fortunately, there are services such as ID Ransomware and No More Ransom that allow victims to upload a ransom note or encrypted file to determine which ransomware was used in the attack. Keep those sites bookmarked!

There are a lot of basics that need to be defined in an IR plan, starting with: What is considered an incident? Obviously, a ransomware attack is an incident. In fact, a modern ransomware attack is likely made up of at least three separate incidents (depending on how an organization defines an incident):

For instance, many organizations would consider gaining access to an Active Directory server an incident in and of itself. The point is, the threshold for what types of events or collection of events qualifies as an incident should be well-defined within an IR plan, as well as what the response should be.

SCHOOLHOUSE

Nation-state groups often monitor email communication for indications that an organization is on to them. Often, they’ll specifically track an email thread that might reveal their presence and look for comments like “let’s take this conversation offline” or “let’s switch over to Signal” to indicate it’s time to back out (or destroy everything, depending on the group and their goals).

Cybercriminals have picked up on this tactic as well, quite by accident. It turns out there’s a lot of juicy and embarrassing information sent via email, so stealing email communication for extortion purposes makes sense. But it’s also a great way to track whether your ransomware attack has been noticed during the reconnaissance phase. Most organizations don’t use email as their primary form of communication during an IR, preferring ticketing systems, but for critical incidents that may involve communication outside of the core security team, there should be an out-of-band communication plan, which should be in place before an incident is detected.

IR teams are going to need access to logs from a number of different sources within the organization. The IR plan should document how to get this information to the team as easily as possible. Some (but not all) of the information that the IR team will likely need access to include:

... that the IR team needs, depending on the type of ransomware attack. Not every organization collects all these logs, but the IR plan should document which systems or servers the logs are being collected from, how long those logs are stored, and how to provide third parties with access to those logs. It’s worth noting that some IR companies will want the raw logs sent to them for analysis because they have their own tools for managing logs. The IR plan needs to allow a large amount of log data from a variety of sources to be extracted, transferred to a portable drive, and delivered to the IR team for analysis. The process determining the format needed should be discussed with the IR company when the retainer is signed.