Ransomware isn’t sexy but it’s certainly an important topic in today’s IT security landscape, with unprecedented growth and relentless evolution, organisations need to constantly keep one step ahead of bad actors eager to make a buck by exfiltrating and encrypting your important data.

Ransomware attacks mainly occur because IT is a complex and everchanging environment, organisations are busy modernising their applications from monolithic designs to new highly distributed container-based services. The remote workforce is now connected more than ever, with the ability to access data hosted across multiple cloud providers, connecting from virtually any device, at any hour of the day. IT departments are being asked to do more with less each and every year while ransomware attacks are on the rise and becoming more costly than ever before.

Beyond encryption of data, some ransomware is taking it a step further and making ransom of leaking data, this is otherwise known as data exfiltration. Unfortunately, stopping ransomware prior to an attack is difficult and at best, inconsistent. No single product or service has all the solutions to the challenges raised by these attacks, instead, it’s recommended to take a multi-layered approach. Apply best practices, keep systems up to date, enforce good data hygiene, configure event logging, and identify anomalies (indicators of compromise) to provide the best chance of discovering an attack as early as possible.

Interestingly, the majority of modern ransomware attacks aren’t masterminded by groups with unlimited resources and vast expertise. Instead, they’re often run like a business, renting hosted infrastructure (on the dark web), buying compromised credentials from an access broker, procuring scripts to target and compromise an environment in order to get their Bitcoin as quickly as possible, and then moving onto the next target.

Ransomware has become profitable enough that entire business models have been built around it, Ransomware as a Service or RaaS for short, where developers sell or lease their ransomware variants to buyers. For better or worse, these RaaS subscribers don’t want to waste time spending weeks or months checking every exploit or vulnerability in a target network. Instead, they are looking for the fastest possible return on their initial investment. That is, they want to run the same script they successfully ran last week against another target and collect the ransom in the shortest turnaround possible, if it’s too difficult they will move on to another easier target.

Applying best practices such as those discussed earlier will help create friction against these attackers, the more friction the more likely an attacker is to move on to somewhere else. Specifically for data exfiltration attacks, encrypting backups helps add friction against these types of attacks, though remember it’s possible for ransomware to encrypt these encrypted backups.

Prior to an attack, most organisations often believe they are already protected by the systems they’ve had in place for years and sometimes even decades, solutions such as high availability, off-site backups and DR environments. It’s often said that “backups are your last line of defence” but running backups and clicking the restore button during a ransomware attack is no longer a sure-proven strategy for successful recovery. Why? It comes down to three main areas.

IT environments are large and ever-changing

• Unencrypted backups were exfiltrated by an attacker who is threatening to leak sensitive data.
• Lack of monitoring of the recovery resources to ensure they were ready at a moment’s notice
• Lack of available capacity to recover
• Over-reliance on snapshots, data within snapshots are immutable by definition but that doesn’t mean the snapshot itself can’t be deleted.*

Recovery is complex and can fail easily

• Backups were not verified and tested regularly. Simply checking the backup logs is not good enough. Ransomware recovery is not the time to find out about backup storage corruption.
• The ransomware attack has encrypted all backups and replicated data
• Malware can live in backups for weeks or months before triggering, restoring backups simply results in ransomware re-activating.
• Backup administrators were not familiar with the recovery process
• Creating an isolated environment to clean recovered data is complicated and time-consuming
• The backup encryption password was lost or even changed by the attacker

DR planning takes time and resources

• Pressure to restore services quickly and lack of automation often lead to mistakes and burnout
• Not knowing what to restore first
• The DR site was not ready
• Not following the 3-2-1 rule, let alone the 3-2-1-1-0 rule

*Several storage vendors are now offering protection against snapshot deletion such as PureStorage Safemode, I wrote another article about safemode snapshots here.

The challenge is, that the strategy of just “restoring from backups” oversimplifies the process and causes many organisations to make assumptions about their ransomware recovery capabilities. The nature of assumptions though, often prove to be false, leading to either the ransom having to be paid or data loss. 

Ransomware best practices extend beyond just recovering from backups, below are some suggestions to improve the odds of success. They have been structured around 3 main topics; before, during and after an attack.

Before an attack

Data Hygiene – One of the most important aspects to focus on is good data hygiene. Whenever a good hygiene program is implemented, the bar is raised to make it harder for the threat actors to launch an attack. Some of the most effective data hygiene practices include but are not limited to, good password management, multi-factor authentication (MFA), ensuring anti-virus and anti-malware software and signatures are up to date, encryption of critical data (backups especially) and using least privilege principles.

Authentication – Beyond MFA, look at credential vaulting. A credential vault is a single place to store and manage all of an organisation’s privileged credentials in a secure location.

Zero Trust Approach – Implement zero trust security which assumes that an attacker is already present in the environment. Isolate VMs with network segmentation to mitigate attacks and prevent lateral movement. Implement strict identity management policies to meet the principle of least privilege. Assume no users can be trusted on the network and explicitly verify each access request. Always assume a breach and run consistent environment scanning and access verification. Protect firewalls with just-in-time and just-enough-access (JIT/JEA) principles.

Awareness training – Not just for end-users but also for executives. Training can help mitigate the risk of a ransomware attack within the organisation while also providing knowledge to spot suspicious behaviour. Run table-top exercises with executives at least once a year, let everyone know how ransomware can play out, or better yet, role play it.

Logging – Collect the right logs, ensure the logs are secured and easily searchable. Look at systems that can provide a holistic overview of the organisation from a single point of view. By aggregating all logs in a centralised location, it’s easier to correlate data. Collect logs from the security, network, server, and endpoint devices.

False Positives – Reducing the signal-to-noise ratio is very important, false positives can be common and often cause more harm than good. Ransomware attacks can start encrypting data in seconds so it’s vital to have systems in place to detect the attacks as they are occurring as accurately as possible.

Prior preparation prevents poor performance – To avoid the worst-case scenario,  a plan must be in place that includes verified, tested and secure backups that can be restored quickly.  It’s important to remind ourselves and everyone that backup infrastructure is part of the overall cybersecurity defence plan and can be the final option for getting back to, or staying in, business. Be prepared that the organisation might never be able to get back to where it came from, data may be lost forever. 

Engage with technology vendors early, they have the expertise and global reach to help you leverage their products to improve the chances of a fast recovery.

Orchestration / Automation – Implement automation solutions to expedite recovery. By orchestrating recovery, the manual intervention required to recover can be drastically reduced. A lot of stress and emotions arise during a ransomware attack, people are rushing to figure out which machines are infected, what can remain running and what needs to be isolated. As is human nature, when people are emotional and stressed, mistakes are more likely to be made.

Malware scanning – These systems are already prevalent in most organisations but they typically only target production data. Consider scanning backups for malware, this helps avoid restoring hibernating malware back into production during a recovery.

Daily Change Rates – Monitor your backup change rates, if you typically see 10% daily rate of change and suddenly you see 20x the amount of change, that is an indicator for encryption taking place.

Immutability – Immutable backups are a key defence against ransomware encryption and can help protect against insider threats such as rogue administrators. There are many methods for achieving immutability, some of which provide more protection than others. Technologies such as storage snapshots, object locking, immutable file systems, WORM tapes and storage systems managed by service providers are just some of the technologies available today.

Remember that while immutability is a critical defence against ransomware attacks, simply backing up to an immutable target does not guarantee a successful restore when ransomware strikes. Yes, data stored within immutable storage cannot be changed but these systems aren’t always perfect, zero-day exploits and social engineering can compromise these systems leading to immutable backups being circumvented. Implement regular backup verification and malware scanning to help avoid storage corruption and hibernating malware.

Verify Recovery – Verify your backups and backup environment are working, this includes testing failover to a disaster recovery site regularly. To improve the odds of a successful recovery, leverage technology to automate verification, testing and securing of backups while ensuring they remain easy and quick to restore.

Backup Encryption – Encrypt backups to help prevent data exfiltration and further risk of data being held to ransom.

During an attack

Every disaster is different, this is especially true for Ransomware. Ransomware recovery is a marathon, not a sprint. Long hours of uncertainty and stress will take a toll physically and emotionally.

Contain Immediately – When you disconnect devices from the network, the ransomware communication is disrupted which can limit its spread to other devices. A common practice during a ransomware attack is to immediately pull the power on systems. Instead, disconnect affected devices from the internet, other networks and USB storage media. It’s OK to keep infected systems running as long as they are isolated from the network. Why? If you drop the power, you lose most if not all the forensic capabilities.

Don’t pay the ransom – Paying the ransom is never recommended since there is no certainty it will work. Even if the attackers provided the decryption key after paying the ransom there are still multiple reasons why decryption won’t work. For example, bugs in the malware could stop the data from decrypting even with the right key. In addition, another subsequent attack may follow shortly after paying the ransom in an attempt to extort more money.

Record the Evidence – Take photos of key details such as the ransom note, web links, emails, or Bitcoin addresses. Record the date, time, file details, first signs of ransomware, and affected devices. Note what was occurring immediately before the first signs of ransomware. Also, note the time devices were disconnected from the network. Record any actions performed during the ransomware attack.

Backup the data – Consider taking a backup of the environment if recovery can’t start without first deleting encrypted data to make space, this is especially prudent if recovery has not been tested recently. It may be possible to decrypt the data at a later stage but that can’t happen if the data is wiped.

Identify the malware – If possible, identify which type of malware strain you’re dealing with. Organisations such as No More Ransom share several decryption tools available on their website.

Presumption of innocence – During the attack, don’t focus on finding the guilty within the organisation. While it’s important to understand if the attack was for financial gain or a rouge administrator (insider attack) with a grudge, it’s more important to focus on the recovery. Pointing fingers and making accusations raises tension which may boil over and hamper recovery efforts. Some organisations have not survived a ransomware disaster because they focused too much on finding the guilty party instead of restoring data and services first.

Ask for help – Asking for help early could avoid further harm. Engage with technology vendors and trusted partners, ransomware recovery can be complex and time-consuming. Call the hotlines, for Australian readers, call the Australian Cyber Security Centre 24/7 Hotline on 1300 CYBER1 (1300 292 371) if you need help during a ransomware attack.

It’s a team effort – Ransomware recovery is a team effort, involving all levels of the organisation including,

• Senior leadership to drive the process and make executive decisions
• Legal, HR and public relations to help determine the best course of action and manage wider communications
• Incident responders to drive the investigation and the recovery process
• The backup team to support the recovery with clean data

Speed – Is the business back up and running yet? If not, why not and when will it be? Be mindful that ransomware recovery is a marathon, not a sprint. People can and will make mistakes. Don’t go full throttle otherwise, people can and will burn out. Make sure people are rotated and sent home to rest and recover. It’s important to have solutions in place so the business can get back up and running as soon as possible.

Washing data – Leverage isolated recovery environments where systems can be first restored and scanned for malware. Solutions such as Veeam DataLabs can automatically create a sandbox environment isolated from the production then Veeam SecureRestore can initiate malware scanning. By creating isolated environments, malware-infected systems are not accessible to unauthorized individuals who might try to re-activate the malware.

After an Attack

The simple fact of the matter, not all organisations fully appreciate the nuances of how to recover during a disaster. In the case of ransomware, options for remediation boil down to either recovering from a backup or paying the ransom. Often organisations decide it’s better to pay the ransom and hope for the best instead of the alternative, unfortunately, hope is not a strategy, hope is not going to help in a disaster and let’s agree, ransomware is a disaster.

Report the incident – Contact the national cyber security centre to report the incident. If required under law, report any data breach to the relevant authorities. For Australian readers, this is the Australian Cyber Security Centre and the Office of the Australian Information Commissioner. If bank accounts or credit card details are at risk, contact the appropriate financial institution to take action as soon as possible.

Hibernation – Scan and check for hibernating malware that may persist in recovered data.

How can Veeam help?

Veeam has several features to assist before, during and after a ransomware attack.

• Veeam is software only and storage agnostic
  • Deployment: Choose the server, storage and cloud that make the most sense for your business
  • Lower costs: scale components independently
  • Future proof: avoid vendor lock-ins and evolve to meet future threats
• Follow the golden rule of backup; 3-2-1-1-0
  • Three different copies of the data
  • Two different media
  • One offsite copy
  • One of which is: offline air-gapped or immutable
  • Zero errors after automated backup testing and recoverability verification
• Prove readiness for a disaster and shrink downtime by leveraging Veeam Disaster Recovery Orchestrator to automate testing, documentation and recovery.
• Deploy Veeam ONE and leverage the built-in ransomware alarms
  • Possible Ransomware Activity alarm
  • Suspicious Incremental Backup Size alarm
• Detect issues with backup and replication verification (SureBackup/SureReplica)
• Encrypt backups using Veeam encryption
• Use Multi-Factor Authentication (MFA) and Group Managed Service Accounts (gMSA) (available in v12) wherever possible
• Use Linux hardened repositories or object storage with immutability support.
  • For Linux hardened repositories on physical servers with built-in disk.
  • For object storage, buy your own or use a public cloud service.
• Consider using Veeam Cloud Connect Backup with Insider Protection
• If using tapes, eject them so they are air-gapped, or use WORM
• Build a VBR environment that is powered off / offline but ready to connect to your Veeam Hardened Repository or object storage repository at a moment’s notice

Summary

Organisations often start their ransomware journey by building up their defences protecting every asset with the same level of importance and in the same way, but as they grow best practices are applied that allow for risks to be categorised and responses to be better defined and measured.

Backup is an essential part of this journey, secure backup is indeed one of the last lines of defence but it never should be the only solution against ransomware. Consider encrypting backups at a minimum, or someone else will do it for you.

The best chance of ransomware recovery is not about one solution, one technology or one group of people. It’s about bringing together everything in an organisation’s environment to protect against these attacks.

If you, the reader, only take away one paragraph, it’s this. Consider how to detect workloads and monitor for errors, how to protect workloads with secure storage and verified backups, how to recover data without reintroducing threats and finally how to improve ransomware defences regularly.

I’ll finish with a saying, what you design today is gold but tomorrow it will be old then in two days it will be wrong.

VeeamON 2022 – Veeam Ransomware Best Practices for Secure Backup & Recovery

VeeamON 2022- Zero Trust Approach to Ransomware Protection

VeeamON 2022- Ransomware How to Survive a Hack When You’ve Already Been Hacked

VeeamON 2022 – Complete Veeam-powered End-to-End Ransomware Solutions Panel