Safeguard your Veeam backups with Pure Storage FlashBlade® SafeMode

Authors – Lawrence AngRhys Hammond and Dilupa Ranatunga

Introduction  

This is the second part of a three-part blog series on Veeam and Pure Storage FlashBlade. In the previous blog post, we configured a Network File System (NFS) share on a Pure Storage FlashBlade as a Veeam backup repository. In this blog post, we will be focusing on configuring SafeMode snapshots to harden the backup files that are residing on the FlashBlade.

Ransomware attacks continue to rise, with constantly evolving sophistication and complexity. A key part of ransomware resilience strategy is backing up data on a regular basis and implementing a strong line of defence against threats targeting the backup data. Adopting industry standards for data protection such as 3-2-1 rule, offline backups and immutable backup storage are effective techniques to protect backup data sets against malicious attacks. Now let’s discuss how to make your FlashBlade system an immutable backup storage target with SafeMode snapshots.    

A storage snapshot is a point-in-time, image-level view of data that are impervious to ransomware. This immutability makes them an ideal layer of defense against ransomware. The problem with storage snapshots is they can still be removed by rouge admins or attackers if they gain access to the storage array management. In the case of a Pure Storage system, the deleted snapshots are temporarily stored in a ‘destroyed state’ that is similar to a recycle bin. If these snapshots are not recovered in a timely manner, they will be auto-eradicated and can even be manually destroyed prior to the auto-eradication. 

The SafeMode snapshots on the other hand, cannot be deleted, modified, or encrypted either accidentally or intentionally. This prevents the manual and complete eradication (permanent deletion) of data backups that are stored within the FlashBlade. Due to their immutability, the SafeMode snapshots serve as an additional mitigation mechanism against ransomware attacks or rogue administrators.

The SafeMode snapshots are created and managed automatically by Purity, the operating software in FlashBlade, independent of administrator control. You can schedule SafeMode snapshot generation of any data that are residing on the FlashBlade file systems. Primary and backup data can be directly recovered from these snapshots. If original copies are corrupted or destroyed, data can still be recovered from the SafeMode snapshots. 

A picture containing screenshot

Description automatically generated

Setup SafeMode 

Overview

Once the SafeMode snapshots are activated, snapshots for all the file systems get created that incurs additional capacity utilisation on FlashBlade. Therefore, prior to SafeMode activation, it is recommended to consult your local Pure Storage Systems Engineer to validate FlashBlade capacity sizing that is required to support SafeMode snapshots. 

Pure Storage has developed a highly secured, well-defined process on how FlashBlade customers collaborate with Pure Storage Support to enable SafeMode or how to make changes to an existing SafeMode policy. Only authorised users from the customer organization can work directly with Pure Technical Support to configure the feature, modify the policy, or manually eradicate SafeMode snapshots. We decided not to discuss the full details here as we do not want potential ransomware perpetrators reading this post to be benefited from the information. You are always welcomed to engage your Pure Storage Sales Team if you’d like a more detailed walkthrough of how SafeMode works, how it is enabled, and protections against it being disabled.

Upon agreement with the customer, Pure Support performs a pre-activation health check that involves checking the FlashBlades growth trend to ensure there is sufficient capacity to enable SafeMode.

Validating the operation of safemode

Once Pure Support has enabled SafeMode, validation is required. 

The screenshot above displays a list of snapshots of the file system <Veeam-NFS-Repo>. There are 2 distinct snapshot groups presented here. 

  1. Standard policy based snapshots: These are created and managed by FlashBlade administrators. These snapshots are represented by the policy name <Veeam-Snapshot-2hr> in the image.
  2. SafeMode snapshots: These snapshots are not managed by FlashBlade administrators. Creation and eradication of SafeMode snapshots are fully automated by SafeMode policy that is defined by the authorised designee. As observed, there is no policy name (-) that is assigned to these SafeMode snapshots. 

When attempting to eradicate a SafeMode snapshot, the file system will be moved to a destroyed state. 

A screenshot of a cell phone

Description automatically generated

The destroyed snapshot will be placed in a destroyed snapshot folder as highlighted below. In our test, eradication timeout for the destroyed snapshots was changed from the default value of 14-days to 7-days. 

When attempting to eradicate (permanently deleting) a SafeMode snapshot, a warning message will be displayed as shown below. 

A screenshot of a cell phone

Description automatically generated

It is evident that under SafeMode, the administrators cannot manually remove a filesystem or snapshots from FlashBlade. 

Simulating an attack

A sophisticated ransomware attack does not just encrypt production data, perpetrators often target the backups to ensure organisations cannot recover. There are cases where the backup servers were compromised by ransomware attacks and became inaccessible. Other scenarios could be when perpetrators and/or rogue admins gained access to the storage systems and attempted to delete the file systems and snapshots hosting backup data. 

A close up of a logo

Description automatically generated

To demonstrate the effectiveness of SafeMode, we have assumed that access to both the Veeam console and FlashBlade interface has been gained. From the Veeam console, a ‘Delete from disk’ operation was performed on all backups that are stored on the FlashBlade NFS repository. This results in the deletion of the following:

  • Records about the selected backups from the Veeam Backup & Replication console and configuration database 
  • Selected backup files from the backup repository

The following screenshot depicts the status of <Veeam-NFS-SafeMode-Repo> after all of its content is removed from the corresponding Veeam backup repository. Note that the used space displays 0TB as the file system is storing nothing.  

The NFS file system that is used for the Veeam repository was then destroyed from the FlashBlade console. 

Recovering with SafeMode

If other protection mechanisms are compromised, the FlashBlade storage becomes a solid final line of defence. As SafeMode snapshots are immutable and cannot be eradicated manually, the recovery process can be started as below.

  1. Delete compromised/encrypted data.
  2. Re-install and reconfigure Veeam backup server. (Only if Veeam server environment is compromised and inaccessible)
  3. Point backup software at data that is stored in the SafeMode snapshot with assistance from Pure Support.
  4. Begin the recovery process at the speed of FlashBlade.

Recovery of FlashBlade destroyed File System

To recover a destroyed file system:

  1. Signin to FlashBlade
  2. On the left navigation pane, click Storage > File Systems > Destroyed. Note that the view is expanded.

Review the file systems in the destroyed container and click Restore. Note that the destroyed file system moves back to the file system folder.

  1. Once the destroyed file system is recovered, it is visible in the file systems view. 

Based on our simulated attack, restoring the file system is just the first step to recovery as both the Veeam backup data and the File System (NFS) were destroyed. The next step is to revert the File System to a previous snapshot before the data was deleted/compromised.

Reverting the file system to a previous SafeMode snapshot requires the intervention of Pure Support. The FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee needs to contact Pure Support to complete the restoration process.

Note: Purity does not allow snapshot restore by an administrator with SafeMode enabled. An error message is shown whenever Administrators attempt to revert a snapshot.

Restoring with Veeam

Let’s consider a few scenarios!

Scenario 1) The backup records in the Veeam configuration database were not destroyed. 

  • Revert the SafeMode snapshot. 
  • Restores from Veeam can immediately be started from the Veeam Console.

Scenario 2) The backup records in the Veeam configuration database were destroyed. 

  • Revert the SafeMode snapshot. 
  • Perform a Veeam rescan on the NFS backup repository.
  • Restores from Veeam can immediately be started from the Veeam Console.

Scenario 3) Veeam Backup & Replication server was destroyed

  • Revert the SafeMode snapshot. 
  • Install VBR. NOTE: If the Veeam Configuration Backup is available, import the configuration backup. If backup files were deleted from the configuration backup before the last configuration backup occurred, a rescan may be required before VM restores can start . 
  • Add the FlashBlade NFS share as a Veeam NFS Backup Repository.
  • Perform a Veeam rescan on the NFS backup repository.
  • Restores from Veeam can immediately be started from the Veeam Console.

Adding a FlashBlade NFS Share to Veeam as a Backup Repository is covered in part one.

A picture containing table

Description automatically generated

Simulating an restore

The simulated attack described above, involved deleting the backup records from the Veeam configuration database but not from the NFS backup repository. Therefore the following steps are performed. 

  • Revert the SafeMode snapshot. 
  • Perform a Veeam rescan on the NFS backup repository.

After the rescan is complete, the restore points will reappear under Backups > Disk (Imported).

Veeam can now perform an Instant Recovery to immediately restore VMs by running them directly from the backup files that are stored on the reverted NFS backup repository.

The following screenshot shows the restored VMs that are available in the vSphere Client.

Additional Information

  • FlashBlade includes the SafeMode snapshots feature with no extra charges.
  • To reconfigure SafeMode after enabling it, please contact Pure Support.
  • Pure Support will only eradicate destroyed items.
  • Besides protecting snapshots, SafeMode also prevents the File Systems from being destroyed for a selected time period. Even if no snapshots are present, the share will be moved to the ‘Destroyed File Systems’ bucket for a defined time period before being destroyed. 
  • SafeMode is a global setting and once enabled all file systems are protected.
  • SafeMode is part of a comprehensive security strategy hence, it should not be solely relied upon to prevent or thwart a ransomware attack.

Caveats

Following are a few caveats about SafeMode that customers should be aware of: 

  • FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The authorised designee will need to contact Pure Support to revert any SafeMode Snapshots.
  • SafeMode does NOT have WORM (Write Once Read Many) capabilities

Security

It is important to consider the security of the device as well. If the storage device can be easily taken over, then the software layer providing the storage immutability may not be of much help. Fortunately in the case of Pure Store SafeMode, even if the management GUI was accessed, the rouge administrator cannot factory reset/wipe the storage and manipulate the storage snapshots.

To further increase security, Multi-Factor Authentication (MFA) will be introduced in an upcoming Purity release (v6).

Veeam Backup Configuration

While not required, it is recommended to configure the Veeam Backup & Replication (VBR) Configuration Backup inthe same NFS Backup Repository. This simplifies the VBR server recovery when the server is corrupted or unavailable. A new VBR can be deployed and the Veeam configuration backup can be recovered onto the server.

Summary

The Pure Storage FlashBlade is an on-premises backup storage that provides immutable storage for data such as Veeam backups. By configuring SafeMode it provides an additional layer of protection against ransomware/cyber-attack.

In the next blog post of this series, we will be discussing how Object Storage presented from the FlashBlade will be used as a backup repository target for Veeam Backup for Office 365.

Disclaimer: For advice regarding retention periods and ransomware security, please speak to your local Pure Storage SE.

Appendix

Standard Snapshot restoration of Veeam backups (GUI)

To restore a non SafeMode snapshot:

  1. To expand the view and detail available file system snapshots, select the file system. 
  2. Based on when the incident occurred, select the appropriate snapshot that is available for the file system. Note that the time and date stamps are included in the snapshot name.
  1. Click Restore that is next to the appropriate snapshot. 
  • A file system can be restored or rolled back from the most recent snapshot of that file system using the purefs copy command. 
  • To restore a file system from a previous snapshot, include the source snapshot name that indicates the restore point (SOURCE) and the target file system name(TARGET). 
  • To indicate that the file system is being restored from the specified snapshot, make sure to include the –overwrite option 
  • To discard the data on the existing target file system (but not existing data in the specified source snapshot) use the –discard-non-snapshotted-data option. 
  • Similar to the file system and snapshot eradication, file system rollback is categorised as a privileged operation under SafeMode. FlashBlade administrators are not allowed to perform a filesystem rollback using GUI or CLI. The company designee will need to contact Pure Support to complete the below procedure. 

One thought on “Safeguard your Veeam backups with Pure Storage FlashBlade® SafeMode

  1. Pingback: Pay the ransom and hope for the best... - rhyshammond.com

Leave a Reply

Your email address will not be published. Required fields are marked *