Implementing the 3-2-1 backup rule with Veeam & Azure

One of the tried and tested rules that can effectively address any failure scenario is called the 3-2-1 backup rule. This backup rule is becoming more and more important as organisations continue to virtualise their mission critical data, making the protection of that data becomes more vital than ever.

The 3-2-1 rule became a popular concept thanks to Peter Krogh, a well-known photographer who wrote that there are two groups of people: those who have already had a storage failure and those who will have one in the future.

In other words, the 3-2-1 backup rule means you should have 3 copies of your production data, on 2 different types of media and 1 offsite.

The purpose of this post is focused on getting your backup data offsite, and how we can leverage Microsoft Azure and Veeam together to help meet the 3-2-1 backup rule.

 

How can I get my data offsite to Azure?

Well, there are currently three main ways to get your data offsite and into Azure Cloud.

3-solutions

Figure 1: Veeam to Azure

Option 1

The first method is to utilise an on-premises storage appliance called a StorSimple. The StorSimple can be deployed either as a physical appliance or as a virtual software appliance which caters for the majority of organisations requirements. By utilising the StorSimple capability to automatically archive Veeam backup data out to blob based Azure storage, we can achieve offsite backups. The StorSimple is a great bolt-on solution for existing Veeam deployments with relatively quick deployments and easier to use interfaces.

The flexible architecture of StorSimple is ideal for customers who want to externalise more than just their backups and is ideal for large volumes with usage of Azure native storage.

Limitations that should be considered are the extra appliance that needs to be deployed and maintained. Also, consider that while there is a virtual software appliance available there is a cost for any physical appliances that are acquired. A faster internet connection is recommended to meet any established recovery point objectives (RPO).

storsimple

Figure 2: Veeam to StorSimple to Azure

If Veeam requires access to the archived data within Azure, the StorSimple will automatically pull the data back. A fast internet connection is recommended to ensure your offsite backup requirements can be met. No additional Veeam licenses are required for this option.

Option 2

Option two is to externalise with a dedicated link to an Azure site as Site to Site Link (Azure ExpressRoute), SSL Direct Link to Azure or using a private network (vNet) in Azure.

Offsite storage is made available by configuring a link between the on-premises Veeam server and the Azure storage which is used to store the offsite Veeam backup data. Optionally a virtual machine can be created in Azure to provide WAN Acceleration to improve performance.

The benefits of this method is available to all organisations, no appliance needs to be deployed taking up valuable rack space or consuming resources living on your hyper-visor hosts as a virtual appliance. This option also does not require any additional Veeam licenses.

dedicated-link

Figure 3: Direct to Azure

Things to consider are the link that is necessary which may impact on performance and possibly workload that may share the link. Costs associated with the Azure ExpressRoute need to be considered as well. This option is not recommended for multi-location infrastructures as it can be complex.

Option 3

The third option is to send backups offsite to Azure using Veeam Cloud Connect.

Veeam Cloud Connect is a technology that enables sending backup data to an offsite location managed by a Service Provider or the organisation themselves.

cloud-connect-for-the-enterprise

Figure 4: Veeam Cloud Connect for the Enterprise

There are two flavours of Veeam Cloud Connect:

1. Veeam Cloud Connect for the Enterprise

Veeam Cloud Connect for Enterprise allows enterprise organisations to operate their own Hybrid Cloud by acting as their own service provider, they configure the necessary infrastructure in Azure to receive and manage the off-site backup data.

There are several considerations for this option as it is geared towards enterprise customers. Veeam Cloud Connect for the Enterprise requires the organisation to own the Enterprise Plus edition of Veeam – also if the customer is not in an Enterprise Agreement (EA) with Microsoft, then there is a 100 socket minimum of Veeam Enterprise licenses. If the organisation owns an EA with Microsoft, then there is no socket minimum. In either case the customer must match licenses 1 to 1.

For example, if ACME company has an EA with Microsoft and owns 86 sockets of Veeam Enterprise Plus, they would purchase 86 sockets of Veeam Cloud Connect Enterprise. Without the EA they would be required to purchase 100 sockets.

2. Veeam Cloud Connect

A service provider will host the offsite backup storage in Azure which is presented to organisations with on-premises Veeam. The service provider’s backup data can be encrypted at the source (before it leaves your network perimeter), in flight and at rest. This method is the easiest way to externalise to Azure, its ideal for multi-site configurations and you keep the same interface and console.

veeam-cloud-connect

Figure 5: Veeam Cloud Connect

Veeam Cloud Connect is included within the Veeam Availability Suite, Veeam Backup & Replication and Veeam Backup Essentials for all organisations at no additional charge and with no additional licensing required. However, organisations will need to acquire a subscription to the appropriate storage resources from a service provider of your choice in order to use it.

 

VeeamOn Giveaway! Win a fully paid trip to New Orleans (EMEA Region Only)

Win a fully paid trip to VeeamOn 2017 in New Orleans.

Do you live in the EMEA region and want to come to VeeamON in New Orleans? Do you want to hear about all the new Veeam features and talk directly to vendors and partners? Perhaps you’re interested in having a go at LabWarz to test your skills amongst the best? If you haven’t secured your ticket yet here is one opportunity you shouldn’t let pass.

Veeam is allowing me to raffle off a trip for one to VeeamOn 2017.  To enter this raffle simply tell me a story regarding Veeam. Whether it’s that time Veeam uncovered that 6-month-old snapshot, helped you reclaim all those over-provisioned vms, that time Veeam saved your bacon or something just interesting and cool.

So tell me why you should be at VeeamON 2017 because if you are the lucky one who is selected you will receive a fully paid trip to VeeamOn 2017. Yes, that even includes flights, hotel stay and conference attendance!

If you are based in EMEA and wish to enter this raffle, simply comment your story below.

Don’t forget to include your full name. The deadline for the raffle draw is April 7th (CET).

 

**UPDATE**

Thank you to everyone that entered the raffle, the lucky recipient has been drawn and notified.

 

Common Veeam B&R Mistakes

Just a small write up regarding the some of the most common Veeam B&R mistakes that I’ve seen lately.

1.Backing up the Veeam B&R Server
It may come as a bit of shock but you shouldn’t actually back up the Veeam B&R server using a backup job in Veeam. In most cases, Veeam won’t be bothered by this but on occasion Veeam can experience the following symptoms.

  • Disconnection from the configuration database.
  • Disconnection from remote Veeam Backup & Replication agents.
  • Disconnection from network storages (for example, storages presented via iSCSI) and so on

This is caused by the freezing caused during snapshot creation and committing so instead, you should rely on Veeams built in ‘Configuration Backup’ function.

2.Veeam Configuration Backup not setup properly
This is by far the most common mistake but the easiest to fix. By default the Veeam B&R server will backup it’s configuration up at 10:00 AM every day to the default backup repository. Often I see this is left in its default state and the backup files are neither copied offsite or encrypted. Getting this configuration backup files offsite is simple and easy so in my mind, there is no reason not to do this.

  • If you have an offsite site you can configure a file copy job
  • If you are using tapes you can use the file to tape job
  • Alternatively, I’ve seen customers use dropbox or google drive to get the configuration backup files offsite.

Why do we care about getting the configuration file offsite? Well, partly because you should always use the configuration backup functionality to back up and restore the configuration of the Veeam B&R server. Secondly, its a lot faster to restore using the configuration backup compared to manually reconfiguring Veeam from scratch.

Enabling encryption for the backup configuration is also critical. If you don’t enable encryption for the Veeam configuration backup this means all those credentials used for the application aware processing will be lost in the event of restoring the configuration backup to a fresh Veeam B&R server. I’ve only migrated one Veeam B&R instances with no encryption for the configuration backup enabled, I learnt pretty quick to always enable it after spending hours reconfiguring it.

3. 1-Click Failover not setup properly

1-Click failover is an awesome feature that reduces the complexity of managing failovers. It’s basically a failover plan which handles the startup order, the delay between startups and automates the running of scripts in a single operation. You could initiate a failover plan using a mobile phone in bed at 3 o’clock in the morning without ever getting out of bed if you really wanted to.

So utilising 1-click failover plans through Veeam Enterprise Manager (VEM) during a disaster means VEM can’t be down. It’s important to note that we can initiate a failover plan through the Veeam B&R console without VEM and it’s really only a minor issue if VEM is offline but I’ve seen customers first hand specifically plan to use the web portal to start their failover plans. Well, that’s great except if VEM is running from the production site.

In this instance, it was managing/federated with the B&R server at DR which was handling the replication but in the event of a total production site loss, this customer would lose access to VEM and the web portal. Luckily it’s not a big deal since the customer could still access failover plans through the DR B&R server. At best a minor inconvenience which delays failover.

I’ve also seen lately customers with a limited budget having a single vCenter managing both production ESXi hosts and the hosts in DR. If Veeam is configured to replicate through the production vCenter to the managed DR ESXi hosts you are going to have a bad day during failover.In this scenario, you can either run up a second vCenter in DR, configure Veeam to replicate directly to a standalone host which introduces its own headaches (SureReplica) or plan to manually power on the vCenter replica first at DR. If your only vCenter at production is a physical server it might be time to consider virtualising it.

4. No Backup Verification

I usually don’t raise an eyebrow when customers choose not to test their replicas but not verifying your backups never has a valid excuse in my book. What’s the point of backing up if you don’t know that you can recover?

Now backups are the always the first thing that comes to my mind when disaster strikes and while it’s a good thing that Veeam replicas can still function as a kind of backup with their guest os file level restores and more but their recovery points are much more limited. I like to consider backups are all about RPOs and retention while replicas are focused on RTOs, Instant VM Recovery tends to blur the lines a bit, though.

If you don’t have the correct licensing for SureBackup, it doesn’t matter. Run up an Instant VMRecovery and test in an isolated network.

What about just using SureReplica but no SureBackup, well I certainly would feel more comfortable knowing I could at least restore recent copies of data from the replica but anything outside of the replica restore points retention will be an unknown regarding recoverability.

6. MS Dedup Backup Repository setup incorrectly

Microsoft DeDupe can be a great for reducing the size of your backup files in the backup repository. Unfortunately, there are few key settings that need to be set at the time of the volume creation as it’s not possible to apply the settings after the volume is created.

For a better explanation of how DeDupe should be configured, check out this awesome article over at kool-aid

http://www.koolaid.info/windows-server-deduplication-veeam-repositories/

Other worthy contenders would be

  • Not excluding Veeam from the A/V
  • Job Chaining instead of scheduling
  • Not following the 3-2-1 backup rule
  • Not using vSphere Tags
  • Backup Jobs using guest indexing when you aren’t not using 1-click restores.

Thank you Veeam – Veeam Vanguard

Early last week I received news from Rick Vanover @ Veeam that I have been selected for the Veeam Vanguard Program. As a Veeam enthusiast, there is no better thing than becoming a Vanguard. Wow what an honour!

For those who are unaware of the Veeam Vanguard program, it’s awarded to members of the Veeam community to show Veeams support and appreciation for past deeds. There are a couple benefits of becoming a Vanguard including access to roadmaps, betas, free keys, awesome swag and access to those in the know at Veeam (I’m just curious to know if we get a bat phone to Anton Gostev (I wish)) just to name a few.  What I’m really looking forward to though is learning more about this awesome program and meeting other Vanguards who share my passion for Veeam.

If you happen to read my blog or know me personally, you may be aware that I tend to champion Veeam a lot. I try and help the community whether it’s writing about errors I fixed in Veeam or the unofficial VMCE practice exam. Whether it’s just a small thank you on twitter, an appreciative comment on a post or being selected for a global program, it really fuels my fire to keeping creating, sharing and interacting to help the community more. So thank you Veeam,  thank you for making such an awesome product to work with, thank you for showing support and a big thank you for recognising members of the Veeam community.

Error: Restore job failed Error: NFS status code: 30

I recently saw the below error occurring for a customer attempting to restore a backup from their NetApp SAN using Direct NFS Access. Backups using Direct NFS access were performing without issue but the restores would only successfully complete if the proxy could use the network mode (NBD).

“Restore job failed Error: NFS status code: 30
Read-only file system.
Cannot get count from WRITE3res.
Failed to download disk.
Agent failed to process method {DataTransfer.SyncDisk}.”

Direct NFS Access Error

Turns out the backup proxy was only configured with read permission not read/write.

According to the Veeam User Guide the backup proxy must have ReadOnly/Write permissions and root access to the NFS datastore.

Using Per-VM Backup Files and Deduplication

Just a quick tip regarding ‘per-vm backup files’

If your backup repository is configured with VM backup files, deduplication is only available within a single VM backup chain.
If your backup repository is not configured to use per VM backup files, deduplication across all VMs within a single job is available.

Post-Migration to Veeam – Considerations for your Legacy Backup Solution

I’ve completed my fair share of Veeam deployments in environments where there is an existing backup solution.
The question that comes up the most is, what do I do with my legacy backup data?

Well, here are my thoughts around best practices for this situation.

Option 1 Perform restores of protected data using the legacy backup software for selected restore points to a staging area, once Veeam has re-protected the VMs the legacy backup solution can be retired. VeeamZip is a great option here.

Pros

  • Removal of the legacy backup solution

Cons

  • Time-consuming if re-protecting a large amount of VM data
  • Very time-consuming if restoring from tape
  • Can be complicated if dealing with large amount of restore points & VMs
  • Requires a staging area to restore the VMs to

Thoughts: I see this option used when it’s not possible for the legacy backup data to be simply left to expire. Perhaps the retention period is too great or restore requests are frequent from legacy restore points.  This method is not very common requires as it requires a lot of time and resources to reprotect the restored data in Veeam.

Currently there is no Veeam migration tool to migrate legacy restore points to Veeam automagically.

Option 2 Suspend existing backup solution and maintain existing legacy backup data in the event of a restore being required before Veeam was implemented. Any backup data that expires/passes their retention period can be deleted to reclaim space.

Pros

  • Less work and much easier

Cons

  • Existing backup solution is taking up resources, if installed on a physical server then it is taking up space, if it’s powered on then cooling & power costs. If it’s a virtual machine it is taking up disk space on your production storage.

Thoughts: This is the most common option taken in my experience, any legacy licenses aren’t renewed and the the legacy backup data is left to expire.

Poor Performance & Power Management on VMware

Poor performance experienced by your VMs may be related to processor power management implemented either by ESXi/ESX or by the server hardware.

One real world case I recently encountered with a customer involved VMware Horizon View and large delays experienced by their end users. Applications took unusually long times to open and general performance was quite bad. This was quite apparent when comparing the same applications on a thick client to running it on a virtual desktop.

After running through the usual checks consisting of VMware Health Analyzer, checking for over-subscription and over-utilisation there were no red herrings immediately apparent. What we did discover though (which is detailed in ‘Best Practices for Performance Tuning of Latency-Sensitive Workloads in vSphere VMs’) involved changing a BIOS setting on all of the ESXi hosts. Specifically the setting for power management on the ESXi HP Hosts to “Static High”, that is, no OS-controlled power management.

While we are working through the other recommendations provided in the VMware Health Analyzer report and have already made some changes to the configuration, nothing has resulted in a noticeable improvement with the exception of the power management setting. The customer has reported that after changing this particular power setting it has provided the most significant improvement in performance of anything previously attempted (hardware and software).

 

Progress Controller: [VCSA ERROR] – Progress callback error

Deploying vCenter Server Appliance 6.0 and run into this beauty,

‘Progress Controller: [VCSA ERROR] – Progress callback error’

Turns out the vCenter Server Appliance installer will fail if more than one DNS server is provided. Fantastic…

To workaround, provide one DNS server IP during the installation wizard. Once the VCSA is installed and running you can then provide the secondary DNS IP.