Manually remove the Veeam CDP VAIO filter driver from an ESXi host

I’ve been troubleshooting a CDP issue in my home lab so I wanted to reinstall the I/O filter drivers. Attempting to remove the filter drivers using the VBR console was failing to place the ESXi host into maintenance mode since the VBR VM was running on that host and I had nowhere to vMotion it to (I used nested virtualisation with a single physical ESXi server).

I also wasn’t having much luck uninstalling the Veeam CDP VAIO filter driver manually as per the instructions in https://www.veeam.com/kb4151 but fortunately, there is another method to remove them.

  1. Place ESXi host into maintenance mode
  2. SSH to the ESXi host
  3. Verify ‘veecdp’ filter driver exists with command, esxcli software vib list

4. Remove the VAIO filter driver with command, esxcli software vib remove -n veecdp 5. Take the ESXi host out of maintenance mode. 6. Go to VBR console and ‘Uninstall I/O filter’

7. The task will fail to uninstall the drivers as it longer exists but the console will now show it’s possible to ‘Install I/O filter’ again.

Failed to connect to the server. Specified user is invalid or does not have enough permissions on the server

I’ve recently had a couple of issues when adding a standalone VBR (Veeam Backup & Replication) server to VDRO (Veeam Disaster Recovery Orchestrator). This is a quick write-up to cover the basic troubleshooting steps performed and how the problems were resolved.

The error thrown in VDRO is as follows; “Failed to connect to the server. Specified user is invalid or does not have enough permissions on the server.”

Continue reading

Tagging with Veeam

Tags are a great way to manage and organise resources across a vSphere environment, with tags we can sort and group VMs together based on any criteria we wish. Veeam can even leverage these “groups of VMs” in many different ways.

For example, Veeam Disaster Recovery Orchestrator leverages tags for orchestration plans and to make sure VMs are restored to the right place with tag-based recovery locations.

Veeam ONE Business View can automatically create and assign tags to VMs based on any desired criteria with its categorization engine, this is on top of VeeamONEs capability to run reports based on tags. I’ve previously written about Veeam ONE Business View which can be found here.

Veeam Backup & Replication (VBR) also can utilise tags for any job. By simply adding tags as the source object to the job, VBR will protect any VMs with that same tag, this can be especially time-saving if VMs are frequently being provisioned. Another benefit of tags is that, unlike folders and resource pools, VMs can be assigned multiple tags.

Let’s take a look at how we could group a selection of 8 VMs into various different backup jobs using tags.

Continue reading

Error: Unable to delete backup in the Capacity Tier because it is immutable

I was recently involved with a Veeam deployment that was running into problems during testing, their only performance tier had run out of space. Though this wasn’t unexpected as the disk provisioned was undersized and just temporary until testing was finished, it was preventing new backups from finishing successfully.

The full performance tier belonged to a Scale-Out Backup Repository that was also configured with a capacity tier (copy + move mode) backed by an immutable AWS S3 bucket. Worth mentioning those backup files in the capacity tier were still within the immutability retention period.

According to the user guide “If you use the scale-out backup repository, keep in mind that the Delete from disk operation will remove the backups not only from the performance tier but also from the capacity and archive tier. If you want to remove backups from the performance tier only, you should move those backups to the capacity tier instead. For details, see Moving to Capacity Tier.”

Attempting to perform a “Delete from disk” operation was failing with the error “Error: Unable to delete backup in the Capacity Tier because it is immutable”.

Continue reading

Pay the ransom and hope for the best…

Ransomware isn’t sexy but it’s certainly an important topic in today’s IT security landscape, with unprecedented growth and relentless evolution, organisations need to constantly keep one step ahead of bad actors eager to make a buck by exfiltrating and encrypting your important data.

Ransomware attacks mainly occur because IT is a complex and everchanging environment, organisations are busy modernising their applications from monolithic designs to new highly distributed container-based services. The remote workforce is now connected more than ever, with the ability to access data hosted across multiple cloud providers, connecting from virtually any device, at any hour of the day. IT departments are being asked to do more with less each and every year while ransomware attacks are on the rise and becoming more costly than ever before.

If you’re questioning whether or not your organisation could be targeted by ransomware, it’s a question of when not if

Beyond encryption of data, some ransomware is taking it a step further and making ransom of leaking data, this is otherwise known as data exfiltration. Unfortunately, stopping ransomware prior to an attack is difficult and at best, inconsistent. No single product or service has all the solutions to the challenges raised by these attacks, instead, it’s recommended to take a multi-layered approach. Apply best practices, keep systems up to date, enforce good data hygiene, configure event logging, and identify anomalies (indicators of compromise) to provide the best chance of discovering an attack as early as possible.

Continue reading

VeeamON 2022

VeeamON 2022, the best data protection conference of the year is about to kick off in Las Vegas and virtually. This will be the 6th time Veeam has run the event with attendees able to access both virtually and in person.

VeeamON 2022 Conference

VeeamON provides glimpses into product roadmaps, with hours of technical content ranging from demos to deep dives of virtually every Veeam product and feature. Hot topics such as v12, Ransomware, Kubernetes, cloud-native backups, Salesforce and Microsoft 365 will be covered by Veeam experts including fellow Veeam Vanguards.

For those who can’t make it in person, virtual attendance is free, running from May 16 – 19 with AMERs, EMEA and APJ specific sessions to ensure easy access for those joining.

With 2 days of awesome content, you might not make it to every session on your agenda. Fortunately, Veeam makes it easy to replay recorded sessions from the VeeamON website after the conference has finished.

To learn more and register, check out the following link; https://www.veeam.com/veeamon

Veeam Vanguard 2022

I received some positive news the other day concerning my Veeam Vanguard 2022 application, I’ve been renewed for another year in the Program. As a Veeam enthusiast, there is no better thing than being a Vanguard member.

For those who haven’t heard about the Vanguard program before, it’s awarded to those who have contributed to the Veeam community.

I recommend anyone interested in becoming a Vanguard to get involved in the Veeam community, whether it be through Veeam Use Groups, the Veeam subreddit or even the Veeam community forums and apply to the Vanguard program during the next intakes.

There are a couple of benefits of becoming a Vanguard including access to roadmaps, betas, free keys, awesome swag and access to those in the know at Veeam.  

You can find all the Veeam Vanguards from past and present at the following link: Veeam Vanguard List.

Veeam Backup Repository / Microsoft Azure Object Storage

Sending Veeam backups to object storage such as Azure Blob has become a hot topic in the last few years. According to Veeam’s quarterly report for the end of 2021, Veeam customers moved over 500 PB of backups just into the top 3 cloud object storage vendors alone.

With many organisations starting to dip their Veeam toes into object storage I thought I would write a bit more about the subject. This blog post is aimed at helping backup administrators who wish to better understand from a Veeam perspective working with public cloud object storage, specifically Azure Blob.

Compared to the traditional NAS or disk-based block storage Object Storage is a completely different shift in how data is stored and accessed. For example, in object storage, it’s intended that files are not modified. In fact, there is no way to modify part of an object’s data and any change requires deletion and replacement of the whole object.

In Azure terminology, objects are stored in a ‘Blob’, which can be thought of as similar to a volume on a disk but far more scalable. Blob storage is a pay-per-use service. Charges are monthly for the amount of data stored, accessing that data, and in the case of cool and archive tiers, a minimum required retention period. In case you haven’t realised, Blob storage is Microsoft’s object storage solution.

There are numerous methods we can utilise to leverage Microsoft Azure Blob with Veeam Backup & Replication. For example, Azure Blob can be used as an Archive Tier target within a SOBR (Scale-Out Backup Repository) for long-term retention of backups, an archive repository for Veeam NAS Backups and some readers may even be familiar with the external repositories function.

The most popular method is using Blob as a Veeam Capacity Tier which is configurable within a Veeam Scale-Out Backup Repository.

Why is Azure Blob so popular?

Continue reading

Improved Veeam Visio Stencils

Veeam makes available, for free, Visio stencils available from their website here which are useful for creating Veeam diagrams in Microsoft Visio. These stencils enable specific Veeam components to be illustrated in greater detail instead of relying on generic stencils found in the stock Visio library. I’ve been using these Veeams stencils for several years now while they’ve proven themselves to be beneficial they aren’t short of a few quirks and issues.

Recently, a colleague of mine, David Summers, mentioned he has been working on improving the Azure Visio stencils and could help improve the Veeam stencils the same tool he has developed. For those interested, David has had quite an interesting journey around Azure Services Visio stencils which is definitely worth a read over at his GitHub David-Summers/Azure-Design

After passing the Veeam stencils through his tool, the following changes were implemented.

  1. Icons are now sorted alphabetically
  2. New connections points have been added (North/North East/East/South East/etc)
  3. New Connection points are relative to the centre/size of each icon
  4. Text box is now placed at the bottom of each stencil icon, instead of the middle
  5. Text box will now dynamically resize it’s width and height based on the amount of text
  6. To make icons stand out, drop shadow has been enabled on all icons, this is a personal preference and can easily be disabled
  7. Bonus PNG and SVG icons have been split out so third-party programs outside of Visio that support PNG and SVG can make use of them

The improved Veeam Visio stencils can be downloaded below;

Improved Veeam Vison Stencil v1.0.zip

Simulating Veeam disk actions with DiskSpd

Did you know that you can use DiskSpd to measure and simulate Veeam disk operations such as Active Full, Forward Incremental, Synthetic Fulls, even SureBackup and restore operations? By simulating Veeam disk operations, we can use this tool for troubleshooting Veeam backup and/or restore performance issues.

I wanted to try the tool myself so I’ve recorded some of the results after testing DiskSpd on my home desktop PC.

Continue reading