Archive Tier was announced back at VeeamON 2017 New Orleans alongside a raft of new features scheduled for release with Veeam Backup & Replication v10. Archive Tier would enable Veeam administrators to easily add regular disk-based backup repositories, object-based storage repositories or even tape as an archive extent to a SOBR (Scale-Out Backup Repository) which could then be configured to copy any backup or move sealed backup files from the SOBR across to said archive extent.
The ability to archive backup files to a particular archive extent such as tape or cheaper disk was a great addition, but the significant improvement was the native integration with object storage which has been a highly requested feature for several years now. During VeeamON it was announced that AWS S3, AWS Glacier, Azure BLOB and Swift compatible object storage to be supported.
Copying Veeam backup files to object storage has always been possible through the use of third-party vendor storage gateways, such as the AWS Storage Gateway or Azure StoreSimple but speaking from my own experiences, these tools don’t always deliver what they promise and require additional skills to support.
I was just checking out Poul Preben’s blog and discovered a fix for an issue I encountered during an earlier Veeam deployment. Don’t you love finding answers to those mysterious issues, I certainly do.
The problem arose whenever I tried to add a particular windows server into the Veeam managed backup infrastructure. The server was earmarked to become a Veeam Proxy and Backup Repository. As per best practices we didn’t join this server to the domain and created a dedicated local account on the server for Veeam authentication. Remember if the logins on the machine to-be-backed up and the backup storage are the same, we call that unwanted correlation.
Unfortunately, we ran into the below issue when trying to install the Veeam Deployment Service.
[my.repository.fqdn] Failed to install deployment service.The Network path was not found–tr: Failed to create persistent connection to ADMIN$ shared folder on host [my.repository.fqdn].–tr: Failed to install service [VeeamDeploymentService] was not installed on the host [my.repository.fqdn].
The Veeam binaries are pushed through the ADMIN$ share and it turns out that this share cannot be accessed with a local administrator account by default, due to Remote UAC being enabled. If we had used the local Administrator (SID 500) account however, this issue wouldn’t have occurred.
Poul details the fix on his blog which I’ll link below.
Anton Gostev recently wrote about a bug that will impact a lot of Veeam environments so I thought it would be best if I mentioned it here to help get the word out. Veeam have also created a KB article you can find here detailing this issue.
If your Veeam Backup & Replication console is showing a “Failed to check certificate expiration date” message upon opening the backup console, it means that your default self-signed certificate is about to expire.
A self-signed certificate is an identity certificate that is signed by the same entity whose identity it certifies. Veeam uses certificates to implement secure communications between your backup infrastructure components, as well as with any managed backup agents in your environment.
Now Self-signed certificates are automatically renewed every 12 months by your Veeam Server but due to a bug introduced in v9.5 U3a, the Veeam Backup Service will still have old information about the absolute certificate even after a new self-signed certificate is automatically generated. If you ignore this message, once the self-signed certificates are automatically renewed after 12 months, agent management functionality, as well as all granular restores will start failing.
Typically this will occur 1 year from the certificates creation date so the best course of action is to remedy the situation as soon as you see the error message and before the self-signed certificates expire. The fix is to manually generate a new certificate as described in this Veeam User Guide. Please note that this process will automatically restart the Veeam Backup Service so it’s is recommended to ensure no active jobs are running.
Worth mentioning, Veeam administrators can select or import their own certificate but most organisations are still using self-signed SSL Certificates which are generated when Veeam Backup & Replication is installed.
I recently had the opportunity to visit Prague courtesy of the Veeam Vanguard program, this is my second year being a member of this fantastic community which is arguably one of the best evangelism/advocacy programs run by any vendor out there. While it was a long journey to get to Prague it was well worth it, to not only catch up with the other Vanguards but to get access to Veeam’s Product Strategy team, R&D personnel and Product Managers for in-depth discussions of everything Veeam related.
The summit consisted of two and a half days of sessions that included content filled to the brim with Veeam goodies ranging from upcoming updates to entirely new products that were still very early in their development cycle (kudos to Veeam for sharing). Veeam certainly was not holding back as questions raised from fellow Vanguards were answered honestly and truthfully, nothing was off the table including any questions about v10. All of this provided an insightful glimpse into the inner workings of the Veeam team and further cemented the value I place in the Vanguard program.
The real golden nuggets of information were found whenever we delved into the reasoning behind how and why certain features and capabilities were developed. For example, session speakers might detail the limitations of a particular feature and how they have worked to address them even if it might mean investing more time than anticipated in developing the feature. Yes, it’s a difficult decision to make but Veeam isn’t in the business of making half-baked software and it certainly shows in just how reliable their software has been to date.
Today’s article is written by Rose Herden, for those who haven’t had the fortune of meeting Rose she is the general manager of Saxons Learning Solutions, a Veeam Certified Architect (VMCE-ADO) and a VMCE trainer. Rose runs her own blog and helped found the Veeam ANZ user group.
G’day everyone! Rhys has yet again graciously allowed me to borrow some space on his blog. This time, I’m here to talk about logs!
A few weeks back, Rhys and I met Vish Venkatesh who spoke about Sublime and Rhys has an amazing article HERE. That tool deserves more downloads than it currently has!
Having read the article, got me talking to Rhys about a tool I usually tell students about on the last of the VMCE training. When we get to Module 12 (Troubleshooting), reading logs makes a wonderful appearance so I usually recommend CMTrace.
We’ll start off with the Install, how to get your logs in and the fun part! Tips!
I was fortunate enough to attend VeeamON Forum in Sydney last week, the company I work for (Data#3) actually sponsored a booth so I was armed with a scanner and had the task of talking to as many potential customers as possible. Usually, I would shy away from such a task but given the subject of the day was Veeam I had a great time. I even had one person mention he knew of this blog!
During the event, I had a great chat with Vish Venkatesh (short for Vishwajeeth) from Veeam also based in Sydney. Vish spent a year and a half as a Support Engineer before changing roles to an SE so I got a chance to ask about the inner workings of Veeam support.Continue reading →
Something that all Veeam administrators should consider is how secure the underlying servers running your Veeam software really are. To help improve security I always try and run through a few recommendations with each Veeam administrator I work with,
Inbound connectivity to backup servers from the Internet must not be allowed (3389 anyone?)
Any accounts used for RDP access must not have Local Administrator privileges on jump servers, and you must never use the saved credentials functionality for RDP access or any other remote console connections.
Ensure timely guest OS updates on backup infrastructure servers
A good resource for keeping up to date on Veeam security recommendations is here. I like to check it out every 3-6 months to ensure I’m still making the right recommendations to my customers.
One other thing I like to recommend in addition to the best practices above is enabling 2FA (Two-Factor Authentication) for all login sessions to underlying servers running Veeam components such as the VBR server, proxies and especially repositories. With 2FA, even if an attacker illegally acquires the correct username and password, the attacker is also required to gain access to the device used to receive the 2FA verification code. Often this device is a mobile phone or a security token which can easily be disabled if lost or stolen.
It must be noted that 2FA for Veeam consoles is currently not possible (it is a heavily requested feature though) and even with 2FA for login sessions into any Veeam servers there is still a risk that an attacker can access Veeam infrastructure via a Veeam Console running from another machine. This is why off-site/offline backups are so so critical in today’s world of ransomware. Leveraging Veeam Cloud Connect Backup with it’s Insider Protection feature is a great way to easily protect against this kind of risk.
This post will go into detail on how to quickly and easily and enable 2FA for RDP and local logon sessions connecting to your Veeam server.
A customer recently reached out to me with the issue below, while I hadn’t seen this issue before I thought I would check to see what I could dig up before they opened a ticket with Veeam support. “Error: Failed to call RPC function ‘StartAgent’: Timed out requesting agent port for client sessions.”
Veeam KB 1922 to the rescue, the cause of this issue is the ‘configuration of a Windows server within the Veeam console being set to have a limited number of ports to use‘ which thankfully can be resolved quite easily. To resolve simply go to the ‘Backup Infrastructure’ section in your VBR (Veeam Backup and Replication) console, go to the properties of any Windows servers that are being used by the job that is failing. So in this customers case, we can start with the backup proxy, then the backup repository, then if the problem still persists we can increase the port range on the VBR server as well. Once the port range is increased we simply click OK to apply the changes, I recommended we start with a relatively small number of ports (50) and increase if the problem still persists.
I haven’t figured out why in this customers case they encountered this port exhaustion issue, I find it curious as I’ve worked on much larger Veeam deployments before that didn’t encounter this issue. Ill need to perform some investigation and report back here once I learn more.
**UPDATE** Restarting the VBR server has resolved the issue without having to increase the port range. If it continues to happen we’ll look at increasing the port range but until then the default settings are good to stay.
So my first go at the VMCE-ADO exam was way back at New Orleans during VeeamON 2017. That experience could quite easily be described as an A+ for attendance but an F for effort, it was a textbook case of the ‘7 Ps’ and I walked away with a measly 50% result. Thankfully I was fortunate enough to be using an exam voucher which included a free reattempt so I thought why not give it a go while I can. That exam showed me first hand just how tough it really is but more importantly, I saw what it was going to take in regards to study to make sure I was really ready for the next attempt.
Unfortunately, it has been over a year since VeeamON 2017, in fact, VeeamON 2018 has already come and gone, yet I couldn’t delay sitting the exam any longer as the reattempt voucher was just about to expire. Timing was not the best as we just sold/purchased/moved houses 3 weeks prior and it was my sons 3rd birthday 2 weeks prior. I think I gained a few grey hairs this month… Nevertheless, the exam was booked and I couldn’t reschedule it without getting out my credit card.
One of the lesser known features of Veeam ONE is its ability to divide the virtual environment into various groupings and categories, essentially creating a view that is easier to digest from a business standpoint. This view can be valuable when you consider that most tools we would use such as vSphere client or SCVMM/Failover Cluster Manager are often configured to present information and data, for say, a more a technical perspective, something which might not be relevant or even make a whole lot of sense for all business stakeholders.
Both Veeam ONE Monitor and Veeam ONE Reporter will use this categorization provided by Business View. Veeam ONE Reporter, enables us to generate reports and build dashboards based on the categorisation created in Veeam ONE Business View. While Veeam ONE Monitor, we can monitor Veeam ONE Business View groups of VMs, hosts, clusters and storage objects.
By leveraging Veeam ONE Business View to group and categorise these objects into a hierarchy that makes more sense from a non-technical perspective such as office departments, projects, SLAs and much more we can easily review and report on resource allocation and utilisation based on these groupings.
Now that part is out of the way, I wanted to demonstrate how Veeam ONE Business View can be configured to help automate this process. We can configure set & forget rules and policies that organise objects into these group and categories. I’ve written this article to dive further into how one can configure Veeam ONE to categorise our infrastructure.