I recently experienced a timeout error while offloading backups to a capacity tier (Azure BLOB). It occurred whenever Veeam offloaded large quantities of backup files simultaneously, typically any more than 6 backup files at a time would result in the offload failing.
This was a problem because the automatic SOBR offload process would process 40+ backup files at a time, most of which would fail until only 6 backup files remained in the queue, at this point the 6 remaining backup files would offload successfully. Typically there would be 250 or so backups in the offload queue, Veeam would offload these backup files for an hour until the timeout error occurred, then Veeam would start the next batch of 40 backups files to be offloaded.
Looking at the Veeam offload job logs (located in the main folder of the Veeam server logs, path ‘C:\ProgramData\Veeam\Backup\SOBR Offload’) we could see the following,
[18.08.2019 11:21:23] <176> Info – – – – Response: Archive Backup Chain: b14e8dd9-2351-4236-bd54-a08339859d49_40f33f92-ca5a-45ac-a2ec-d674efd0383d
[18.08.2019 12:57:26] <844> Error AP error: WinHttpWriteData: 12002: The operation timed out
[18.08.2019 12:57:26] <844> Error –tr:Write task completion error
[18.08.2019 12:57:26] <844> Error Shared memory connection was closed
A customer recently reached out to me with the issue below, while I hadn’t seen this issue before I thought I would check to see what I could dig up before they opened a ticket with Veeam support. “Error: Failed to call RPC function ‘StartAgent’: Timed out requesting agent port for client sessions.”
Veeam KB 1922 to the rescue, the cause of this issue is the ‘configuration of a Windows server within the Veeam console being set to have a limited number of ports to use‘ which thankfully can be resolved quite easily. To resolve simply go to the ‘Backup Infrastructure’ section in your VBR (Veeam Backup and Replication) console, go to the properties of any Windows servers that are being used by the job that is failing. So in this customers case, we can start with the backup proxy, then the backup repository, then if the problem still persists we can increase the port range on the VBR server as well. Once the port range is increased we simply click OK to apply the changes, I recommended we start with a relatively small number of ports (50) and increase if the problem still persists.
I haven’t figured out why in this customers case they encountered this port exhaustion issue, I find it curious as I’ve worked on much larger Veeam deployments before that didn’t encounter this issue. Ill need to perform some investigation and report back here once I learn more.
**UPDATE** Restarting the VBR server has resolved the issue without having to increase the port range. If it continues to happen we’ll look at increasing the port range but until then the default settings are good to stay.
Sometimes you see an error and think it’s going to take all night to resolve. Remoting Channel Sink UriNotPublished.. What the hell? Luckily the pointer to the /VeeamService is the dead giveaway though.
In my case, the Veeam Backup Service hadn’t started yet. Patience is a virtue, Rhys…