Synology ABB backup speed

[HCHTech]

I've got this solution out there for about a dozen clients and it is working very well. Seriously, no complaints at all...until now. I did a new setup last week for a client with a comparatively large backup footprint; about 11TB before dedupe or compression. Both the NAS and the server VM have lots of horsepower and network bandwidth, but I'm only getting 8MB/s progress after 4 days running the initial backup job. At this rate, I'm looking at at least a couple of weeks for the initial backup to finish. As of this morning, it shows 36% complete, as of yesterday morning, it was 35% complete - so maybe 65 more days until it's done! Plus, now that I'm hyper-focused on this long-running job, I'm noticing how little "in-progress" data you actually get with ABB.

I am doing deduplication & compression, which slows things down, but I'm not encrypting...and I know I get way better performance than this at my other sites that have less network bandwidth and lower-powered equipment. Synology support says they don't see anything wrong with the setup, so I should just wait it out, but it's driving me crazy - haha. I hate waiting, plus I can't have no backups while we wait, so I've kept the temporary solution going that we had in place while waiting for the new NAS, which is a nightly file-level backup to an external drive using Syncovery - which takes about 30 minutes, btw.

@Sky-Knight , I know you have this setup out there for some clients - do any of them have this much data? And if so, did you also have this agonizingly-slow initial run? Have you found anything you can recommend to speed things along?

 

[Sky-Knight]

Dedup is normal, compression and encryption are both a bit of a problem sometimes. I do compression sometimes, but it really can muck with busier image chains. My 4 bay units with 2 drives in them transfer about 462GB an hour. That's about 131MB/s...

So yeah something seems off here. Did you configure any throttling? If you didn't... try configuring some, I had a few backups go faster when throttled... strange but true.

 

[HCHTech]

Did you configure any throttling?

Hmm, no. I've never done that. Doesn't hurt to try it though - I'll try anything at this point. Can I do this hot, or should I stop the job make the change, then start the backup again? Does the number matter? I'll just limit it to 500MB/s, then...

Speaking of stopping the job...do you know exactly what happens if I cancel a job before it finishes and then start it again? Does it start over from the beginning, or where it left off? I never worried about that before, but with this one, it would be helpful if I knew I wasn't starting all over again from scratch with a restart...

Lastly, I felt I had to use compression - 11TB dataset going to a 24TB NAS hoping to keep several versions in the retention policy. Although I'll admit, I'm surprised at the amount of space saved by dedup alone - The storage tab says 8TB written so far (I've restarted the job 3 times) and 6.3TB saved by deduplication, with only 225GB saved with compression. Now I'm thinking I need to ditch compression.

 

[HCHTech]

Update: I removed compression from the job & rebooted everything this weekend - the Synology, the switch, the individual VMs and the HyperV Host. I also added 500MB/s throttling to the ABB settings. Just for fun, I installed the SynoCLI app so I could run some iperf tests. From the Host, I've averaging 4Gb/s with a single stream and 9Gb/s for 4 streams. Similar results from the VM. That sure looks good to me.

I tried copying a folder from the VM to the Synology using Windows Explorer - total size 1.6GB, containing 72 subfolders and a total of 2260 files. This took 45 seconds, for an average speed of 35.6MB/s. I know this is a ton of small files, but I was certainly hoping for faster.

Next, I went through all of the various Synology settings again and I found that I had NOT enabled Fast File Clone. I don't think that would affect anything ABB is doing, but it's a standard setting I use, so I enabled it anyway. Lastly, I disabled Transport Encryption Mode. I normally leave this enabled, but read that this setting WOULD have a definite impact on speed, so disabling it seemed like the right answer.

I restarted the backup and here are the results from the first 2 hours:

Speeds very fast from the start.

@ 20 minutes in, it had copied a total of 317GB, which works out to 264MB/s
@ 40 minutes in, it had copied a total of 605GB, which works out to 205MB/s
@ 60 minutes in, it had copied a total of 607GB, which means for the last 20 minutes, it was copying at only about 3.5MB/s
@ 70 minutes in, it had copied a total of 609GB, which means for the last 10 minutes, it was copying at only about 2.8MB/s
@ 120 minutes in, it had copied a total of 617GB, which means for the last 50 minutes, it was copying at only about 2.7MB/s

I can't think about this any more today, so I'm just going to let it go overnight again and see where we are tomorrow.

 

[nlinecomputers]

Are you sure the issue is with the Synology? What do the event logs on the host and VMs report?

 

[HCHTech]

N

Are you sure the issue is with the Synology?

No, I'm not sure at all - haha.

The event logs of the VM are not helpful. No errors at all in either the system or security logs since the backup started, the application log only show Perfnet 2004 errors in groups of 5 once each day. Research on that error basically said if the server service is running (it is), then to ignore the error.

No errors at all in the logs of the host since the backup started either.

 

[HCHTech]

Well it looks like I have a solution at long last.

The problem VM (with the 20TB / 11TB Used data drive attached) ALSO had an external USB hard drive attached (passed through from the host) that held some old archive data for a several-month long project they are in the middle of. It also had a 2nd USB hard drive attached that we were using for a file-level backup (using Syncovery) while we were working through the Synology ABB issue. When you backup the VM with ABB by connecting to the Hypervisor host, it backs up the entire VM, including attached drives. When I installed the agent on the VM itself and then setup a backup job treating that VM like a standalone PC, I could exclude those attached drives, and that gave me an 75MB/sec backup rate, which completed the 11TB in about 42 hours over this past weekend. Both drives are USB3, but I have no doubt the passthrough throughput via ABB has got to be a hindrance.

I missed this issue because the total size reported for the backup on the Details page when I was doing the VM backup was almost exactly equal to the total DISK size of my large data disk. So when I saw 20TB, I assumed it was disk size, and it didn't look out of place. It was only when I killed that backup after 5 days in progress and started a backup of the VM itself using the agent that this misinterpretation of mine came to light.

So, I would still have expected way faster than 75MB/sec given the hardware we have, but that's acceptable for now. Once we get to the point where we can remove those 2 passthrough drives, I'll try it again as a backup through the HyperV host to see if it's any faster.

Now that I have a full backup, I can see how much the deltas are each night and move forward with right-sizing the retention settings and then getting a Hyperbackup to a B2 bucket going. I wonder how many days it will take to move 11TB to B2 - haha. At least they have both FIOS and Comcast internet connections, so there's that...

 

[Sky-Knight]

Wait... wut?!?

Synology Backup for Business when aimed at HyperV throws an error about VSS and being unable to backup passed through USB devices, and then it automatically excludes them from future backups. This happens on both agent installs and direct to the hypervisor.

I know this only because I got the error on two of my servers and had to dig into it. So the question now becomes... Why the heck your platform wasn't properly reporting the passthrough drives as external.

There is literally a limitation of HyperV that prevents this from happening. I've run into this uniformly on HyperV 2012, 2016, and 2019.

but yes, those USB drives are so slow, you don't really want to be backing them up like this. So excluding them is correct.

 

[HCHTech]

Synology Backup for Business when aimed at HyperV throws an error about VSS and being unable to backup passed through USB devices, and then it automatically excludes them from future backups.

I've seen this error on physical machines with USB drives attached, but it isn't common for me to pass them through to the VMs - this was kind of a one-off. Active Backup for Business does NOT give you much at all in the way of troubleshooting. The only logs there are basically show start and stop (or cancel) times and that's it. There wasn't anything special done with the passthrough that I know of. I did the one drive with the archive data a couple of months ago, and the recent drive a couple of weeks ago so we could use it for some kind of backup while we were working out the ABB problem. Frankly, Syncovery works so well and is so cheap, I'm inclined to leave it in place for quick "I accidentally deleted my file" requests.

Also, USB3 should be able to do 85MB/s, which is about what I'm getting anyway - although I'm sure there is overhead involved with passing it through to the VM.

 

[Sky-Knight]

But there is no error on physical machines.

HyperV cannot VSS snapshot USB attached drives that are passed through. Windows can snap USB drives attached directly.

I can't explain how or why you're seeing that.