4k video editing 50TB storage required!

[goldmercury]

My good friend is looking to update his video editing rig. He's PC and has been working with Avid for many moons.

Anyway he has stated to me he's jumping to 4k and requires 50TB storage area for video.

I have spec'd up a system and was going to go down the NAS route with a 10GBe connection back to the rig however he works alone so I'm thinking of just getting a tower case and putting in a rock solid RAID card and 8 to 12 of the new 8TB Seagate drives.

Did not even know we were at 8TB! http://www.hotukdeals.com/deals/8tb-internal-seagate-hard-drive-hdd-205-07-bt-shop-delivered-2138685

Question I have is what is a good RAID card for the purpose of video storage and will the 8TB drives do the job? The mobo will be based on the x99 chipset.

 

[lcoughey]

Ouch...what are you planning to use for the backups? The cost and time to recover the data from such a large RAID will be huge.

 

[goldmercury]

Good question! They are on a budget so I'm guessing having a rock solid RAID array is key with proper alerting when disks fail.

The backups will be an after thought I know that for a fact but I will probably advise a synology nas. What would you recommend on a budget?

 

[lcoughey]

Good question! They are on a budget so I'm guessing having a rock solid RAID array is key with proper alerting when disks fail.

The backups will be an after thought I know that for a fact but I will probably advise a synology nas. What would you recommend on a budget?

Seriously, an after thought? I don't think you understand how RAIDs work, if you think that the RAID itself will be a safety net.

1. You are likely going to insert drives of the exact same model, probably from the same assembly batch. If one drive has a defect, the odds are, they will all be prone to the same defect around the same time
2. The larger the drive, the higher the probability for bad sectors. The RAID controller only reports a drive as bad when a drive reports back a bad sector during use. So, there may be bad sectors on multiple drives that aren't being touched by general use. But, when a drive does report bad and goes offline and is replaced by another drive to rebuild the array, you will have to touch every sector on every drive, even the blank zones. In doing so, you will hit another drive with a bad sector and it, too, will go offline. Now you have a failed RAID with no backup.

It doesn't matter what the client's budget is. You must insist that the entire project include a solid backup. If they cannot afford the backup, they cannot afford the RAID solution.

 

[goldmercury]

Thanks for the info - ignorance is bliss I guess. So what would you recommend for backup of such a large volume of data?

 

[lcoughey]

Thanks for the info - ignorance is bliss I guess. So what would you recommend for backup of such a large volume of data?

I wish I had an answer. I just specialize in data recovery and always refer my clients to my resellers for any backup related support. I should like to think that 1 of the almost 15,000 members of this forum has a few suggestions on how to backup a 50TB system.

 

[jbartlett323]

What RAID level you going with?

An array this large, yeah backup is an issue. Fact is, only way to do backup on an array this large, is to duplicate your storage solution and replicate all data. Obviously, this will literally double the cost, and quickly become cost prohibitive to most small businesses, and especially one-man-bands. Therefore, because some redundancy is better than no redundancy, solid RAID hardware and proper Level choice is critical.

To mitigate Luke's scenario, I would recommend a RAID level greater than 6, and also building the array out over a period of time, or buying drives from different sources (to mitigate same manu date/local). This big of an array, performance becomes an issue also, so keep that in mind. The Synology runs a hybrid RAID, looks like basically a RAID 6 with an extra hotswap but somehow using all space. Makes for a fast, redundant array. It can also handle and custom RAID level you choose.

Backup of an array this large is not feasible for anyone on a budget, and you just have to build it so the only thing that can really stop it is a natural disaster. For 4K video, I can understand an array this size, and he's gotta have it to do his job. Hopefully he can quickly make enough money that suddenly a backup solution is easy/cheap! I agree to insist as hard as you can for backups, but if they can't afford it, redundancy is better than a single drive if built right.

 

[phaZed]

Geez 50TB!, I would definitely setup RAID 10 or a Couple of RAID 10's in a single box. Like jbartlett says, I don;t see any other way to backup that type of volume without duplicating the initial setup.

I would recommend looking at a Synology Mid or High end business storage solution, perhaps like this:
https://www.synology.com/en-us/products/RS2414+

It's scalable and claims up to 144TB. I imagine it may be possible to setup multiple volumes, say, one for the working set and one as a backup set. Really, though never worked with such a unit so I am not aware of the in's and out's of this thing!

One thing is for sure... whatever you do it isn't going to be on a budget!

 

[goldmercury]

I'm thinking RAID10 - preformace and redundancy.

The Synology mid range looks tempting. I've been on other forums asking about the 8TB disks and they are slow, not ideal from what I'm doing.

I'm probably going to have to go to the avid forums and ask a few guys that already have a workflow on this just to get the correct insight. Really appreciate comments above. I'll report back with findings. Rig will be built for April.

 

[lcoughey]

Even with RAID 10 or RAID 6, you had better backup the data. I've done enough RAID 0,1,5,6,10 data recovery jobs to be confident in saying, they all need a backup. I'd put the cost of the double drive redundancy towards the cost of a solid backup. Single, double or triple drive redundancy does nothing to prevent human error (accidental deletion), corruption, theft or virus (ransomware).

 

[JRDtechnet]

I like Synology too for this but FYI on the Synologies the Diskstations models (DSXXXX) are only GbE you have 2 ports for Link Aggregation but it's not upgradable to 10GbE, The Rackstation models (RSXXXX) also only come with regular GbE interface but you can put a supported 10GbE NIC in the expansion slot. As far as backup goes you're not backing up 50TB to a cloud backup probably take years before it gets caught up if it ever gets caught up. You're best bet would be to get a second Synology backup to that...the synology software has this feature. I'd get a Diskstation model for this task since they are a lot more compact and easier to move around. I'd do a weekly backup and then keep it a friends house or at the office (or if he works at an office...bring it home) or a minimum keep in on the opposite end of the house or under the bed where it can be grabbed at a moments notice...or better yet if he has a large fireproof safe. Or wait until this comes to the market http://www.sony.net/SonyInfo/News/Press/201404/14-044E/index.html although a shutter to think how much the deck and each tape is going to cost!

 

[HFultzjr]

http://money.cnn.com/2013/05/07/technology/enterprise/cray-supercomputer/

Should be a good start to build upon.

Make sure to back-up.....maybe Crashplan unlimited with Google Fiber LOL.

 

[gikstar]

Who really needs 50Tb to start off with, even considering what he has up to now?
I would think building this up in increments would work better, adding on as one needs it and when the business can afford it.
I think I would like to have a Dell Precision 7810 workstation running one or two Xeon E5 v3 cpu's with a 500Gb SSD drive for programs, a DAS storage unit (since he works alone) running Raid 6 with the amount of Tbs of disk space he needs to start off with and a Backup device of your choice (I like Qnap while others prefer other brands).
When the money flows, add another DAS storage unit and backup for that unit. I would think that unless you're working NASA or the NAS, you don't really need 50Tbs at your fingertips often, so you could attach the 1st DAS unit when needed.

 

[YeOldeStonecat]

Dattos NAS currently just go up to 36TB....great offsite package for backup, and revisions. But too small.
So look at a larger Synology unit, or an EMC model like the Isilon, pickup a pair of them, and host one in an offsite data center somewhere...do you can do the duplication. Because offsite backup with a traditional cloud service would be prohibitive for costs.

 

[Mercenary Roadie]

50TB for 4k video is nothing, in fact that is on the low end.

What type of video editing do they do? Movies? Commercials? Shorts?

You can end up with a lot more than 50TB of data with all of the active sessions you are working on and the back-ups for each of them.

 

[lcoughey]

I know that this is opening an old thread, but I'm just finishing up an assessment of a 12 x 4TB RAID 6 RAID with a 3 drive failure.

 

[Alexey]

Well that may be an old thread, but I think it is worthwhile to comment on this:

only way to do backup on an array this large, is to duplicate your storage solution and replicate all data

Exactly duplicating a storage system and use the duplicate for backup is not a good idea. Duplicate system will have the same dormant issues as the original system. It is better to have a backup system which does not share anything with the primary system. Preferably different hardware, different filesystem, and different hard drive vendor.

 

[mr m]

I know that this is opening an old thread,

I just read it and did not realize it was two years old!

I had to comment anyway. You are right on the money about the need to backup RAID.

For a client with 50 TB I would recommend tape or a cloud based solution. If the client does not trust the cloud and needs speed LTO tapes are the answer. The last time I looked you can get LTO-7 cartridges that hold up to 6TB of uncompressed data. I think LTO-8 might be available now but I am out of the loop, pun intended. And some of these tape drives can accommodate multiple cartridges if needed for large unattended backups and compression can also increase the capacity.

By physical design all reel or cartridge tapes are written and read sequentially but through software they can appear to be a random access device like a platter or flash memory. Some LTO cartridges are WORM (write once, read many) devices. I used these for a law firm years back. They had two separate basements full of tons of paper in fire-resistant cabinets that was slowly transferred to tape storage and they required the data to be permanent after written to tape.

The bottom end of a single cartridge LTO system starts at about $2500 and goes up, way, way up.

 

[mr m]

Well that may be an old thread, but I think it is worthwhile to comment on this:

Exactly duplicating a storage system and use the duplicate for backup is not a good idea. Duplicate system will have the same dormant issues as the original system. It is better to have a backup system which does not share anything with the primary system. Preferably different hardware, different filesystem, and different hard drive vendor.

And multiple, off-site copies.

 

[Krynn72]

I know that this is opening an old thread, but I'm just finishing up an assessment of a 12 x 4TB RAID 6 RAID with a 3 drive failure.

That would be hilarious if it was OP's customer.