The Problem
For my YouTube channel, I have been using Final Cut Pro on my Mac to assemble and edit all of my videos. I generally create one library for each video, sometimes I will create multiple videos in one library, but all on the same subject. It often matters as to when I think I will be recording and editing them. I have no hard set rules.
When I am working on these videos, I keep the Final Cut library on a faster Solid State Drive. But when I am finished, I move it to one of two 8TB external Hard Disk Drives as an archive/backup. But these drives are starting to fill up.
What I need is a place to archive these old projects, so I never lose them. If I need them, I probably won’t need them right away. And a way I can do that cheaply. I could just buy another hard drive, but my worry is that they can and do fail. Even if just sitting on a shelf. I’d still need some other kind of backup for my backup.
Commercial Online Cloud Backup
I started looking at a commercially available cloud based solution. The ones I look at have some pretty steep prices, but they seem to be targeted to the user that might need ready access to the files in the archive, or even unattended backup/archiving of files. That’s not what I am looking for.
Amazon Web Services
I am an AWS Solution Architect so I started looking at the cost of storing these files in Amazon’s Simple Storage Service, or S3. There are several different storage tiers with S3. And some of them can be costly.
For example, if I were to upload all 16TB of archived projects I have right now, it would cost well over $350 a month for the standard, ready access storage class. But I don’t need that. I’m only interested in moving the files up that I haven’t looked at for a year or more. And I probably will never need to look at them. But I’d like to be able to, even if I don’t need them right away.
The least expensive S3 storage class is called Glacier Deep Archive. That same 16TB of storage would only be a little more than $15 a month. Which is perfect for me. Even storing 100TB is only around $100 a month, 1/3rd the price of the standard tier S3 tier and 1/6th the price of the more popular online backup solution. I’m not saying those other options aren’t worth it, they are for what they provide, which is a lot more than what I need.
My Solution
So I set off, with the help of AI, to write a script that will scan a directory and upload my Final Cut libraries to Amazon S3 Glacier Deep Archive. I wrote it in BASH to make it portable to other operating systems and for other uses. So far, I have uploaded well over a terabyte of data to S3 and I might be paying $1.00 for the month. It is going to take me a while to upload all of these files, but it sits and does it in a terminal window.
The script scans the directory structure, locates any Final Cut libraries, creates multi-part zip files using 7Zip, then used the AWS S3 API to upload the multiple parts. Once all parts have been uploaded, it’s completed. I still manually verify that the file made it before I delete the local version. If it fails at any point during the upload, I can execute the script again and it will skip all of the parts that have already been uploaded.
I’ve uploaded the script to a public github repository. It is licensed under the Creative Commons NonCommercial License if you want to check it out for yourself. I enjoyed writing this and it’s been nice getting some of that hard drive space back.