This blog is here to show people why I wrote JS3tream.

It's been a while since I finally found a backup solution for my home file server. Since it's been working very well, I thought I'd write a quick entry for other people to benefit from. I'm not going to get into the "why you should backup". That is a question that can easily be answered with a Google search. I'm going to elaborate on the "why and how I implemented a backup".

My dilemma was a common and simple one. I have a Linux (Gentoo) based home file server where I keep a great deal of important data. I'm not talking about my MP3 collection here, I'm talking about years worth of irreplaceable personal photos and videos taken with our pair of heavily used digital cameras. Since 2001, I'm well into the 7-gig range, and with the birth of our first child, it's growing daily. These are memories, that I cannot afford to lose for any reason. I need a solution that will keep these memories safe from even the worst case scenario. A thief breaking into my home and taking my server, or heaven forbid, my house burns down. I can replace my computer, but not it's contents. My server is fully configured with RAID redundancy, but that won't help against fire.

I've been an advocate and user of Linux in various forms since about 1994. I've run some sort of Linux based server at home since about 1996. I graduated University with a Degree in IT back in 1995, and have made a living as a Software Engineer ever since. Being a big fan of Linux, I fully support the idea of Open Source and free software. So, I really wanted a decent backup solution for my Linux machine that was in the Open Source world.

There are a few obvious choices to secure my data at the level I need. I could house an external drive at a friends house and run RSync over SSH. Nothing wrong with that idea actually. But, I honestly don't want to flood my friends internet bandwidth. I could house this drive at work. We have a nice big pipe here. My boss has given me permission to do so. But, what if I loose/leave my job. Now I have to start over again. What about Carbonite? Awesome online backup idea, but they don't support Linux (yet). I want something that I can drop in place, that will automate the backup, and NOT have to keep a close eye on it. I want something off site, and I don't want to have to pay a huge amount of money either.

Some time in the middle of 2006, I stumble upon the Amazon S3 web service. Now this is cool stuff! Amazon will provide you with a massive amount of scalable storage for very low cost. So I start searching for utilities that will allow me to backup my server to S3. And, there are a few tools out there to choose from. Some work good, others not so good. My testing did not find a utility that worked with the level of success I wanted. What I want is a tool I can automate with cron that can also allow differential backups. If only TAR could talk to S3?

DING DING DING! Why not find a way to connect S3 with tar? Tar has been backing up Linux and Unix based machines for... well... ever! It's pretty much a proven tool. And, it supports differential archives. It also supports STDIO! Now, how to get TAR to talk to S3! I didn't manage to find a tool that would do what I needed. So... I wrote my own. This is where JS3tream (http://js3tream.sourceforge.net) comes into play. It was actually rather easy to write. A simple bridge utility that streams data through STDIO to and from the Amazon S3 storage system. This way, TAR is doing the hard work of archiving and compressing (gzip) my data. It's also tracking the differentials. All JS3tream needs to do is send the archive to Amzaon.

I released S3 in beta at the end of 2006. I was not ready to call it a "Solution" just yet. I needed testing on my part, and hopefully a few other people who had a similar need to mine. A few bug fixes and additional features, and at the end of 2007, JS3tream seems to be working very well. A handful of other Linux users have been helping me hammer out the bugs.

Now, with cron and a simple bash shell, my data is archived through TAR and S3 every 15 days. Twice a year I do a full archive, then every 15 days I do an incremental archive. The incremental archives save on time, bandwidth and S3 space. At present, I'm backing up a little over 30 gigs of data at a monthly cost of just under $15. The piece of mind I now have is well worth the cost.

Now, it's the beginning of 2008, and JS3tream is finally put to it's ultimate test. I just did an rm -rf on the WRONG directory! D'Oh! I really have to stop sudoing to root. What a rookie error. I just nuked a directory of data I didn't mean to! Thankfully, it was not any of my photos, but it was valuable none the less. I pull down the archives from Amazon, restore the TGZ files, and I'm right back where I was. I had not lost anything since my backup was recent enough.

So there you go! If your a Linux user, and are in the market for a decent online backup solution. JS3tream is well worth a look.

Shane