The Blog of Matthew Baxter-Reynolds

A perennial problem that I have in the office is worrying about backups. It of course doesn’t matter how many copies of something you physically have in the office, you can only solve the burglary/fire/flood problem by getting a copy of the data offsite. And you would think that with bandwidth being as cheap as it is these days that would be easy – but it isn’t as our data is too “chunky” to get offsite each night. What I want is one snapshot of a point in time of the operational databases that we run (one for source control, one for project management, one for invoicing and finally Exchange) copied offsite each night, but the reality is that we cannot get one of those databases offsite in a single night because they are all too big. In addition you have the problem of where to put it – i.e. where do you put the backups so that you can get them back easily should the office burn down.

We have solved one aspect of this problem using Jungle Disk and Amazon S3. For those who do not know, one of Amazon’s strategies is to expose its own IT infrastructure to the IT community in general as a paid service. One aspect to this serviceis Amazon Simple Storage Service (S3). For virtually no money, you rent space on Amazon’s servers.

We use Vault for our source control, and we have a separate application that monitors the Vault database for new labels. When a new label is found, the code under the label is retrieved and written to a folder. This means that we can go to our file server and see a list of every project and every build and grab the source when we we need it.

So, the problem with S3 is that you can’t just expose it as a fileshare, which seems to me to be a missed opportunity, but then again it is supposed to be a Web-based resource so you can understand that. What Jungle Disk does is exposes your S3 account as a network fileshare. This means that you can access it directly in Windows Explorer. What we did was modify our utility so that when the label was “got” from Vault, it would create a zip file (using #ziplib) and copy the zip file over to the file share and dump it straight up on Amazon’s servers. This means that customer source code is copied safely offsite the moment the developer sets up the label within Vault.

The only caveat is that Jungle Disk’s developers for some reason have not implemented it as a Windows service, which means that the shared network drive is only available whilst the user is interactively logged in. Whilst there are many utilities out there for hosting a regular executable as a Windows service, frankly I could not get any of these to play nicely with Jungle Disk and so we have a machine setup to logon as a local user and lock the console immediately.

The other nice thing about Jungle Disk is the licensing. For “desktop” use, you can install it on as many machines as you like providing it accesses a single S3 account. They also offer a “workgroup” option, which is something like $2/user/month which is more “proper” for corporate use.

My next objective is to stream SQL Server log files over to S3 to create a log shipping/offsite incremental backup of our databases. Then it’s just offsite Exchange to solve – although really I wonder whether some cloud-based solution for that is a better long-term plan.