Paul Hammond's Blog

I have a question for you all.

I am in the HAVE LOST bucket.  The few times I have lost data in the past have certainly contributed to my current backup strategy.  More recently though, the shear amount of important data I have has shaped my backup strategy even more.

I have a single 500Gb disk that holds my entire "My Documents" directory.  When I look at it in detail, the drive contains:

• 23,665 photos, stretching over 9 years; this includes ALL the photos and videos of my kids first 8 years of life (75.4Gb).
• 2,090 video files from my DV camera and my HD camera, also going back 9 years (172Gb).
• 15,062 MP3 and M4A audio files from my iTunes catalog (65.7Gb).
• Many MIDI and Audio files from my music composition and production.
• Documents, Financial files etc.

As you can see, "My Documents" = "Almost My Entire Digital Life".  Losing any of this data would have been depressing.  Losing all of it would be completely heart-breaking.

While I was away on business, at some point this data drive died.  Completely.  No hope of connecting it to copy off whatever I could salvage.  It was simply unreadable.  Dead.

At this point, think about how you would feel, given the above data.  Would you be a little sad?  Sobbing?  Suicidal?  Or mildly annoyed?  Thankfully, I was simply "mildly annoyed".

Restoring My Data

Twenty four hours after arriving home, I am now back up and running with very little pain (apart from the cost of a new drive to replace the dead one).  My attention to a backup strategy completely saved my backside on this occasion.  However, this failure has also made me aware of another round of tweaking that I need to do to ensure total and flawless recovery the next time this happens.  AND REST ASSURED, IT WILL HAPPEN AGAIN.

So, what is my current approach?  I have a single drive that holds all of my data, as you have seen.  You could have multiple drives, this works just the same.

• I use the online backup service Mozy (http://www.mozy.com/) to backup most of my data.  This guarantees that I have off-site copies of my files.  An alternative service I have also used in the past is Carbonite (http://www.carbonite.com/).  Both have good features, and are reasonably priced.
• I use FolderShare (http://www.foldershare.com/) to create multiple copies of my most important files, or files that I use daily on my multiple home and work machines.
• I have a nightly backup that runs, which copies all of the files from the system and data drives to an external 1Tb drive.  I use a product called Retrospect that came with one of my external drives.  There is a great alternative called SyncBack that is either freeware (v3) or a very cheap option (v4).

In this particular case, it was from the 1Tb backup drive that I was able to salvage my data.  If that had failed, I would have resorted to a combination of Mozy.com data and FolderShare'd data on my other machines.  I attached both my 1Tb backup drive and the new drive to my laptop, and copied the data back to the new drive to create an identical version of what was there before.  Then I reattached the drive back to my desktop and booted up.  Once logged in, I simply reset the ownership and permissions of the "My Documents" directories to the correct settings.

Potential Flaws

There are a couple of possible issues with the above that I have been considering since the failure occurred.  Luckily they didn't occur this time, but I will be altering my approach to cater for these.

1. When the drive died, it disappeared from Windows.  The Mozy client then decided the files had been deleted, and proceeded to remove them from the current online backup set.  Luckily I can go back to older backup sets, but "deleted" files are only kept for 30 days.  If I had been away for a longer period of time, they may have been purged.
2. My nightly backup currently does a "mirror" of my filesystems, with no history.  Files that are added are added to the backup, and files that are deleted are removed.  Luckily, in this case, the software recognized that the drive was missing and skipped the backup.  If it had seen the drive and decided the files were no longer there (as Mozy did), it would have proceeded to wipe the data off the backup too.  I need to look at the options I have for my backup software to keep historical versions of files.

I mitigated the possible issues above by simply powering down the machine remotely to ensure no further damage could occur until I could be back at home there to figure things out!

Side Note 1: Windows Home Server

This incident has also made me think more seriously about buying a Windows Home Server.  One of the great features about this solution is the backup software.  It can backup complete machines, and has historical backups so you can get at older versions of your files if you need to.  Scott Hanselman did a review of Home Server that you can find here: Review - HP MediaSmart Windows Home Server

Side Note 2: FolderShare and Off-Site Backups For Family

One other thing I use FolderShare for is to backup data from various family machines.  I set FolderShare up on their machines and then invited my own FolderShare account to share their key folders.  The files get copied to my machine here.  That way, they benefit from having an off-site backup as well as duplicate data via my nightly backup.

Go And Do It.  Now.

Let me finish by saying this.  WHOEVER YOU ARE, WHATEVER YOU DO WITH YOUR COMPUTER, WHATEVER FILES YOU HAVE, MAKE SURE YOU HAVE A BACKUP STRATEGY THAT SAFEGUARDS *ALL* OF YOUR DATA.  Do it now.  Turn the TV off, and get working.  Don't stop until you have everything backed up, preferably in more than one place.  Your data drive could fail tonight.  It won't give you any helpful warnings before it does.  Make data loss an event of "mild annoyance", not complete despair.

If any of you checked out Sara Bareilles from one of my earlier posts and enjoyed what you heard, you should also take a listen to Colbie Caillat.  More great song-writing, great lyrics, and a fabulous voice.

Once again, Walmart has a Soundcheck session that you can watch.

I travel a reasonable amount for work.  I live 25 miles from London's Heathrow airport and 20 miles from London's Gatwick airport and neither is conveniently accessible to me using public transportation.  As a result, I regularly need to use a taxi company to deliver me to and pick me up from the requisite terminal.

I have been using the same company for a couple of years now.  For almost every single journey, they have been good enough that I would have given them a "completely satisfactory" rating.  They were always on time to pick me up from my house, the were always waiting for me a little sign bearing my name when I came through the arrivals door with even when my flight was delayed, and their cars were always clean and comfortable.

Note that I said "almost" in the last paragraph.

There have been 3 occasions lately when they let me down.  All three were for early pick-ups at my house (usually at 4:30am) for day trips to Dublin.

The first time, no car had arrived by 4:40am, so I called the company.  The dispatcher called me right back and told me that the driver had gone to Bayseed Road instead of Hayseed Road (where I actually live).  This seemed reasonable until I used Windows Live Local to discover that there wasn't a Bayseed Road within 200 miles of where I live; a little white lie to save face with the customer.  He arrived at my house at 4:50, and I got to the airport with plenty of time to spare, so I let it go.

The second time, once again I had to call the company at about 4:40am.  Again, the dispatcher called me back.  "Sorry, the driver overslept, he's just getting ready and will be with you in 15 minutes".  At least they were honest this time.  Once again, he arrived at about 5am and I just made my flight (although I didn't get the decent breakfast I wanted).  Being a fairly easy going guy, and given the continued excellent service the rest of the time, I let it go.

Yesterday, I had another 4:30am pick-up.  At 4:35am, when the car didn't show, I called the office.  No answer.  And there was no answer the following 15 times I called between 4:35am and 5:00am.  A no-show, and no explanation.  At this point I had to take my own car, leaving my wife without it for the day; she had plans so this was pretty inconvenient.  I called the office later on, and spoke to the owner of the company - he was very apologetic.  He told me he would call back with the reason for the no-show.  He didn't call me back.

Three strikes, they are out.  I am now looking for other taxi companies to do business with, and the original company have lost business to the tune of £80 for every single trip I make.  Despite a very high satisfaction rate for most of the journeys I made with them, just three failures were enough to make me "fire" them.

So how can you apply this story to your development team?  For me, this was all about trust, about making a commitment and sticking to it.  The taxi company was providing me a service, one that had potentially difficult consequences for me if it wasn't provided to a satisfactory level.  Despite providing a good enough service most of the time, the few times they failed completely killed any trust I had in them.  I would feel 0% confident about booking another 4:30am taxi from them.  Also, I had to chase them for status updates in each case, and for that last incident I couldn't get an answer.  If the drivers were running late, someone should have called and reset my expectations.  For that matter, someone should have called the drivers to make sure they weren't running late in the first place.

Once your team makes a commitment, they MUST do whatever it takes to keep that commitment.  If they cannot meet the commitment, then they MUST communicate as early as possible with their customer to understand the ramifications of missing the commitment, as well as collaborating on what an alternative delivery might look like.  Even though the trust between your team and your customer will be eroded a little, proactively facing negative news will help maintain a "win-win" attitude between the two parties.

I snapped these two pictures of Ruby the other night while she was snoring away on the sofa.  How is that a comfortable position to be sleeping in?  It must be, because it is pretty much her default position, even when sleeping on the hardwood floor.  I often find her upside down, right up against the front door!

Click on the images for a larger version.

[P.S. please excuse the state of the sofa - she was playing in the mud all day and decided our sofas needed some light brown highlights.]