Beware of data corruption devastating your Workflow Lynchpin

Of the four project development variables – scope, cost, time and quality – quality isn’t really a free variable. The only possible values are “excellent” and “insanely excellent”, depending on whether lives are at stake.
“XP Explained” by Kent Beck

Circus Ponies Notebook (CPN) is the lynchpin in my workflow — my notes end up in CPN files which I then use to write my own works. It works great and I rely highly on it. There is a lot of information in these files, information I will not get back — either because it takes too much time & effort or because they were ideas that I will not have again.

Thus, you can imagine the kind of moment I had when I saw the following message a couple of days ago:

problem
The error message you get when you content is gone. It goes along with a tightening of the chest, a clenching of the anal sphincter, and a frantic mental search of the number and integrity of backups.

Looking at the file itself, the information seems to have been … gone. The file size is much smaller than it should have been.

The terrible and sneaky thing here is that the notebook file opened without problems. I could add and work on the content of other pages without problems. It was only when I tried to access this page (and others) that I saw this error.

Thus, it is a hard to detect but devastating error.

At the moment, I am concerned. I open my notebooks frequently, but I do not go through all pages (there are simply too many). The idea is to have the information available when I need it. And for that the information has to be available.

problem2s
One of the pages that lost its content.

I have contacted Circus Ponies Notebook support and asked for possible causes of this error. They replied within hours (they are a great company :-)). Apparently, CPN is highly protective of its data — as it should be. Other applications might have damaged the file (which is actually a folder), but besides DEVONthink I don’t use any with CPN. I don’t think that DEVONthink is responsible, but I don’t know. They wrote:

I know you mentioned Devonthink and use it for storing your Notebooks. I’m not sure if that could be related (we’ve not had reports of Devonthink problems like this that I can recall).  If you only use the Notebook and Devonthink on a single Mac, I would surmise that this should be OK, since Devonythink comes from a reputable OS X software developer. But, maybe if you are syncing the Devonthink files via some means, that could be a possible area to investigate.

In general, when moving a Notebook (this includes syncing and e-mailing) it’s best to archive it first to prevent the data issues that sometimes can arise with other applications and file packages.  To archive a Notebook,in the Finder, Control-click the Notebook and choose “Compress” from the menu that appears.  Then you can sync, e-mail, and move it by other means with much more confidence.

I think it’s more likely that it was either a fluke or that it happened when I copied data from another (encrypted) notebook in it. All other notebooks are fine (flipped through each and every page). So, so far it’s strange … and unnerving.

In any case it showed me a serious vulnerability in my workflow — corrupted data might not be noticed for quite some time. The problem is that I keep my backups on rotation: I back up data to different hard disks in a predefined order and when I use the same hard disk I overwrite the last backup. Thus I could easily overwrite the last working copy. The whole situation reminds me a bit of Douglas Adam’s quote:

“The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly goes wrong goes wrong it usually turns out to be impossible to get at or repair.”
Douglas Adams, Mostly Harmless

CPN is highly reliable and very well done, but bugs and errors can happen. And unfortunately, data corruption in the file is very hard to detect.

I have planned against it now, by:

  • setting up appointments in my calendar to check the integrity of the files every month (at least for a year), and (more importantly)
  • keeping separate backups of the CPN files that are not overwritten by future backups

In case this ever happens again I can go back to the last working version of that page and copy & paste together a new working version of the entire notebook. I think the crucial aspect here is that something that is highly reliable can go wrong — with devastating effects. And if it’s the lynchpin of your work, you should have an uninterrupted line of backups to fall back upon.

5 Trackbacks / Pingbacks

  1. Warning: Do not store Circus Ponies Notebook files in DEVONthink | ORGANIZING CREATIVITY
  2. I’m a digital squirrel — what are you? | ORGANIZING CREATIVITY
  3. Circus Ponies Notebook Version 4 available | ORGANIZING CREATIVITY
  4. Circus Ponies Notebook 4 | ORGANIZING CREATIVITY
  5. Using Circus Ponies Notebook to collect and sort information for your thesis (or article) | ORGANIZING CREATIVITY

Comments are closed.