Media-rich/enhanced eBook

Reading the Horizon Report 2010 (PDF) reminded me of an idea I had years ago. I was creating a book of quotations which somehow got out of hand and ended up as a 2600 pages .pdf-file with several thousand quotations. A few of these quotations came from movies, which means that it would be possible to use a video clip of the quotation to “enrich” the quotation.

I didn’t try to do it at that time, but since electronic books have become mainstream, I’m beginning to wonder. It was impossible to print the quotations book due to its page count, but for an electronic book, page count does not matter (as much). And it would be possible to include these video clips — and audio clips. Even create intros to chapters or pages, and extros. And reading the Horizon Report showed me that this is not only useful for quotation books.

Imagine a fiction book where you have background sounds according to the setting. If the protagonist is in the forest by day, you hear birds singing, in a tavern, you hear the fire cracking, on a ship you hear the sounds of the wood and the flapping of the sails.

The effect should be as in computer games, but must not be repetitive like in the early games. Instead of fixed and soon-to-become annoying sound clips, generators should use short clips and combine them in always new combinations. For example, I once made a CD for a needy/self-absorbed ex-friend with encouraging and astonished sound bits, and some sound files that contained no sound (each 1 to 7 seconds long). Played on shuffle it would provide her with an audience on demand in always new combinations. Played on “shuffle + repeat all” she could bitch the whole day without stressing me … no, I did not give her the CD. I realized that is was actually quite evil and I do not want to use my creativity for that. I rather ended the relationship, but it was very cathartic for me to create this CD. And the same principle should work here.

What would be needed? An ebook-reader would need to be equipped with a function that recognizes ebook-reader commands within the text. For example an author would not “only” write the text but go through it again and enter commands like {birds+singing+happy@occasionally} and the ebook-reader would access an open library of sounds, searching for sounds that are tagged with all three terms and play them occasionally when the reader is on that page (for unusual or specific sounds, the book could pack it’s own sound library with it). Commands for intros/extros, time limits for sound repetitions, and on demand animations and sounds should also be available. Given that an effect might only make sense when the reader reads a paragraph that is, e.g., on the middle of the page, the device could calculate the average reading speed per page (excluding outliers due to external influences) and start playing a sound that is anchored in the middle of the page after 1/2 of that time. Oh, and — of course — a disable sound button and audio jack would also be needed.

In the future, an ebook reader could be equipped with an eye tracker (using button presses on the screen to auto-calibrate it continuously — we usually look at a button when we press it, especially if there is no physical part you can feel like with a virtual button), not only to automatically flip the pages, but also to play sounds at the exact moment a person reads a corresponding sentence. Imagine you go with your eyes over a scene description where two swords clash and the device gives this out as a sound. It should not play dialogues — that is your imagination — but the sounds, the non-essential and non-written background sounds that come in the setting the characters are in.

Hmm, given that mobile devices become more and more powerful (I have high hopes for future iPhones/iPod touch’s and a coming(?) iSlate), I think there will be some interesting enhanced/media-enriched books in the future. Some application for creating these books do already exist (e.g., Sophie 2.0), albeit not for mobile devices (and I mistrust any writing environment that needs an own reader solely for that purpose — exporting to HTML/Flash would be more sensible — who wants to install another reader in slow-Java?).

Hmm, it would probably need a new job/skill set to do so — kinda like the guys who do the sound effects for pornos. For this effect to be really effective (or work at all), the background effects author must not overdo it. Imagine that the sounds are on low volume and fade in and out almost imperceptibly, so that they do not distract you. It would be very interesting to see what it would do to transportation, to your ability to imagine the story. Critics might say that it is the job of the author to create these sounds in the imagination of the reader, but I wonder. Would it be so bad to enhance a text this way? To provide non-essential and non-described sounds? Or should they be left out simply because they are non-essential? Would it “unify” the reading experience because the imagination gets constraint, or would it actually help readers imagining the worlds they read? I mean, not everyone can imagine a tavern in a fantasy world and ‘hear’ the sounds that usually come with it. Perhaps it could help readers feel at home in stories that play in settings they do not know …

But I’m curious what happens in this area in the future. Maybe it’s the birth of a new genre … maybe only a fad, we’ll see.