wIzQubes^TM unite traditional storytelling with mixed reality.

torytelling is an important part of human culture, having a role both in entertainment and in education. As digital technologies make inroads into the ways that we educate and entertain ourselves, it is natural to explore, too, how they may enhance the ways that stories are told. This article introduces interfaces that give children a novel way to control and interact with a story unfolding on a computer screen.

Traditionally, storytelling is a multisensory experience. Reading a book, for example, involves vision, touch for turning pages and pointing, and even speech when stories are narrated. Digital versions of books — referred to as e-books — are widely available, but although these often provide enriched multimedia support, a reader's interaction is mainly limited to the mouse and keyboard. The content of such books is generally predefined, and its display is mainly two dimensional.

Researchers in my group in the Department of Electrical Computer Engineering at the National University of Singapore (NUS) have been working on "mixed reality" interfaces that combine the tangible experience of a book with the capabilities of a story encoded electronically. Aimed at children, the stories merge 3D graphics with images captured by a webcam to give the impression on screen that the user is holding the story in their hands (refer to Fig. 1).

Our first project was called Magic Story Cubes. Unlike some mixed reality interfaces that are based around a traditional book, Magic Story Cubes uses an unfoldable puzzle cube as its physical interface. The puzzle cube is made of eight small cubes hinged together to form a large cube that can be unfolded and refolded in different ways, such that different pictures appear on its sides. The user's folding and unfolding actions are detected by a webcam and the image on screen is augmented with a 3D story scene. The scene changes depending on how the cube is folded. The folding of the cube brings excitement to the storytelling experience, but young children found the folding cube difficult to manipulate.

To address this problem, and to give users the opportunity to interact with and change the storyline, the NUS team has since developed a patented interface which led to the fully commercialized product wIzQubes^TM. In this system, a pair of solid cubes is used to control the mixed reality story. One cube has numbers on each side; it is used to navigate through different scenes of the story. The other cube has pictures on each side, representing different interactive items that can be introduced into the story. Bringing the two cubes together with the correct matching of the symbols on the cubes' top surfaces will advance the story (refer to Figs. 2 and 3). For example, in the story of Cinderella, the child can send the fairy tale's heroine home from the ball by bringing the ballroom, chosen with the number cube, together with the carriage, chosen with the picture cube. Narration and music will play at certain timings.

In a preliminary user study, kids aged from five to eight said that they felt much more entertained when they could physically manipulate and explore the story. We had asked them to compare the same story told through wIzQubes^TM, a book, and a video. One eight-year-old girl spent 3.5 continuous hours using wIzQubes^TM — the longest continuous concentration span we witnessed. I interviewed her, asking "why do you never feel bored when viewing Little Red Riding Hood more than 10 times?" She replied along these lines: "it is all different! This time I can see from inside the house, noticing that the Big Bad Wolf is sneaking into the window. Next time, I can see from outside the house, seeing that grandma is sick and laying down on the bed. And I can try different ways to explore the story!"

We intend to carry out further studies on how mixed reality helps children to understanding complex 3D structures and scientific concepts that require 3D visualization, such as the structure of the solar system. Formal user studies will be conducted with schools in Singapore to examine the usability of the interface as well as its effectiveness in the teaching process. In the meantime, wIzQubes^TM have already gone on sale. An NUS Figure: 3 An example of wIzQubes^TM being used to tell the story of Cinderella. In (a), the user flips the cube to play the corresponding story scene. In (b), the user matches the cubes' patterns to advance the story, here bringing a carriage to collect Cinderella from the ball. The system applies simple state transition models for interactive storytelling. The state transition is triggered only when the contents of the current state have been fully played to ensure the continuity of the storytelling.

Click here to download the full issue for USD 6.50