Mixed Reality Cookbook

What we perceive in the world is highly influenced by what we are looking for. European researchers have used this theory to create a convincing and engaging mixed reality, and they have put together a cookbook so others can do it, too. That is new news. In a famous experiment, a group of volunteers observed a video of two teams, one dressed in black and one in white, passing a ball between them. The volunteers had to count the number of times the ball was passed directly from one player in black to another player in black. They performed the task excellently. What they failed to notice was the man in the gorilla suit who walked on screen and jumped up and down during the game. It proved that what you see is strongly influenced by what you are looking for. In ophthalmology, researchers have found the eye does not see everything you perceive; neural processing fills in parts of the scene by inferring from those bits that are observed. In quantum physics, researchers discovered that particles change behaviour depending on whether you are looking at them or not. In field after field researchers have discovered that perception is not linear; it is fuzzy; and it can be strongly influenced by carefully choosing the right cues. The cues do not necessarily require complex technology. The Wii, a very popular gaming platform, abandoned the arms race of ever-more powerful processors and graphics cards and instead incorporated a simple motion sensor. Now users' gestures and reflexes drive the game, changing the pastime from a solitary, passive experience into an active, social one. Those two additions, sociability and physicality, dramatically enhance the sense of experienced reality engendered by the game. Up to now technologies, such as virtual and mixed reality, were thought by most to rely on more power, more technologically advanced interfaces, more animation and textures; but it now seems mixed reality is more powerfully and realistically evoked by combining perceptual dimensions with novel technologies in order to create a greater depth of experience. In IPCity, an EU-funded mixed reality project, researchers studied dozens of technologies to find those that dramatically enhance a user’s experience of a given task, all in an effort to increase citizens’ participation in civic life.

Using virtual experiences (or V-Ex if you want) like this to bring citizens closer to the city, the project embarked on what is probably the largest concerted effort, looking at the widest variety of mixed reality implementations, in recent times. The project created applications for town planning, gaming, environmental awareness and storytelling. It enhanced engagement with the social, cultural and historical fabric of a city through location awareness and mapping, and it developed social storytelling rooted at locations within the streetscape. Using a combination of easy-to-understand yet state-of-the-art technologies and location sensing, the researchers were able to create convincing cross-reality experiences by engaging multiple senses in parallel. The project took perceptual and mixed reality research out of the lab and into the real world with a combination of large-scale field trials and longitudinal studies. As a result, the IPCity team has developed cookbook-like guidelines for creating mixed reality experiences. Take Urban Renewal, an urban redesign application. Here, the researchers used a wide variety of media and interfaces to engage citizens in an exercise for redesigning an urban space. IPCity’s Colour Table is a particularly innovative interface, using tokens to represent elements within a scene, such as buildings or other objects. An overhead camera projected the design table onto a wall, revealing changes as they developed from a bird’s eye view. Another camera ‘interprets’ the tokens and projects virtual mock-ups onto a backdrop of the real site. Meanwhile on a screen, users can see how they have arranged the tokens, and on another they see how that would impact the real landscape. The entire set-up, along with other tools, is part of a mobile tent that is transported to the actual location for the new building, so participants can visualise the real-world environment. The combination of these technologies, along with subtle audio streams, evokes a very convincing air of engagement in the task.

More information:

http://cordis.europa.eu/ictresults/index.cfm?section=news&tpl=article&BrowsingType=Features&ID=91375