IllumiRoom

IllumiRoom is a Microsoft Research project that augments a television screen with images projected onto the wall and surrounding objects. The current proof-of-concept uses a Kinect sensor and video projector. The Kinect sensor captures the geometry and colors of the area of the room that surrounds the television, and the projector displays video around the television that corresponds to a video source on the television, such as a video game or movie.

IllumiRoom was first introduced at the 2013 Consumer Electronics Show.^[1] At the show, Microsoft, with Samsung, showed a video presentation of the system. At CHI 2013, Microsoft presented more details of the system, including a paper written with a researcher at the University of Illinois at Urbana–Champaign.^[2][3]

The system prototype uses a wide field of view projector and a Kinect for Windows sensor. The Kinect captures the color and geometry of the room environment and the projector renders images onto the depth map acquired by the sensor.

The IllumiRoom concept is based on prior work and research using focus-plus-context screens and projection mapping. The focus-plus-context technology generally uses a high resolution screen surrounded by a lower resolution display. Microsoft's CHI 2013 research paper cites Philips' Ambilight as an example of a focus-plus-context display. In the case of IllumiRoom, the television represents the high resolution screen and the surrounding projection is the lower resolution display. The purpose of this technology is to provide the user with additional visual information in the visual periphery, both simulating and taking advantage of peripheral vision. While the center of a person's gaze is in high-resolution and is sensitive to color and detail, peripheral vision is less sensitive to color and detail, but highly sensitive to movement.

IllumiRoom combines the focus-plus-context concept with real-time projection mapping. This allows the system to be used in any room, not just one where a television is surrounded by flat, white wall.

The Kinect sensor is used to calibrate the system and projection. The projector displays a system of gray patterns and the Kinect camera reads the size of the pattern across the projection in order to map the 3D environment. Once calibrated, the Kinect sensor is no longer needed for the IllumiRoom system and can be used for gaming.^[3] IllumiRoom was developed with the open-source first-person shooter Red Eclipse as prototype application.^[4]

The system can currently display video game video in one of several modes. These modes require the system to have access to the game's rendering process:^[3]

• Focus + Context Full: The full game content is projected around the television.
• Focus + Context Edges: Only high-contrast edges (e.g., buildings, characters, scenery) are projected around the television.
• Focus + Context Segmented: Game content is projected onto only a segment of the surrounding environment, most commonly the flat wall.
• Focus + Context Selective: Only select game content (e.g., bullets, flames) is projected around the television.

Without access to the game's rendering, several other projection modes are available:^[3]

• Peripheral Flow: The system displays a grid or starfield that moves with the video game camera around the television.
• Color Augmentation: The system changes the appearance of physical objects in the room to match the theme or look of the game by saturating colors, making them appear black and white, or creating a cartoon appearance.
• Texture Displacement: The illusion of distortion of physical objects in the room is created by the projector. The radial wobble effect creates the illusion that objects in the room are being affected by a rippling force field emanating from the television.
• Lighting: Since the project provides the lighting for the room, it can project lighting effects that match the lighting from the video game.
• Physical Interaction: Objects within the game can directly interact with the room environment. For example, a ball may bounce out of the game and fall onto objects in the physical environment.

Although widely expected to be used in an Xbox application, the researchers have stated that the technology is, for now, only a research project and not ready for commercial use.^[5]

RoomAlive, a related Microsoft Research project, also uses a depth camera and video projector in a projector-camera, or "procam" setup. It is a scalable system for dynamic, real-time interactive projection mapping in which multiple such procams can be used together in a room to generate an immersive unified projection mapping that is automatically adapted to the room environment, and which users can interact with physically. Unlike IllumiRoom, which implements focus-plus-context visual presentation centered on a television screen, RoomAlive focuses on spatial augmented reality applications.^[6]

In April 2015, Microsoft released the RoomAlive Toolkit, an open-source MIT licensed^[7] software development kit for calibrating a network of video projectors and Kinect sensors, which can be used to develop systems like those of the RoomAlive and IllumiRoom projects.^[8] The source code is available in their public repository.^[9]

• IllumiRoom - Microsoft Research
• IllumiRoom with HighRes pictures - Brett Jones
• RoomAlive
• RoomAlive Toolkit