The Project

Our project was to create a physical, motion based interface using cheap, common electronics and parts, and readily available software. The idea was to show that the implicit notion of human-computer interaction based upon mouse, screen keyboard is just that, an implicit which can be undermined using simple technologies.

Goals

Our primary goal was to expand our awareness of user interface technologies, with a view to implementing a prototype system which could form the basis of a physical interface for use in our undergraduate thesis. We set ourselves these conceptual goals:

• To explore & research alternative user interface devices.
• To conceptualize, and design a physical, gesture based, system capable of carrying out simple interface tasks.
• To investigate and evaluate for our purposes various methods of motion tracking that could be used in a limited timeframe.
• To build our system at minimal cost.
• To evaluate the potential use of this system in future projects.

Construction

After researching and discussing several ways of implementing the physical interface, including motion and tilt sensors directly connected to the computer, we conceived a basic system which would be based upon a vision-tracking/target method. We felt that this method was advantageous because of its separation from the actual computer, and because it represented a ‘discrete’ interface (one which could be worn and not used, or used and then left in place). Our vision based system would be based on three key components:

1. A PC running our software system
2. A device with coloured LEDs bright enough to be discerned from background objects
3. A camera connected to the PC to follow the LED device.

Having evaluated several programming environments, including Java and Processing, we decided to use Macromedia Director for our application. Java presented a high barrier to entry, as we would have been required to learn, or relearn the language, in a short space of time. The experimental Processing environment was extremely attractive, as it represented a middle ground between Java and a multimedia environment, but after tests the robustness and speed of the environment on anything but the fasted hardware was questioned, and we abandoned it in favour of a proven environment; Director. In future implementations of the system, a more robust environment would be used to directly communicate with, and interpret camera input.

Director, while familiar presented several challenges. We had to settle for a third party Xtra to expand its capabilities, and were therefore constrained by its functionality. In a high-level language such as Java or C++ we would have been able, although with difficulty, to manipulate information coming via the USB camera, but with Director we had to use the functions and information being interpreted by the Xtra. We chose to use the the freely available WebCam Xtra by Josh Nimoy to handle the interface between the camera and the Director application, and some of the basic colour tracking functionality.

Tracking Modules

Initial prototypes involved a basic ‘tracking module’, nicknamed “the orb”. The module would be an everyday object which could be picked up used, and left down again. Similar to a remote control, this ‘orb’ would be a type of wand interface, which would be followed by the camera. This system was rejected as a general purpose device, as making it feel like a part of the users body, beyond a mere remote control was difficult, and the movement it produced was less natural than we had expected. We think that it has potential usage in simple applications, particularly art installations where limited features are required, but we wanted a system that could be used in a wide range of industrial, domestic and commercial implementations, and which would translate physical movement naturally, beyond the idea of a ruggedised remote control.

Glove

After exploring these alternative ideas we returned to the idea of a glove interface, resolving that it was offered more potential, and created a more natural and flexible interface environment. A number of prototypes were made including, functional mockups for use in software testing, and then a final version which enclosed the wires and switches as much as possible to create a comfortable, attractive device. We tested many different types of LEDs to find the ones that offered the most colour contrast without flooding the camera.

Software

The software was custom made in Director, and utilized Josh Nimoy's webcam xtra for the basic tracking technology. Various version of the software were made taking basic

The final revision of the prototype system, which interprets the user's gestures and translates them into audiovisual controls. Gestures to the compass points of the screen produce the following actions:

 

Right = Fast Forward
Left = Rewind
Top Right = Play
Top Left = Pause
Up = Increase Volume
Down = Decrease Volume
Down Right = Skip
Down Left = Restart

 

Results

Through our tests we achieved a solid motion tracking technique, which works well in controlled conditions. Performance was heavily related to lighting conditions & background, due to the extremely limited pixel resolution of the camera, and the broad nature of the software used. Best results are achieved in a dimly lit environment with a neutral background.

For industrial usage, more 'intelligent' image tracking code would be needed. Motion tracking software is an industry in itself, and we achieved quite decent results considering the time and resources available. Use of alternative tracking solutions could be used such as tracking Infrared LEDs. Although that was out of the scope of our budget for this project, we strongly feel that it could reap significant benefits over a traditional camera and light source set up.

The gesture engine recognises and interprets actions, and while functionality is limited, it carries out definite tasks well based upon user motion. The goal of controlling the computer based on physical motion captured through a camera works well, as demonstrated in pmi21 where video controls are tied to specific gestures.

Future/Lessons Learned

This project represented a valuable learning experience, and gave us an insight into user interfaces, alternative user interfaces, hardware and electronics and software implementations of gesture systems.

Director does not represent a robust, scalable system sufficient for commercial development of the system. The challenges presented by the Xtra were great, but it did allow us to work at a high programming level, which was advantageous in the timeframe. For future development a more low-level approach would be needed where we could directly access data from the camera. Ideally, a tracking and gesture system would be hardware based for deployment in everyday applications.