Research is done on sensor-based recognition of coded human gestures with particular attention to gestures produced with one's hands and arms. For this, hand movements have to be measured as accurately and completely as possible: the position and orientation of the hands in 3-space, their position and orientation with respect to the human body, finger flexion and bending as well as the pressure distribution on the palms during grasping.
The collected sensor data has to be correlated to the morphology of meaningful gestures. The Research Center for Semiotics is composing a comprehensive dictionary of the emblematic everyday gestures that occur in Berlin. In addition, specialist gestures of various professions are studied. The dictionary provides a rich empirical basis for the gesture recognition experiments within the project. It will be made available both as a book as well as a CD-ROM. By definition (Ekman, Friesen) the meaning of emblematic gestures can be formulated in a verbal phrase. In a first step, the gestures are collected through video-interviews with Berliners and Non-Berliners. In a second step the video-encodings of suitable stimuli have to be decoded by other subjects. Besides the empirical work there are theoretical questions to be solved. According to the subdivision in three semiotic disciplines - syntactics, semantics and pragmatics - there are several levels of gesture description. As a syntactic metalanguage a categorial grammar for the Hamburg Notation System for sign languages (HamNoSys) is being elaborated. It will be of high value for systematic gesture recognition, or more generally, for any further symbolic gesture processing in a computer. On the semantic level, each gesture is described in terms of a core locution and a series of specifying operations (person, modality, time, ...). These problems are very closely connected with the pragmatic level, where the actual meaning of a "real life gesture" has to be analysed in context on the basis of a given dictionary entry.
The computer science research group concentrates on gesture recognition and on the further development of the TUB-SensorGlove. Measurements are conducted with several different sensors: An ultrasonic ranging system developed as part of a diploma thesis at the Real-Time Systems and Robotics Research Group measures absolute spatial position and orientation of the hands; in addition, finger flexion and grasp pressure distribution are captured by the TUB-SensorGlove, which was developed as part of a pre-diploma thesis at the same institute. The patented SensorGlove was exhibited at the Hanover Industrial Fair in 1993. Its variety of sensors and their high accuracy render it superior to many commercially available systems. An improved prototype was presented at the Hanover CeBIT'95. Twelve position sensors fastened on the glove's back measure the user's finger flexion with a resolution of approximately 1 to 1/3 of a degree. Twelve pressure sensors on the glove's palm measure contact forces occurring during object grasping.
Sensors currently used in the project will be supplemented by new ones which are being developed by the microsensorics research group. During the first phase of the project, the group concentrated on developing acceleration sensors for the glove. Accelerometers have a much higher resolution in measuring fast movements than ultrasonics. The glove's path in 3-space can be reconstructed mathematically if simultaneous acceleration measurements are made for all three dimensions. This calls for the development of new micromechanical devices, as up to now, sensors capable of triaxial, on-chip acceleration measurements are not available. The microsensorics group's newly developed accelerometers have to be integrated on the glove and suitable interfaces to existing hardware must be created. Then, the new prototype must be tested and calibrated. Only thereafter is it possible to obtain reliable data for gesture recognition applications.
Gesture data can be analyzed with many different pattern recognition methods and algorithms. Among other things, classical statistical methods, neural networks, genetic algorithms and fuzzy methods are being evaluated for gesture recognition. Partly, existing methods may be adapted to the new problem, partly, completely new methods must be developed. As data analysis is very time-consuming, a fast workstation is essential for real-time gesture recognition (in the project, a DEC Alpha is used).
There are many applications for sensor-based gesture recognition: From navigation commands (catchword "cyberspace"), applications in medicine and industry (eg. precise telecontrol of surgical robots, telecontrol of robots and machinery in outer space or at other locationswhich are too dangerous for humans to access), gestural control of novel musical instruments (i.e. "physical modelling") right up to enabling applications such as communication enhancements for the deaf-mute (communication among themselves and communication with hearing persons).
In the course of the project, a complete gesture recognition system will be built, consisting - amongst others - of modules for gesture input, gesture preprocessing and -analysis as well as of an integrated gesture database (containing multiple gesture dictionaries), graphics display routines for gesture data, and control modules for a robot and other devices. For demonstration purposes, one of the project's goals is the control of a robot and a browsing tool for the gesture dictionary.
The complete automatic recognition of human sign languages is a long-term research goal, small parts of which we hope to achieve in this project. If feasible, it would allow the deaf-mute to communicate with their environment in a simple natural way. For example, a "gesture telephone" could then be realized transmitting gesture data (captured by two SensorGloves and some other devices) via an ordinary telephone line to a computer displaying the data on its screen - either as written text or as a graphical image of moving body limbs (the videophone is not a viable alternative, as its transmission bandwidth and -rate are as yet far too low for the highly detailed and fast hand movements of a signing person). Instead of an optical display one could also imagine a direct outlet for speech, thus rendering the computer a translator between the hearing and the deaf-mute.
Contact:
Frank Hofman (Computer Science): fgh@cs.tu-berlin.de
Thomas Noll (Semiotics): noll@cs.tu-berlin.de
Thomas Velten (Microsensorics): velten@tubtmpo1.ee.tu-berlin.de