TIAN Feng

Structuring and Manipulating Hand-Drawn Concept Maps

Concept maps are an important tool for knowledge organization, representation, and sharing. On pen-based devices (e.g., tablet PCs), users can draw concept maps with a pen or stylus more naturally. Although a hand-drawn concept map is easy to create, it is still a challenge to digitize it and extract its structures (e.g., nodes and links) for further modification. Most current concept map tools do not provide full support for hand-drawn concept map creation and manipulation, largely due to the lack of methods to recognize hand-drawn concept maps. We propose a structure recognition method. Our algorithm can extract node blocks and link blocks of a hand-drawn concept map by combining dynamic programming and graph partitioning and then build a concept-map structure by relating extracted nodes and links. We also introduce structure-based intelligent manipulation techniques that support the management and modification of hand-drawn concept maps. Results from our evaluation research show high structure recognition accuracy in real time using our method, and good usability of intelligent manipulation techniques. Based on the structure understanding algorithms, we built a pen-based note-taking tool - IdeaNote. It not only supports natural note taking, but also supports efficient note manipulation by providing structured manipulation techniques for concept map structures and list structures.

Pen-based math calculator

We built a pen-based math calculator. Users can write mathematical expressions naturally with a digital pen, and the system recognizes the handwritten mathematical expressions and calculate the resuts. We can recognize some complicated structures. The system could be used on pen-based devices, such as mobile phones, Tablet PC, etc.

The project is partly sponsored by Nokia Research Center.

Recognition of Flexible Multi-Touch Gestures with Key Stroke Extraction

Gesture inputs on multi-touch tabletops usually involve multiple fingers (more than two) and casual touchdowns or liftoffs of fingers. This flexibility of touch gestures allows more natural user interaction, but also poses new challenges for accurate recognition of multi-touch gestures. To address these challenges, we propose a new approach to recognize flexible multi-touch stroke gestures on tabletops. Based on a user study on multi-touch unistroke gestures, we develop a gesture recognition method by extracting key strokes embedded in flexible multi-touch input.

GeoAssistor

A tool helps students analyze and solve geometry problems actively. It is built on tablet pc platform.
The project is partly sponsored by MSRA.

Tilt Menu

A one-handed menu for simultaneously generating the secondary input while drawing/interacting with a pen tip, that uses 3D orientation information of pens.

Tilt Cursor

A new cursor for enhancing stimulus-response compatibility of touchpad . The cursor can significantly reduce targeting times and response latencies.

PIBG Toolkit

A pen-based user interface toolkit used to develop specific Pen-based applications. The software architecture and interaction control can be constructed automatically. And developer can choose various widgets to build their application.

SketchPlan

A  pen-based training management system for China Diving Team, China Badminton Team, and China Women Water Polo Team.

Pen Office

A pen-based system for teacher preparing their presentation and delivering lectures to a class.

Pen Builder

A pen-based user interface design and run environment based on data centric theory. Pen-based UIs are designed by pen according to scenario-based method and saved as XML data. An interpreter will read data at running time to construct whole UI including static appearance and dynamic behavior.

ShadowStory

With the fast economic growth and urbanization of many developing countries come concerns that their children now have fewer opportunities to express creativity and develop collaboration skills, or to experience their local cultural heritage. We propose to address these concerns by creating technologies inspired by traditional arts, and allowing children to create and collaborate through playing with them. ShadowStory is our first attempt in this direction, a digital storytelling system inspired by traditional Chinese shadow puppetry.

Mobile learning in education

In many developing countries such as India and China, low educational levels often hinder economic empowerment. We argue that mobile learning games can play an important role in the Chinese literacy acquisition process. Based on an analysis of 25 traditional Chinese games currently played by children in China, we present the design and implementation of two culturally inspired mobile group learning games, Multimedia Word and Drumming Strokes. These two mobile games are designed to match Chinese children's understanding of everyday games.

Creative Pen 

A 3D sketch system for children easily creating and interact with virtual worlds.

Leveraging Psychophysical Data in Sports Training

Using psychophysical data to help coaches monitor and analyze Badminton players' states. Berkeley Tricorder, the device used in this project was donated from Berkeley Institute of Design, UC Berkely.

Multimodal Chinese text entry with speech and keypad on mobile devices

Proposes a multimodal text entry technique for Chinese. In this method, Chinese user can enter Chinese text by simultaneously using the simplified phonemic input method named Jianpin with keypad and speech utterance. The key of the technique is a multimodal fusion algorithm, which synchronizes speech and keypad input and fuses redundant information from two modalities to get the best candidate.

Multimodal error correction for continuous handwriting recognition in pen-based user interfaces

Introduce a crossmodal error correction technique, which allows
users to correct errors of Chinese handwriting recognition by speech. By fusing handwriting and speech recognition, the algorithm can correct errors in both character extraction and recognition of handwriting.

Motion Sensor device DIY 

I and some guys designed and implemented a new motion sensor device which includes 3D accelerometer, 3D gyroscope, and bluetooth module. Comparing with the widely used device: WiTilt, it's more powerful, smaller and cheaper. It can be applied in many interaction scenarios where computing vision technology could not be easily used.