Gesture Interface

Gesture Interface

How to apply equipment for a various kinds of person with disabilities?

We have collected data about gestures of persons with disabilities with using a various kind of image range censors for 5 years.The data is from 55 person with disabilities, amount of 211 parts of the gesture that is shown in Fig.1.

It is difficult to adapt all persons of disabilities but we collected gestures that actual users needed in using actual interface, in order to apply to many users.

We divide the data into hand (3 parts), head (3 parts), legs (3 parts), shoulder and other. This is because to apply for more users as fewer number of recognition engine as possible.



Figure 1.Classified Gestures Depended on Body Parts.(November 5th,2018)
 Hands,Arms  Folding Fingers 24
 Arm movement 16
 Upper arm movement 22
 Head  Hole head movement 34
 Mouth(Open and close mouth,Tongue) 32
 Eyes(Glance,wink,open and colse eyes) 31
Shoulder  Up and Down, move foward and back 12
 Legs  Open and close legs 7
 Step 1
 Tapping 5
Other than those above  - 27
 Total  Total number of parts 211
 Total number of test subjects 55

Multi gesture recognition engines

We develop 9 types of basic recognition engine according to the collected gestures shown in Fig.1. It is not enough number of persons with disabilities however; we aimed to collect 50 – 60 people’s actual data since we started this project.

We developed several recognition engine shown in Fig.2 with observing and classifying the data in detail.


Fig.2 List of Multi Gesture Recognition Modules.
  Part Gesture Module name
Hands,Arms Folding Fingers Finger
Hand,upper arm movement Front object
Fingers and hands fine movement Slight movement
Head Head right and left, up and down Head
Big wink Wink
Mouth,tongue Tongue
Shoulder Shoulder up and down, forward and back Shoulder
Legs Open and close legs Knee
Step Foot
Tapping Foot object
Slight movement
Site dependence Movement of the closest part of the camera Front object
none Fine motion of specified area Slight movement

Gesture Music

First, a user choses adaptable recognition engine and then, play music game. The system adapts the each user’s body parts and user’s motion. So the system learns the user’s motions.

Usually, users have to learn how to use the equipment (way of spatial movement) such as keyboards and mouse, however the point is that, our system learns user’s motion.