How to apply equipment for a various kinds of person with disabilities?
We have collected data about gestures of persons with disabilities with using a various kind of image range censors for 5 years.The data is from 55 person with disabilities, amount of 211 parts of the gesture that is shown in Fig.1.
It is difficult to adapt all persons of disabilities but we collected gestures that actual users needed in using actual interface, in order to apply to many users.
We divide the data into hand (3 parts), head (3 parts), legs (3 parts), shoulder and other. This is because to apply for more users as fewer number of recognition engine as possible.
Figure 1．Classified Gestures Depended on Body Parts.（November 5th,2018）
| Hands,Arms | Folding Fingers | 24 |
| Arm movement | 16 | |
| Upper arm movement | 22 | |
| Head | Hole head movement | 34 |
| Mouth（Open and close mouth,Tongue） | 32 | |
| Eyes（Glance,wink,open and colse eyes) | 31 | |
| Shoulder | Up and Down, move foward and back | 12 |
| Legs | Open and close legs | 7 |
| Step | 1 | |
| Tapping | 5 | |
| Other than those above | - | 27 |
| Total | Total number of parts | 211 |
| Total number of test subjects | 55 |
Multi gesture recognition engines
We develop 9 types of basic recognition engine according to the collected gestures shown in Fig.1. It is not enough number of persons with disabilities however; we aimed to collect 50 – 60 people’s actual data since we started this project.
We developed several recognition engine shown in Fig.2 with observing and classifying the data in detail.
Fig.2 List of Multi Gesture Recognition Modules.
| Part | Gesture | Module name |
| Hands,Arms | Folding Fingers | Finger |
| Hand,upper arm movement | Front object | |
| Fingers and hands fine movement | Slight movement | |
| Head | Head right and left, up and down | Head |
| Big wink | Wink | |
| Mouth,tongue | Tongue | |
| Shoulder | Shoulder up and down, forward and back | Shoulder |
| Legs | Open and close legs | Knee |
| Step | Foot | Tapping | Foot object Slight movement |
| Site dependence | Movement of the closest part of the camera | Front object |
| none | Fine motion of specified area | Slight movement |
Gesture Music
First, a user choses adaptable recognition engine and then, play music game. The system adapts the each user’s body parts and user’s motion. So the system learns the user’s motions.
Usually, users have to learn how to use the equipment (way of spatial movement) such as keyboards and mouse, however the point is that, our system learns user’s motion.
