Sign Language Classification

Sign Language Classification Submitted By: Mustafa Ali Raza Reg. No. 11PWCSE0909 Abdul Qadir Reg. No. 11PWCSE0954 Jamal Khan Reg. No. 11PWCSE0918 Supervised by: Engr. Madiha Sher A thesis submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Systems Engineering Department of Computer Systems Engineering University of Engineering and Technology Peshawar Certificate and approval This is to certify that the work contained in this thesis under the title of “Sign Language Classification” was carried out by Mustafa Ali Raza Abdul Qadir Jamal Khan Under my supervision and that in my opinion, it is fully appropriate, in scope and quality, for the degree of BSc Computer Systems Engineering from university of engineering and technology Peshawar. Approved by: Supervisor: Engr. Madiha Sher Signature: ___________________________ Verified by: FYP Coordinate: Dr. Hammad Ali Zai/ Engr. SaleemUllah Signature: ___________________________ Verified by: Chairman: Dr. Laiq Hassan Signature: ___________________________ Department of Computer Systems Engineering Stamp:  Abstract Sign language is a process which is used widely for the communication of the deaf community. Generally sign language is considerably more formable for communication which helps to promote and to increase the communication. In different countries of the world there are different forms and variation of the sign systems exists, in which the ASL is said to be the international one. There are less number of principal notions and assigned appearances in the sign language. The basic idea behind this work is to develop a system of sign language which will facilitate the deaf community and will increase the process of communication. The project focuses on implementation of a software model for sign language recognition system using classifier. The approach for this is to recognize the gestures and to analyze the features using classifier. The recognition is done using principal component analysis and by using classifier the features of the gestures are analyzed.  Acknowledgments Thanks and praises to the Allah Almighty for granting us success in completing this project. This thesis is the result of the blessings and dedication of many individual persons, especially our parents. Our sincere and heartfelt appreciations go to all of them. First of all we would like to thank Engr. Madiha Sher, the most humble and knowledgeable person for she was not only our supervisor but she was the binding force of our group. She helped us a lot in this project and always appreciated our questions and queries. The method for solving the problems was very easy and understandable. Special thanks to Dr. Nasir Ahmed for his guidance and constant support throughout the project. We whole-heartedly want to thank him and appreciate his special interest in this project. In addition we would like to thank our teachers and fellows who were ready to help us including Engr. Jibran Khan, Engr. Shakir Ullah, and Engr. Sidra Gul. We had a wonderful experience working as a team. We hope that our work will be of some use to our juniors for their research and projects.  List of Abbreviations SL : Sign Language SLR : Sign Language Recognition ASL : American Sign Language FSL : French Sign Language 1D : One Dimension 2D : Two Dimensional LER : Low Error Rate PCA : Principal Component Analysis S.D : Standard Deviation E.M : Electro Magnetic CCU : Central Control Unit GM : Gaussian Model GMM : Gaussian Mixture Model  List of Figures NO TITTLE PAGE 3.1 Non Maximum Suppression 19 3.2 Results of the Canny Algorithm 20 6.1 Recognition of the hand gesture using PCA 35 6.2 Average probability of the Hu-Moments 39  List of Tables NO TITTLE PAGE Effect of number of features on 6.1 classification 36 Results of PCA using different 6.2 Classifiers 37 Classification Results Using Hu- 6.3 moments 38 6.4 PCA vs Hu-moments 40  Table of Contents Abstract.............................................................................................................. i Acknowledgments................................................................................................ ii List of Abbreviations............................................................................................ iii List of Figures..................................................................................................... iv List of Tables....................................................................................................... v Table of Contents................................................................................................ vi Chapter1............................................................................................................ 8 Introduction........................................................................................................ 8 1.1 Overview............................................................................................... 8 1.2 Introduction to the Project........................................................................8 1.3 Problem Statement................................................................................10 Chapter 2......................................................................................................... 11 Literature Survey............................................................................................... 11 2.1 Overview............................................................................................. 11 2.2 Previous Work...................................................................................... 11 2.3 Issues with the Gesture Recognition..........................................................13 Chapter 3......................................................................................................... 15 Feature Extraction............................................................................................. 15 3.1 Overview............................................................................................. 15 3.2 Definition of Edge..................................................................................15 3.3 Edge Detection...................................................................................... 15 3.4 Canny Edge Detection............................................................................ 16 3.5 Steps of Canny Edge Detection.................................................................16 3.5.1 Removing of Noise.......................................................................17 3.5.2 Finding Gradients........................................................................17 3.5.3 Non Maximum Suppression.........................................................18 3.5.4 Double Thresholding...................................................................19 3.5.5 Edge Tracking By Hysteresis........................................................20 3.6 Principal Component Analysis.................................................................21 3.7 Steps for PCA....................................................................................... 21 3.7.1 First Step..................................................................................... 21  3.7.3 Third Step.................................................................................... 22 3.7.4 Fourth Step.................................................................................. 22 3.7.5 Fifth Step..................................................................................... 23 3.7.6 Sixth Step.................................................................................... 23 3.7.7 Seventh Step...............................................................................24 3.8 Moments and Invariant..........................................................................25 3.9 Hu-Moments........................................................................................ 25 Chapter 4......................................................................................................... 27 Classifiers......................................................................................................... 27 4.1 Overview............................................................................................. 27 4.2 Random Forests Classifier.......................................................................27 4.3 Methods of Random Forests Algorithm......................................................28 4.5.1 Feature Weighted Method...........................................................28 4.5.2 The Normalized Weights..............................................................29 4.4 Random Tree........................................................................................ 30 Chapter 5......................................................................................................... 32 Methodology..................................................................................................... 32 5.1 Overview............................................................................................. 32 5.2 Heading............................................................................................... 32 5.3 Flow Chart........................................................................................... 34 Chapter 6......................................................................................................... 35 Results............................................................................................................. 35 6.1 Overview............................................................................................. 35 6.2 Classification Using PCA........................................................................35 6.3 Effect on Accuracy by Varying Number of PCA Features..............................36 6.4 Results of Different Classifiers.................................................................37 6.5 Classification Using Hu-moments.............................................................38 6.6 Hu-Moments Vs PCA............................................................................. 39 6.7 Conclusion........................................................................................... 41 References:....................................................................................................... 42  Chapter1 Introduction 1.1 Overview In this chapter the introduction to the project is discussed. The introduction of the sign language and also different types of the images and gestures are discussed. This chapter also covers problem statement of the project. 1.2 Introduction to the Project As the common computer input devices introduced in the field of the technology does not changed a lot. The reason behind this is that these devices are satisfactory. As now days computers becomes so common in our daily life, that the research made in this field make it easy to introduce new applications and the hardware. To deal with the computers we have are restricted to keyboards, light pen and also now a days keypads are introduced to communicate with the computers. These devices become so much popular but it limits the speed. As the technology is growing the vision based interfaces are becoming popular among the people and with these devices the computers are able to see. So this development will lead us to develop new interfaces of the devices through which the commands that are not executed with the current input devices will be executed by these devices. The relation or the interaction of the humans with the machine is generally called the Man Machine Interaction or it is also called the Human Computer Interaction. This refers to the relation of the human 8 beings with the machine. The two main characteristics that are functionality and usability should be kept in mind while designing the HCI model. Functionality of the systems is the services that are provided to the user of the system by the system, while usability of the system is that the specific tasks that the user does can be done precisely. The gestures are way of communication between the machines and humans and also the gestures are used for the communication among the people by using a language i.e. sign language. [1] Now a day’s recognition of the hand gestures has taken a huge interest. There are numerous applications of the hand gesture recognition like controlling the machines, computer games replacing the mouse. The most significant structures are the sets of the gestures exist as a sign language. In the sign language each and every gesture is assigned to a specific meaning. Sign language is basically a way of communication between the humans and the deaf people. As with the advancement of technology sign language has become common using the sensors in the data gloves. The main idea behind the sign language is to recognize the images of the gestures of the human hands. Early the recognition was difficult but with the advancements in technology it is becoming easier and more precise. The satisfactory recognition of the hand gestures are provided by more advanced computer interfaces, allowing a three dimensional model in which rotation of hand gestures are also recognized. The hand gestures are generally divided into two categories of static and dynamic. [2] The first category that is static hand gestures is basically a specific hand pose and configuration which is represented by single image. The second category of hand gesture is dynamic in which we deal with 9  moving gestures and it is represented by a number of images not with a single image like static. In our project for hand gestures we focused on the static images. 1.3 Problem Statement Generally the impaired people in our society have a lot of problems in communicating with the normal people. The problems which they go through is to convey their thoughts and also their ideas which they want to transfer and share with the normal people of the society. The main problem through which this kind of situation arises is the limitation and less knowledge in the sign language. By this the impaired society people lose hope and also interest in the normal activities and it effect their communication with the people and they spent an isolation time in the society. To overcome this great problem and also the handle the situation concerned with these people a lot of sign languages recognition systems were developed by different researchers but still there is need of accuracy and effectiveness in recognition of the signs. The proposed system will focus on solving these problems and try to develop an accurate and effective sign recognition system. 10  . Chapter 2 Literature Survey 2.1 Overview This chapter covers the literature review including the previous work done in this field and also covers the methods which were used in designing this system. This issues related to this project are also discussed in this chapter. 2.2 Previous Work Sign languages (SL) are natural languages that use different means of expression for communication in everyday life. The primary mode of communication for many deaf people from many countries in the world is the sign languages (SL). But the question arises that who invented sign language? [3] As no one has got the answer to this question, but it is most likely that the ones who created the sign languages are the deaf themselves who created a variety of gestures in order to communicate. The researchers have studied and state that there are three categories of the hand gestures. Among the three categories of the hand gestures 11 the first category is glove based analysis. Second category is vision based analysis and at the last the analysis of the drawing gesture. In the first category that is glove based analysis the sensors that are mechanical or optical are attached to the glove that converts the finger movements into electrical signals by which the hand posture can be determined. Generally the sensor which is attached to the glove is an acoustic or a magnetic sensor. For such type of recognition there are toolkits are available with the glove to be used for the recognition of the hand gesture or SL. The second category is vision based analysis which is based on the way through which the humans get the information about their surroundings although its satisfactory implementation is really difficult. So for this different approaches are tested. One of the approach is to make a three dimensional model of the human hand. In this model the images of the hand are matched to the model by one or several cameras and the parameters which correspond to the orientation of the palm and the angles of the joint are estimated and later they are used for the gesture classification. This three dimensional hand skeleton model was developed by Lee and Kunii. Marked gloves were used to simplify the model. The third category was the drawing gestures analysis. In this type of gesture recognition the stylus is used as an input device. The drawing gestures analysis can also lead to the recognition of the written text. [4] The majority of the work in the hand or sign gesture recognition is done for manipulation of the apparent environment and also for the symbolic communication by using the mechanical sensing. Number of problems arises by sensing mechanically the hand gestures which 12 includes the E.M noise, reliability and also accuracy, so visual sensing makes the recognition of the hand gesture more practical. The ASL recognition fully consolidates the data gloves. In addition to this a Data glove based system is discussed by the two Japanese Takashi and Kishino. That system could recognize the 34 of the 46 Japanese gestures using the hand orientation coding and joint angle technique. From their research it seems that for PCA and also for cluster analysis the user made 46 gestures 10 times which provide each gesture a reasonable time. These systems are interesting technically but they are affected from lack of training. In machine SLR and outstanding work has been done by Parish and Sperling. These two have done a deep research in the bandwidth which is necessary for the conversation of the sign from the sub sampled and temporally images. In most of the cases the systems are isolate or static gestures while in many other cases there are finger spelling sign. In general the gestural visual channel is the main principle characteristic of FSL and of SL. [5] Thus, compromise between articulation, visual interpretation and comprehensibility is there. With a continuously purpose of economizing signs are articulated, while to retain a meaningful message which can be easily be interpreted by another person. By means of hand parameters which have generally hand-shape, movement, orientation and location, the meaning is conveyed but also the facial expression, mouth movements, and chest movements are mandatory for conveying the message. These parameters are articulated in the space and occur simultaneously. In our project we focus on the hand gestures recognition. In this we used an available data set of the images of the hand gestures and all of the other computations are done on that available given data set. 13  Canny algorithm is used for the recognition of the hand gesture. And then the result of this is passed to the PCA for further manipulations and computations. 2.3 Issues with the Gesture Recognition There are some issues in the hand gesture recognition. In gesture recognition the first step is the segmentation process in which the image is divided into segments and theses are separated by lines. The process of segmentation depends on the gesture type because for dynamic the hand gesture should be traced and for the static hand gesture only segmentation can be carried out. The useful and common method for the segmentation is the skin color of the hand, because segmentation is a challenging process. There are various methods and tools that use skin pixels and also the non-skin pixels in order to model the hand and these methods are known as the parametric and non- parametric. The parametric techniques are Gaussian Model (GM) and the Gaussian Mixture Model (GMM) and the non-parametric technique is generally base on the histogram. [6] The problem in this model is that they are greatly affected by the changes in the illumination condition. There are also some factors which make the process of the segmentation more difficult and they are complex background, low quality images and the light changes. 14 Chapter 3 Feature Extraction 3.1Overview This chapter covers the all the methods used for the feature extraction. The first section covers the canny algorithm for the edge detection. This chapter covers the principal component analysis method of the feature extraction and also the Hu-moments in recognition of the signs. 15  3.2Definition of Edge Those points in a digital image where the intensity of light change abruptly are called edges. Edge is generally the portion of the object where the light changes in such a way that its intensity suddenly varies. 3.3Edge Detection The set of operations performed mathematically whose goal is to identify those points in an image where the light intensity changes abruptly or those points have some discontinuities is called edge detection. If the same problem performed in one dimensional signal then it is called step detection. Edge detection is widely used in image processing and computer vision. Edge detection is also used for features extraction as well. Different algorithms used for edge detection but canny is the best among others for edge detection. 3.4Canny Edge Detection The canny edge detector was developed in 1986 by John F. Canny. Canny edge detection is a famous edge detection algorithm Canny is most widely used edge detection algorithm because of the following few reasons  Low Error Rate: 16  Low Error Rate means a better detection on only active edges. [7]  Good Localization: Good Localization means the distance between edge pixels detected and real edge pixels have to be minimized.  Minimal Response: Minimal Response means only one detector response per edge. Canny edge detection is a multi-stage algorithm and we will go through each step. 3.5Steps of Canny Edge Detection Canny edge detection algorithm contains the following steps.  Removing Of Noise  To find gradients  Non maximum suppression  Double thresholding  Edge tracking by hysteresis Each step is described in the following sub sections. 3.5.1 Removing of Noise Since edge detection is susceptible to noise so the first step we will perform is to remove the noise and make the image smooth. The Gaussian filter is used for this purpose [8]. An example of Gaussian Kernel of size=5 that might be used as shown below. 17  1 3.5.2 Finding Gradients The canny algorithm basically finds edges where the image’s intensity changes abruptly. These areas are found by determining gradients of the image. Gradients at each pixel in the smooth image can be determined by using the sobel-operator. Two steps are applied here. The first step is to find the directional change in x and y direction by using the following equations. [ ] −1 0 1 KGX = 2 0 2 2 −1 0 1 [ ] 1 2 1 KGY = 0 0 0 3 −1 −2 −1 The second step is to determine the gradient magnitudes using Euclidian distance measures by using Pythagoras theorem as shown in equation (4). It is also some times can also be calculated using Manhattan distance as shown in equation (5). 18  |G|=|√ G2 x+ G2 y| 4 |G|=|Gx|+¿ |Gy| 5 The direction of the edges must be determined and stored as shown below in equation (6). Gy θ=tan −1 (|| ||) Gx 6 3.5.3 Non Maximum Suppression After getting the gradients magnitude and direction, now we want to remove any unwanted pixels that not constitute the edge. For this, we will look at very pixel and checked it if it is a local maximum in its neighborhood in the direction of gradient. 19  Figure 3.1 Non Maximum Suppression Look point A located on the edge in the vertical direction. Gradient direction is perpendicular to the edge. Point B and C are in the direction of gradient. So we will first check point A with other two points B and C to confirm that whether it forms a local maximum or not. If it forms local maximum so it is considered for next stage, otherwise, it is suppressed (put to zero). 3.5.4 Double Thresholding Those pixels which are remaining after non maximum suppression are still marked with their strengths pixel- by –pixel. Many of the pixels will be true edges in the image, but some may be because of noise or because of color variations for instancing due to rough surfaces. The simplest way to differentiate between them is to use a threshold. [9] Thresholding basically converts the image into a binary image. So that only edges stronger than a certain value would be preserved. The canny edge detection algorithm uses the double thresholding. Edge pixels have value stronger than the high threshold value are marked as 20 strong; while those whose values are less than the value of low threshold are marked as weak edges. 3.5.5 Edge Tracking By Hysteresis Only the strong edges we have find out will be involved in the final edge image, because the strong edges are the only edges which are extracted from the true edges in a digital image. However there will be some conversation on the weak image pixels, because weak edges are either extracted from the true edges or because of noise etc. Now in order to get correct result so these edges should be removed. The criteria to determine which case the weak edge belongs to is that, usually the weak edge pixels caused from true edges will be connected to strong edge pixels. [10] Figure 3.2 Results of the Canny Algoritm 21  3.6Principal Component Analysis The PCA is totally a mathematical technique which has different steps including S.D (standard deviation), Eigen vectors, Eigen values as well as calculation of covariance matrix. Each of the mentioned steps is discussed below. 3.7Steps for PCA The various steps involve in the PCA are discussed below. 3.7.1 First Step The first step of PCA is to get the required data. The data will be in the form of images of different gestures of the hands as we are applying PCA for the recognition of the hand gestures. After the data in the form of the images is taken the other steps of the PCA will be applied on the given images. 3.7.2 Second Step The next and the second step that follows the second step of the PCA are to find the mean of the given input data. To find the mean the data dimensions are added and then they are divided by the total dimensions. [11] PCA does a fine work on the given data by subtracting the mean of the data from each of the dimensions. So this gives the average data across each and every dimension. All values are subtracted from this average data. So this step gives the mean of the input data. 22 3.7.3 Third Step The third step of the PCA is to get the covariance values. The best and the useful way for getting the possible covariance values among the different dimensions are to calculate them and after the calculation put these values in a matrix. The formula for the calculation of the covariance matrix is given as; Cmxn = (ci,j, ci,j, =cov(Dimi, Dimj)) 7 The calculated data is arranged in a 2x2 dimensional matrix as this data will be (2D) two dimensional. 3.7.4 Fourth Step The next and the most important and necessary step of the PCA is the finding of the Eigen Vectors and the Eigen Values. This part gets more importance because this will give us the essential features on the basis of which we will perform other processing on the data. For calculating the Eigen vector and Eigen values we need the covariance matrix which is indeed a square matrix. So for the covariance matrix we will find the Eigen values and also the Eigen vectors. These two are more important because they give more useful information about our data. The example of the Eigen value and the Eigen vector is as under; .0580833989 Eigen Values = ( 1.17402661 ) 8 −.624167545 −.56672299 Eigen Vectors= ( −.56672299 −.624167545 ) 9 23 3.7.5 Fifth Step This step is related to the fourth step because the manipulations and also the compression of the data are carried out after the Eigen values and the Eigen vectors are calculated. As we look at the previous section, the values of both Eigen vector and Eigen values are quite different. The Eigen values are quite different values. The principal component of the given data set is the highest Eigen value of the Eigen vector. This gives a clear view of the significant relationship among the data dimensions. Our main goal is to collect the required data and removes or ignores the unwanted data. After this the main work is to construct a feature vector. The construction of the feature vector involves the Eigen vectors which we take from the covariance matrix and with the Eigen vectors in the columns of another matrix forms a new matrix. This new constructed matrix is given below; Feature vector = (eigen1 eigen2 eigen3 eigen4…… eigenn) 10 3.7.6 Sixth Step After doing all of the above steps of the PCA the last and final step of the PCA is to adjust the required data in a specific manner. In this step the important and required features which are gained are actually adjusted in a specific and proper manner in the form of matrix. After this the transpose of the same matrix is taken and we multiply this matrix with the original matrix. This is given by the following relationship as under; Final or Resultant data = original data x transposed data [12] 24 The data which we get after the computation is totally different from the original data and on the basis of this data we do the rest of the computations. We generally select the vectors or values with a higher values which are we will use in for further recognition process. 3.7.7 Seventh Step In this step the older data is retrieved. In the sixth step of the PCA the new data is obtained by the conversion of the older data. Sometimes the older data is requires for the computations, so in this step the new data is converted into the older data which is generally required to get the initial given data. As this step is very important so it is carried out more carefully to retrieve the initial data. So before we move ahead it should be kept in mind that all the Eigen vectors will have exactly the same original data because if the Eigen vectors are reduced then it will definitely lose some of the information. As the following relation is used for obtaining the newer data; Final or Resultant data = Original data x Transposed data So the above relation is re-arranged in order to get the older data: Transposed data = Original data -1 x Resultant data Original data -1 → Original data T It should be kept in mind that to get the actual original data back it is needed to add the mean to the above relation. Transposed data = (Original data -1 x Resultant data) + Mean of the original data 25  3.8Moments and Invariant Moments and the invariants have been analyzed to characterize the patterns in images different applications. The most important moments include geometric moments, Zernike moments, rotational moments, and complex moments. 3.9Hu-Moments Hu derived six absolute orthogonal invariants and one skew orthogonal invariant which are based upon algebraic invariants. [13] These are independent of position, size and orientation and also independent of parallel projection. Moment Invariant are widely applied to image pattern recognition in a variety of applications due to its invariant properties to image pattern recognition, image registration and image reconstruction. The moment invariant are used to trace image patterns regarding the image translation, scaling, and rotation under the assumption of images with continuous function and noise free. The invariant features can be achieved using central moments, which are defined as follows: 26  11 The pixel point ( x , y ) are the centroid of the image f(x, y). The centroid moments μ pq computed using the centroid of the image f(x, y) is equivalent to the mpq whose center has been shifted to centroid of the image. Therefore, the central moments are invariant to image translations. Scale invariance can be obtained by normalization. The normalized central moments are defined as follows: [14] 12 Based on normalized central moments , Hu introduced Seven moment invariants : 13 27  The seven Moment invariants have a property of being unchanged under image scaling, translation and rotation. Chapter 4 Classifiers 4.1 Overview This chapter covers all the classifiers related with this project. In this chapter the random forests and the forest tree classifiers are discussed. The first section covers the random forest and its algorithm and the next section is about the forest tree classifier. 4.2 Random Forests Classifier Random Forests is the classifier used in image processing for extracting features of the images. Random forests are a scheme for building a classification ensemble with a set of decision trees that grow in randomly selected subspaces of data. Random Forests is tree based classifier consists of many decision trees. Each tree gives a classification and the output is the aggregate of these classifications. Experimental results have shown that random forest classifiers can achieve high accuracy in classifying data in domains of high dimensions with many classes. Recently, interests in random forests have been growing in image classification and bioinformatics. Several methods have been proposed to build random forest models from subspaces of data. Among them, one of the most popular forest construction procedures, proposed by Brieman is to randomly select a subspace of features at each node to grow branches of a decision trees, then to use bagging method to generate training data subsets for building individual trees, finally to combine all individual trees to form random forests model. 28  Random forests are the combination of trees predictors [15] such that each tree depends upon the values of a random vector sampled independently and with the same distribution for all trees in the forests. Image data has many features which are uninformative to a class feature. During this forest building process, informative features would have the large chance to be missed, if we randomly select a small subspace from high dimensional data. 4.3 Methods of Random Forests Algorithm The random forests classifier has two methods of implementation. I. Feature weighted method II. The normalized weights 4.5.1Feature Weighted Method Consider an M-dimensional feature space {A 1, A2, …. , AM}. We present how to compute the weights {w1, w2, …., wM} for every feature in the space. These weights are then used in improved algorithm to grow each decision tree in random forest. To compute the feature weight, we measure the in formativeness of each input feature A as its correlation to the class feature Y. A large weight indicates that the class labels of objects in training data are correlated with the values of feature A. Therefore, A is informative to the class label of objects and has a strong power in prediction of class labels of new objects. 29 Given the class feature Y which has q distinct values or classes, denoted as yj (for j=1,…, q). The feature A can take p values, denoted by ai (for i=1,…, p). If A is numerical, it is discretized with a supervised discretization method. Let D be a data set consisting of (1, 1, q, p, j, i, ij, λ) = = Σ Σ data samples. [16] The number of samples in D where A=ai and Y=yj is denoted by λij. Given the contingency table of feature A and the class feature Y of a data set D, the chi-square statistic based correlation is computed as p q ( λij−t ij)2 Corr(A,Y) = ∑∑ t ij 1 i=1 j=1 Where ij is the observed frequency given in the contingency table and tij is the expected frequency that can be computed as: q p ∑ λij X ∑ λij j=1 i=1 tij = q p 2 ∑ ∑ λij j=1 i=1 The larger the corr(A,Y) measure is, the more informative the feature A is with respect to the class feature Y, and the higher weight is assigned to feature A. 4.5.2 The Normalized Weights 30  Feature weights are normalized for feature subspace sampling. Supposing the correlation between a feature Ai and the class label feature Y is corr(Ai,Y) for i=1,…,N. [17] We define √ corr ( A i ,Y ) N Wi = ∑ √ corr( A i, Y ) 3 i=1 The extraction of square root of the correlation is a common technique for smoothing. It can be easily seen that the normalized weight wi measures the relative In formativeness of feature Ai. This weight information will be used in feature subspace sampling when designing our algorithm. 4.4 Random Tree A Random tree is built by choosing components from random subspaces. Each random tree is trained on entire training data. The random trees randomize the selection features and decision criteria. A random Tree is structurally Homogeneous to a classical decision tree. Classical decision tree and random tree are grown recursively top down. Classical decision trees find a decision criterion that best divides the multi class training data into two groups at each node. A decision criterion consists of an attribute and a related threshold. 31 Random trees on the other hand randomize the decision criteria. A random decision criterion is defined by a random attribute and a random threshold. A random tree thus divides the training data on completely random attributes. This randomization mainly addresses high dimensional data and generalization. The random Forests algorithm can be summarized as:  Draw ɳ tree bootstrap samples from the original data. [18]  For each bootstrap sample, grow a regression tree.  Predict new data by aggregating the predictions of the ɳ trees. The error of a forests tree classifiers depend on the strength of individual trees and correlation between them. For random forests the, an estimate of an error can be obtained, based on training data. 1- At each bootstrap iteration, predict the data not in bootstrap sample (Out-of-bag) OOB using the tree grown with a bootstrap sample. 2- Aggregate the OOB predictions, (On average, each data point would be out-of-bag around 36% of the times, so aggregate these predictions). Calculate the error rate, and call it a OOB estimate of error rate. Random Forests create a large number of independently trained class classifiers by the random sampling of features. 32  Chapter 5 Methodology 5.1 Overview This chapter covers the all the methodology related with this chapter. The first section covers the canny algorithm for the edge detection. This chapter covers the principal component analysis method of the feature extraction and also the Hu-moments in recognition of the signs. 5.2 Heading For sign language recognition, a dataset of hand’s gestures is required. One can use the available data set and can also use self-made dataset. In this project, the dataset of hand gestures used is self-made. As it can be seen from the flow chart of this project, the first step is to collect the data. So first we collect the data from the available dataset. 33 Then the next step of the project is edges detection. Once the data has been collected, then edges detection algorithm can be performed on this data. Canny algorithm has been used in the project for the edges detection. Once edges detected from the dataset, then moved to the next step of project. Once the edges detection algorithm has been performed, next is to extract features from the dataset. Features can be extracted using different algorithms. Two algorithms used in the project for features extraction named as PCA and Hu moments. To obtained features using principal component analysis, features can be extracted either passing the result of canny to principal component analysis or directly pass the dataset to principal component analysis. Both steps are applied in the project but the result of passing dataset directly to principal component analysis as compared to the result of canny passing to principal component analysis is better. The features in the PCA are varied between 5 and 15 and calculate the result. As the features increases up to 15 so there is improvement in the result. The same process can also be done using Hu moments. Hu moments take the dataset and extract the features from it. Both principal component analysis and Hu moments result are used in the project. Once the features have been extracted from the data, then its ARFF file can be made. ARFF file stands for attribute relationship file format. Arff file contains the features of the dataset. After extracting features from the data set its features are stored in arff file. Both the features obtained from PCA and Hu moments are stored in separate Arff file. One Arff file contains the features of PCA obtained through PCA and the other contains the features of Hu moments obtained through Hu moments. Once the Arff file made, and then checks the result by applying different classifiers on both the file separately. Different classifiers are 34  applied on the Arff files. Each classifier gives its own result for PCA and Hu moments. In the last, the result of different classifiers is stored in a table for both principal component analysis and Hu moments. Both the result are compared and shown in a table. Each one has a different result for the same classifier. 5.3 Flow Chart 35  Result Figure 5.1 Flow chart of the Project 36  Chapter 6 Results 6.1 Overview In this chapter we will conclude our final results. What we have achieved and what we can be presented now. In other words this chapter is about the results of our project. 6.2 Classification Using PCA By passing the dataset of hand’s gestures to PCA its results are generated. After passing the gestures to the PCA made the ARFF file. ARFF file stands for “Attribute Relation File Format” and it contains the features of the provided dataset. In figure 6.1 the result of the PCA is shown which shows that PCA has recognized the input hand gesture. This is shown below. Figure 6.1 Recognition of the hand gesture using PCA 37 6.3 Effect on Accuracy by Varying Number of PCA Features By changing the number of features the results of the PCA are generated and analyzed. The results shown in table 6.1 are gathered by varying the number of features. For each result we obtained the result in terms of the average probability of correct classification. By increasing the number of features the average probability increases. It means by increasing the number of features the accuracy of classification increases but up to certain features and onwards the probability repeats. The results are as shown below in table 6.1. Results of PCA by Varying Features Number of PCA Features Accuracy 1 Using 5 features 70.833% 2 Using 6 features 70.411% 3 Using 7 features 70.833% 4 Using 8 features 70.0833% 5 Using 9 features 71.6667% 6 Using 10 features 72.9167% 7 Using 11 features 73.333% 8 Using 12 features 74.166% 9 Using 13 features 74.1667% 1 Using 14 features 74.1667% 0 1 Using 15 features 74.1667% 1 Table 6.1 Effect of number of features on classification 38 6.4 Results of Different Classifiers Classification results of the PCA by different classifiers are shown in table 6.2. We obtained different result from different classifiers. Some classifiers give best results in terms of classification. Different classifiers with their correctly classified result are given in the following table. Results of Different Classifiers Classifier Correctly classified 1 BFTree 67.08% 2 RandomForest 86.66% 3 RandomTree 80.33% 4 SimpleCart 68.75% 5 IBK 87.91% 6 Kstar 56.25% 7 NaiveBayes 65.41% 8 SMO 47.5% 9 MultiClassClassifier 77.08% 1 Bagging 77.07% 0 1 HyperPipes 70.83% 1 1 VFI 74.58% 2 1 Ridor 62.08% 3 1 DTNB 67.91% 4 Table 6.2 Results of PCA using different Classifiers 39 6.5 Classification Using Hu-moments Hu moments features give quite different result from the features of PCA. The average probability of the Hu-Moments is shown in the figure 6.2. All its features give different result as compared with result of PCA. The obtained results of different classifiers are shown in the given table. Classification Using Hu-Moments Classifers Correctly classified 1 BFTree 5.41% 2 RandomForest 81.66% 3 RandomTree 74.58% 4 SimpleCart 66.25% 5 IBK 75.41% 6 Kstar 57.91% 7 NaiveBayes 38.75% 8 SMO 27.08% 9 MultiClassClassifier 56.66% 1 Bagging 67.5% 0 1 HyperPipes 45.83% 1 1 VFI 68.75% 2 1 Ridor 39.58% 3 1 OneR 26.25% 4 40  Table 6.3 Classification Results using Hu-moments Figure 6.2 Average Probability of the Hu-Moments 6.6 Hu-Moments Vs PCA After getting result from both PCA and Hu moments the comparison of both algorithms gives an idea that which algorithm will best classified the features obtained from the dataset means that the comparison gives an idea of which algorithm should be used further. In this project the algorithm used is best in terms of the result. The comparison result of both the algorithms is showed in a table 6.4. Comparison is best for 41 getting idea that which classifier gives best result in which algorithm so that someone used that classifier in that algorithm. The result of comparison is shown in the table 6.4. Comparison between PCA and Hu-Moments Classifier PCA Hu-Moments 1 BFTree 67.08% 5.41% 2 RandomForest 86.66% 81.66% 3 RandomTree 80.33% 74.58% 4 SimpleCart 68.75% 66.25% 5 IBK 87.91% 75.41% 6 Kstar 56.25% 57.91% 7 NaiveBayes 65.41% 38.75% 8 SMO 47.5% 27.08% 9 MultiClassClassifier 77.08% 56.66% 1 Bagging 77.07% 67.5% 0 1 HyperPipes 70.83% 45.83% 1 1 VFI 74.58% 68.75% 2 1 Ridor 62.08% 39.58% 3 1 OneR 20% 26.25% 4 Table 6.4 PCA Vs Hu-Moments 42  The above table 6.6 shows the results of the PCA and Hu-moments. From the table it is clear that majority of classifiers give best result for PCA while some of them give best result in terms of Hu moments. As we can see that Kstar classifier gives best result for Hu moments. 6.7 Conclusion In real world sign language is comparatively more formable for communication which helps to increase the communication. In different areas of the world there are different forms and variation of the sign systems exists. The main problem is that there are less number of principal notions and assigned appearances in the sign language. The basic idea behind this work is to develop and implement a system of sign language which will facilitate the deaf people and will increase the process of communication among the deaf and the normal people of the society. This project basically focuses on implementation of a software model for hand gesture recognition system using classifier. The approach for this is to recognize the hand gestures in the self- made data set of the hand gestures and to analyze the features using classifier. The recognition is done using PCA and also using Hu- Moments features of the gestures are analyzed and classification is done using different classifiers. 43 References: [1] Rafiqul Zaman Khan and Noor Adnan Ibraheem “HAND GESTURE RECOGNITION” International Journal of Artificial Intelligence & Applications (IJAIA), Vol.3, No.4, July 2012. [2] A History of Sign Language Brook Larson & Dr. Hallen Mid-Term Paper 24 February, 2003. [3] Annelies Braffort “Computer Science and Sign Language: Ethical Aspects” Orsay cedex international research center, 2009. [4] VonAgris, Blomer, Kraiss,K.F.:Rapid signer adaptation forcontinuous sign language recognition using a combined approach of eigen voices, MLLR,andMAP.In:Procs.ofICPR, pp. 1 – 4. Tampa, Florida, USA (2008). [5] Klimis Symeonidis “Hand Gesture Recognition Using Neural Networks” School of Electronic and Electrical Engineering on August 23, 2000. [6] Zieren, Kraiss, Robustperson “independent visual sign language recognition. In:Procs. of IbPRIA, pp. 520 – 528. Springer, Estoril, Portugal (2005) 44 [7] Vrushali Y Kulkarni, Dr Pradeep K Sinha “Random Forest Classifiers :A Survey and Future Research Directions” International Journal of Advanced Computing, ISSN:2051-0845, Vol.36, Issue, April 2013 [8] Pradeep K Sinha, Vrushali Y Kulkarni “Efficient Learning of Random Forest Classifier using Disjoint Partitioning Approach” Proceedings of the World Congress on Engineering 2013 Vol II, WCE 2013, July 3 - 5, 2013, London, U.K [9] Grahn H, Lavesson N, Lapajne M, Slat D, A CUDA implementation of Random Forest – Early Results, Master Thesis Software Engineering, School of Computing, Blekinge Institute of Technology, Sweden. [10] Kun Zhang, Wei Fan, Bill Buckles, Xiaojing Yuan, and Zujia Xu: "Discovering Unrevealed Properties of Probability Estimation Trees: On Algorithm Selection and Performance Explanation", 2006 IEEE International Conference on Data Ming (ICDM'2006), December 2006. [11] Raymond Lockton and Andrew W.Fitzgibbon, Real time gesture recognition using deterministic boosting, Department of engineering Science, university of Oxford. [12] Sebastien Marcel, Olivier Bernier, Jean-Emmanuel Viallet and Daniel Collobert,Hand gesture recognition using input-output hidden Markov mode, France Telecom CNET 2 avenue pierre Marzin 22307 Lannion,France. [13] Ray Lockton, Hand gesture recognition using computer vision, Balliol college oxford university [14] Canny edge detection “09gr820” on March 23, 2009 45 [15] Aly A. Farag, Shireen Elhabian, A tutorial on principal component analysis, university of Lousisville,CVIP Lab,2009. [16] Steven M.Holland, Principal Component Analysis,Department of Geology, University of Georgia, Athens,2008. [17] B.Vanajakshi, Dr.k.Sri Rama Krishna,classification of boundary and region shapes using Humoment invariants,2012. [18] Laura Keyes, Adam Winstanley, using moment invariants for classifying shapes on large scale maps, department of computer science, National University of Ireland Maynooth. 46