Three experiments were set to evaluate the system. Beginning with the experiment varying the number of states and Gaussian mixtures of HMM is to find the optimal point for the emotional speech recognition system. Later on considering about emotional segmentation was the second experiment. Finally, we applied the binary classification technique with HMM to improve the accuracy of the system. The speech data that was used as training and test set for the following experiments are described in details of each class of emotions in Table 3. The number of speech data in test set is around 30% of speech data. The happiness class is a large number of utterances while the fear is the least number of utterances in this emotional speech data.