Abstract: According to the theory of traditional Chinese medicine, the facial complexions are divided into four categories named as red, yellow, white and black, and deep learning method is used to realize the key points recognition and automatic segmentation of interested region. This study innovatively combines elements such as color space features, facial texture statistical features, and lip color features, and uses a variety of machine learning methods to classify and recognize the extracted facial features. In order to verify the effectiveness of the proposed method, 575 facial images are collected by professional instruments to form a database, and the face color is calibrated under the guidance of experts of traditional Chinese medicine. The result showed that the best recognition rate of the fusion of facial skin color features, texture features and lip color features reached 91.03%, Color feature is one of the most important features of classification and recognition.