SONG Cheng-xiang, CHEN Xiu-hong, NIU Qiang. Improved feature selection method based on CHI for text categorization[J]. 2019, (2): 37-40. DOI: 10.13873/j.1000-9787(2019)02-0037-04.
文本分类中基于CHI改进的特征选择方法
摘要
针对传统卡方统计量(CHI)方法在全局范围内进行特征选择时忽略词频信息问题
提出了一种改进的文本特征选择方法。通过引入特征分布相关性系数
选择局部出现的强相关性特征
并利用修正因子解决CHI方法的负相关困扰
从而提升语料集的分类指标。对网易新闻语料库和复旦大学中文语料库进行实验时
利用以上方法进行特征选择
使用改进后的词频—逆文本频率(TF-IDF)权重计算公式加权
分类器选择支持向量机(SVM)和朴素贝叶斯法。结果表明:改进的方法不仅在分类效果上有明显的提高
而且性能更加稳定。
Abstract
Aiming at problem that traditional CHI method chooses the feature in the global scope and ignores the information of word frequency
an improved feature selection method is proposed. By introducing feature distribution correlation coefficient
localized strong correlated features are selected
and the correction factor is used to solve the negative correlation obsession of the CHI method
thus the classification index of corpus set is improved. When experiments are conducted on Netease news corpus and Fudan University Chinese corpus
feature selection is first performed by using the above method
and then the selected features are weighted by the improved term frequency-inverse document frequency( TF-IDF) weighting formula. The classifier uses support vector machine( SVM) and naive Bayes method. The experimental results show that the improved algorithm not only shows a significant improvement in classification effect