作　　者： (于洁）;

机构地区： 福建信息职业技术学院计算机工程系

出　　处： 《北京信息科技大学学报（自然科学版）》 2017年第4期64-68,74,共6页

摘　　要： 从互联网海量文本中抽取出词语的定义对知识库的自动构建具有重要意义。针对定义抽取研究中N-gram语言模型携带的语义信息有限、容易产生大量稀疏数据的问题,提出了一种DN-gram语言模型。在N-gram基础上融入上下文词语间的语法依赖关系,能表达更为丰富的语言学特征;采用词形和词性的组合代替只用词形的特征选取方法来降低稀疏数据的影响;引入TF-IDF定义隶属度进行特征降维;在Spark大数据平台下采用并行定义抽取管道将互联网文本转换成特征向量,使用随机森林分类器进行学习和训练,在实验中取得了较好效果。 The definition of the words extracted from the massive text of the Internet is of great significance to the automatic construction of the knowledge base. The semantic information carried by the N-gram language model is limited and easy to generate large amount of sparse data,so a DN-gram language model is proposed,which integrates the grammatical dependencies between context words on Ngram. The feature selection method applies the combination of word form and part of speech to reduce the impact of sparse data,instead of using word form only. TF-IDF is used to calculate the membership of definition for reducing the dimensionality of features. Under the Spark large data platform,the parallel definition extraction pipeline is employed to convert the Internet text into the eigenvector,using the random forest classifier to learn and train. The experiment has achieved good effect.

关 键 词： 知识发现 文本分类 定义抽取