Detection of approximately duplicated records based on entropy feature selection grouping clustering

ZHANG Ping1; DANG Xuan-ju1; CHEN Hao2; YANG Wen-lei2

doi:10.13873/j.1000-97872011.11.030

Chinese

您当前的位置：

首页 >

文章列表页 >

Detection of approximately duplicated records based on entropy feature selection grouping clustering

更新时间：2025-09-11

- Detection of approximately duplicated records based on entropy feature selection grouping clustering
- Vol. 30, Issue 11, Pages: 135-137+141(2011)
- 作者机构：
  
  1. 桂林电子科技大学电子工程与自动化学院
  2. 桂林电子科技大学计算机科学与工程学院
- 作者简介：
- 基金信息：
- DOI：10.13873/j.1000-97872011.11.030
  CLC： TP311.13
- Published：2011
- 稿件说明：
移动端阅览
[1]张平,党选举,陈皓,杨文雷.基于熵特征优选分组聚类的相似重复记录检测[J].传感器与微系统,2011,30(11):135-137+141.

ZHANG Ping1, DANG Xuan-ju1, CHEN Hao2, et al. Detection of approximately duplicated records based on entropy feature selection grouping clustering[J]. 2011, 30(11): 135-137+141.
[1]张平,党选举,陈皓,杨文雷.基于熵特征优选分组聚类的相似重复记录检测[J].传感器与微系统,2011,30(11):135-137+141. DOI： 10.13873/j.1000-97872011.11.030.

ZHANG Ping1, DANG Xuan-ju1, CHEN Hao2, et al. Detection of approximately duplicated records based on entropy feature selection grouping clustering[J]. 2011, 30(11): 135-137+141. DOI： 10.13873/j.1000-97872011.11.030.

摘要

针对目前相似重复记录检测方法不能有效处理大数据量的问题

提出一种基于熵的特征优选分组聚类的算法。该方法通过构造一个基于对象间相似度的熵度量

对原始数据集中各属性进行重要性评估

筛选出关键属性集

并依据关键属性将数据划分为不相交的小数据集

在各小数据集中用DBSCAN聚类算法进行相似重复记录的检测。理论分析和实验结果表明:该方法识别精度和检测效率较高。

Abstract

At present

the approximately duplicate records of massive data can not be detected effectively by current methods

an algorithm based on entropy feature selection grouping clustering（FSGC） is proposed.The basic idea is that through constructing an entropy metric based on similarity between objects

the importance of each property can be evaluated and a key property subset can be obtained.According to the key property to split the data sets into small data sets

the approximately duplicated records are identified based on the algorithm of density-based spatial of applications with noise（DBSCAN）.The theory analysis and experimental results show that identification precision and detection efficiency of the method are high and it can effectively solve the problems of identification in approximately duplicate records of the massive data set.

关键词

Keywords

references

大数据量的高效重复记录检测方法 [J]. 庞雄文,姚占林,李拥军. 华中科技大学学报(自然科学版) . 2010(02)

数据仓库中的相似重复记录检测方法 [J]. 李星毅,包从剑,施化吉. 电子科技大学学报 . 2007(06)

一种大数据量的相似记录检测方法 [J]. 韩京宇,徐立臻,董逸生. 计算机研究与发展 . 2005(12)

基于聚类分析技术的数据清洗研究 [J]. 刘芳,何飞. 计算机工程与科学 . 2005(06)

高维聚类中的一种特征筛选方法 [J]. 淦文燕,李家福,李德毅. 解放军理工大学学报(自然科学版) . 2003(06)

一种基于密度的快速聚类算法 [J]. 周水庚,周傲英,曹晶,胡运发. 计算机研究与发展 . 2000(11)

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

An improved method based on LHMM for gathering abnormal event realtime detection

Game fusion modeling based on Bayes network

Method of fusion for multi-sensor data based on entropy

Related Author

DENG De-ying 1

TANG Xiao-hui 2

CUI Hai-xia1

WEI Gang1

WAN Shu-ping1

PAN Peng2

Related Institution

School of Information and Communication,Guilin University of Electronic Technology

Department of Electronic Engineering,Guilin College of Aerospace Technology

School of Electrical and Information Engineering,South China University ofTechnology

College of Physics & Electronic Engineering,Guangzhou University

China Netcom

⁰