Data deduplication using machine learning: a clustering-based approach for improving the data quality

Abstract

With the rise of big data, data duplication has become a common problem in many industries. Data duplication wastes storage space and increases the risk of data inconsistency and errors. In this thesis, we propose a machine learning-based approach to identify and delete duplicate data in large datasets. Our approach first involves the pre-processing of the data to identify possible duplicates based on similarity measures such as edit distance, Jaccard similarity, and cosine similarity. We then train a machine learning model using a supervised learning approach to classify the identified duplicates and non-duplicates. The model is trained on a large labeled dataset and evaluated on an independent test dataset. Our experiments on a large real-world dataset demonstrate that our approach is effective in identifying and deleting data duplicates. Furthermore, our approach can handle large-scale datasets efficiently, making it practical for real-world applications. Overall, our approach provides a practical solution for eliminating data duplication in large datasets. The proposed approach can be used in various industries where data duplication is a common problem, such as healthcare, finance, and e-commerce, to improve data quality and reduce storage costs.