Dual-carbon data fusion cleaning model based on Kafka cluster

Zhichang Li; Changyou Jiang; Qi Zhang; Dashan Du; Ran Zhang; Shaozheng Chai

doi:10.1117/12.3024786

3 April 2024 Dual-carbon data fusion cleaning model based on Kafka cluster

Zhichang Li, Changyou Jiang, Qi Zhang, Dashan Du, Ran Zhang, Shaozheng Chai

Proceedings Volume 13078, Second International Conference on Informatics, Networking, and Computing (ICINC 2023); 130780U (2024) https://doi.org/10.1117/12.3024786
Event: Second International Conference on Informatics, Networking, and Computing (ICINC 2023), 2023, Wuhan, China

Abstract

The conventional cleaning model of dual-carbon data fusion is more inclined to data cleaning, and there are mistakes in attribute judgment during data fusion, which leads to the problems of missing and duplication of the output cleaning data, which affects the application of dual-carbon data. Therefore, this paper studies the dual-carbon data fusion cleaning model based on Kafka cluster. Select the comprehensive attribute weights of the dual-carbon data fusion cleaning model, fuse the data with similar characteristics according to the characteristics of the data attributes, and find out the missing values in the process of repeated cleaning, so as to avoid the data fusion deviation. Based on Kafka cluster, the regularization term of dual-carbon data fusion cleaning target is constructed to reduce the dimension of dual-carbon data, and a regularization term is constructed on the basis of data fusion cleaning target function to make the fused dual-carbon data more meaningful. The comparison experiment shows that the cleaning performance of the model is better and it can be applied to real life.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Zhichang Li, Changyou Jiang, Qi Zhang, Dashan Du, Ran Zhang, and Shaozheng Chai "Dual-carbon data fusion cleaning model based on Kafka cluster", Proc. SPIE 13078, Second International Conference on Informatics, Networking, and Computing (ICINC 2023), 130780U (3 April 2024); https://doi.org/10.1117/12.3024786

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available