Research on error correction method of Tibetan text based on deep learning

Huaguo Cairang; Secha Jia; Cairang Jia

doi:10.1117/12.2622716

1 December 2021 Research on error correction method of Tibetan text based on deep learning

Huaguo Cairang, Secha Jia, Cairang Jia

Proceedings Volume 12079, Second IYSF Academic Symposium on Artificial Intelligence and Computer Engineering; 120791F (2021) https://doi.org/10.1117/12.2622716
Event: 2nd IYSF Academic Symposium on Artificial Intelligence and Computer Engineering, 2021, Xi'an, China

Abstract

In order to avoid too many Tibetan text errors from affecting the text quality, a method of Tibetan text error correction based on soft masked Bert is proposed. In this work, the definition and classification of Tibetan text error correction are pointed out, the embedding composed of word embedding, position embedding and paragraph embedding is used as the input of Tibetan text error detection model. The probability labels of characters in the text correspond to the output of the error detection model. The output of the model detects characters with high error probability.The final soft masked embedding result is obtained by the output of the weighted sum error detection model and the input embedding. The best model constructed by the encoder layer sets all soft masked embedding sequences as input, and the residual connection values are obtained by connecting the input embedding results and all hidden states of the encoder layer. The obtained residual connection values are input to the full connection layer, and the softmax function is used to obtain the probability that the output characters can be accurately corrected as candidate characters to complete Tibetan text error correction. Experimental results show that this method can achieve efficient Tibetan text error correction, and use the error probability of different candidate words to obtain the best error correction results.

Citation Download Citation

Huaguo Cairang, Secha Jia, and Cairang Jia "Research on error correction method of Tibetan text based on deep learning", Proc. SPIE 12079, Second IYSF Academic Symposium on Artificial Intelligence and Computer Engineering, 120791F (1 December 2021); https://doi.org/10.1117/12.2622716

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
11 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Error analysis

Data modeling

Computer programming

Data processing

Data corrections

Legal

Performance modeling

Show All Keywords

Keywords/Phrases

Search In:

Publication Years