JACIII Vol.28 No.2 pp. 231-238
doi: 10.20965/jaciii.2024.p0231

Research Paper:

Deep Convolutional Neural Networks Based on Knowledge Distillation for Offline Handwritten Chinese Character Recognition

Hongli He* ORCID Icon, Zongnan Zhu** ORCID Icon, Zhuo Li*** ORCID Icon, and Yongping Dan**,† ORCID Icon

*Rail Transit Institute, Henan College of Transportation
No.259 Tonghui Road, Baisha Vocational Education Park, Zhengdong New District, Zhengzhou 450061, China

**School of Electronic and Information, Zhongyuan University of Technology
No.41 Zhongyuan Road, Zhengzhou 450007, China

Corresponding author

***Graduate School of Science and Engineering, Ritsumeikan University
1-1-1 Noji-higashi, Kusatsu, Shiga 525-8577, Japan

December 8, 2022
July 6, 2023
March 20, 2024
deep convolutional neural networks, handwritten Chinese character recognition, attention mechanism, knowledge distillation

Deep convolutional neural networks (DNNs) have achieved outstanding performance in this field. Meanwhile, handwritten Chinese character recognition (HCCR) is a challenging area of research in the field of computer vision. DNNs require a large number of parameters and high memory consumption. To address these issues, this paper proposes an approach based on an attention mechanism and knowledge distillation. The attention mechanism improves the feature extraction and the knowledge distillation reduces the number of parameters. The experimental results show that ResNet18 achieves a recognition accuracy of 97.63% on the HCCR dataset with 11.25 million parameters. Compared with other methods, this study improves the performance for HCCR.

Cite this article as:
H. He, Z. Zhu, Z. Li, and Y. Dan, “Deep Convolutional Neural Networks Based on Knowledge Distillation for Offline Handwritten Chinese Character Recognition,” J. Adv. Comput. Intell. Intell. Inform., Vol.28 No.2, pp. 231-238, 2024.
Data files:
  1. [1] C.-L. Liu, “High Accuracy Handwritten Chinese Character Recognition Using Quadratic Classifiers with Discriminative Feature Extraction,” 18th Conf. on Pattern Recognition (ICPR’06), Vol.2, pp. 942-945, 2006.
  2. [2] T. H. Hildebrandt and W. Liu, “Optical Recognition of Handwritten Chinese Characters: Advances Since 1980,” Pattern Recognition, Vol.26, pp. 205-225, 1993.
  3. [3] X. Zhang, Y. Bengio, and C. Liu, “Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark,” Pattern Recognition, Vol.61, pp. 348-360, 2016.
  4. [4] R. Dai, C. Liu, and B. Xiao, “Chinese Character Recognition: History, Status and Prospects,” Frontiers of Computer Science in China, Vol.1, No.2, pp. 126-136, 2007.
  5. [5] C.-L. Liu, F. Yin, D.-H. Wang, and Q.-F. Wang, “Online and Offline Handwritten Chinese Character Recognition: Benchmarking on New Databases,” Pattern Recognition, Vol.46, No.1, pp. 155-162, 2013.
  6. [6] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet Classification with Deep Convolutional Neural Networks,” Communications of the ACM, Vol.60, No.6, pp. 84-90, 2017.
  7. [7] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv:1409.1556, 2015.
  8. [8] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper with Convolutions,” 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015.
  9. [9] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
  10. [10] H. Li, Z. Wang, X. Yue, W. Wang, T. Hiroyuki, and L. Meng, “A Comprehensive Analysis of Low-Impact Computations in Deep Learning Workloads,” Proc. of the 2021 on Great Lakes Symp. on VLSI (GLSVLSI’21), pp. 385-390, 2021.
  11. [11] W. Park, D. Kim, Y. Lu, and M. Cho, “Relational Knowledge Distillation,” Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
  12. [12] R. Z. Tan, X. Chew, and K. W. Khaw, “Neural Architecture Search for Lightweight Neural Network in Food Recognition,” Mathematics, Vol.9, No.11, Article No.1245, 2021.
  13. [13] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv:1503.02531, 2015.
  14. [14] Z. Huang and N. Wang, “Like What You Like: Knowledge Distill via Neuron Selectivity Transfer,” arxiv:1707.01219, 2017.
  15. [15] S. Zagoruyko and N. Komodakis, “Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer,” Int. Conf. on Learning Representations, pp. 24-26, 2017.
  16. [16] J. Yim, D. Joo, J. Bae, and J. Kim, “A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning,” Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 4133-4141, 2017.
  17. [17] H. Bagherinezhad, M. Horton, M. Rastegari, and A. Farhadi, “Label Refinery: Improving ImageNet Classification Through Label Progression,” arXiv:1805.02641, 2018.
  18. [18] S. Lai, L. Jin, and W. Yang, “Toward high-performance online HCCR: A CNN approach with DropDistortion, path signature and spatial stochastic max-pooling,” Pattern Recognition Letters, Vol.89, pp. 60-66, 2017.
  19. [19] R. Sharma and B. Kaushik, “Offline Recognition of Handwritten Indic Scripts: A State-of-the-Art Survey and Future Perspectives,” Computer Science Review, Vol.38, Article No.100302, 2020.
  20. [20] Q. Hao, X. Wu, S. Zhang, P. Zhang, X. Ma, and J. Jiang, “Research on Offline Handwritten Chinese Character Recognition Based on Deep Learning,” 2019 9th Int. Conf. on Formation Science and Technology (ICIST), pp. 470-474, 2019.
  21. [21] W. Luo and S. I. Kamata, “Radical Region-Based CNN for Offline Handwritten Chinese Character Recognition,” Proc. of the 2017 4th IAPR Asian Conf. on Pattern Recognition (ACPR), pp. 542-547, 2017.
  22. [22] Y. Ge, Q. Huo, and Z.-D. Feng, “Offline Recognition of Handwritten Chinese Characters Using Gabor Features, CDHMM Modeling and MCE Training,” Proc. of the Conf. on Acoustics, Speech and Signal Processing (CASSP’02), pp. I-1053-I-1056, 2002.
  23. [23] C.-L. Liu, “Normalization-Cooperated Gradient Feature Extraction for Handwritten Character Recognition,” IEEE Trans. on Pattern Analysis and Machine Int., Vol.29, No.8, pp. 1465-1469, 2007.
  24. [24] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, Vol.521, Article No.7553, pp. 436-444, 2015.
  25. [25] Z. Zhong, X.-Y. Zhang, F. Yin, and C.-L. Liu, “Handwritten Chinese Character Recognition with Spatial Transformer and Deep Residual Networks,” 2016 23rd Int. Conf. on Pattern Recognition (ICPR), pp. 3440-3445, 2016.
  26. [26] N. Bi, J. Chen, and J. Tan, “The Handwritten Chinese Character Recognition Uses Convolutional Neural Networks with the GoogLeNet,” J. of Pattern Recognition and Artificial Intelligence, Vol.33, No.11, Article No.1940016, 2019.
  27. [27] P. Melnyk, Z. You, and K. Li, “A High-Performance CNN Method for Offline Handwritten Chinese Character Recognition and Visualization,” Soft Computing, Vol.24, No.11, pp. 7977-7987, 2020.
  28. [28] Z. Li, Q. Wu, Y. Xiao, M. Jin, and H. Lu, “Deep Matching Network for Handwritten Chinese Character Recognition,” Pattern Recognition, Vol.107, Article No.107471, 2020.
  29. [29] Y. Dan and Z. Li, “Particle Swarm Optimization-Based Convolutional Neural Network for Handwritten Chinese Character Recognition,” J. Adv. Comput. Intell. Intell. Inform., Vol.27, No.2, pp. 165-172, 2023.
  30. [30] M. H. Sadi and A. Mahani, “Accelerating Deep Convolutional Neural Network Base on Stochastic Computing,” Integration, Vol.76, pp. 113-121, 2021.
  31. [31] N. Passalis and A. Tefas, “Training Lightweight Deep Convolutional Neural Networks Using Bag-of-Features Pooling,” IEEE Trans. on Neural Networks and Learning Systems, Vol.30, No.6, pp. 1705-1715, 2019.
  32. [32] C. Buciluǎ, R. Caruana, and A. Niculescu-Mizil, “Model Compression,” Proc. of the 12th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD’06), pp. 535-541, 2006.
  33. [33] J. Ba and R. Caruana, “Do Deep Nets Really Need to Be Deep?,” Advances in Neural Information Processing Systems, Vol.3, pp. 2654-2662, 2014.
  34. [34] Z. Xu, Y.-C. Hsu, and J. Huang, “Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks,” arXiv:1709.00513, 2017.
  35. [35] S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient Knowledge Distillation for BERT Model Compression,” arXiv:1908.09355, 2019.
  36. [36] G. Ros, S. Stent, P. F. Alcantarilla, and T. Watanabe, “Training Constrained Deconvolutional Networks for Road Scene Semantic Segmentation,” arXiv:1604.01545, 2016.
  37. [37] P. Luo, Z. Zhu, Z. Liu, X. Wang, and X. Tang, “Face Model Compression by Distilling Knowledge from Neurons,” Proc. of the AAAI Conf. on Artificial Intelligence, Vol.30, No.1, 2016.
  38. [38] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learning Efficient Object Detection Models with Knowledge Distillation,” Proc. of the 31st Int. Conf. on Neural Information Processing Systems, pp. 742-751, 2017.
  39. [39] J. Shen, N. Vesdapunt, V. N. Boddeti, and K. M. Kitani, “In Teacher We Trust: Learning Compressed Models for Pedestrian Detection,” arxiv:1612.00478, 2016.
  40. [40] Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu, “Distral: Robust Multitask Reinforcement Learning,” Proc. of the 31st Int. Conf. on Neural Information Processing Systems, pp. 4499-4509, 2017.

*This site is desgined based on HTML5 and CSS3 for modern browsers, e.g. Chrome, Firefox, Safari, Edge, Opera.

Last updated on Apr. 05, 2024