Research Paper:
A Continual Self-Supervised Learning Framework Using a Decoder and a Clustering-Based Data Selection Method
Ryosuke Fujii*
, Naoki Masuyama*,
, Yusuke Nojima*
, Chu Kiong Loo**
, and Liew Wei Shiung**

*Osaka Metropolitan University
1-1 Gakuen-cho, Sakai, Osaka 599-8531, Japan
Corresponding author
**University of Malaya
Kuala Lumpur 50603, Malaysia
Continual learning aims to learn a sequence of tasks without forgetting the previously learned tasks. Promising methods for achieving continual learning include applying a replay buffer that stores previously learned samples in existing learning frameworks. Among these methods, lifelong unsupervised mixup (LUMP) interpolates the current task samples and samples from the replay buffer to prevent forgetting. LUMP randomly selects data for the replay buffer. This random selection is likely to have a negative effect on learning performance. In this study, we propose a clustering-based data selection method for the replay buffer. In addition, we introduce a denoising autoencoder to the self-supervised methods to improve the performance of the learning representations. We compare the clustering algorithms used in our framework and demonstrate the effectiveness of the proposed framework through computational experiments. Experimental results on CIFAR-100 and Tiny-ImageNet demonstrate that the proposed framework improves the average accuracy compared with LUMP while reducing forgetting. In particular, the CA+-based selection approach enhances the learning performance. The source code is available at https://github.com/Masuyama-lab/LUMP_SidAE_Clustering
Proposed continual SSL framework
- [1] Q. Sang, Z. Shu, L. Liu, C. Hu, and Q. Wu, “Image quality assessment based on self-supervised learning and knowledge distillation,” J. of Visual Communication and Image Representation, Vol.90, Article No.103708, 2023. https://doi.org/10.1016/j.jvcir.2022.103708
- [2] V. Rani, S. T. Nabi, M. Kumar, A. Mittal, and K. Kumar, “Self-supervised learning: A succinct review,” Archives of Computational Methods in Engineering, Vol.30, No.4, pp. 2761-2775, 2023. https://doi.org/10.1007/s11831-023-09884-2
- [3] D. Madaan, J. Yoon, Y. Li, Y. Liu, and S. J. Hwang, “Representational continuity for unsupervised continual learning,” Proc. of Int. Conf. on Learning Representations, 2022.
- [4] F. Baier, S. Mair, and S. G. Fadel, “Self-supervised siamese autoencoders,” I. Miliou, N. Piatkowski, and P. Papapetrou (Eds.), “Lecture Notes in Computer Science,” pp. 117-128, 2024. https://doi.org/10.1007/978-3-031-58547-0_10
- [5] J. Gui, T. Chen, J. Zhang, Q. Cao, Z. Sun, H. Luo, and D. Tao, “A survey on self-supervised learning: Algorithms, applications, and future trends,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.46, No.12, pp. 9052-9071, 2024. https://doi.org/10.1109/tpami.2024.3415112
- [6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” Proc. of the 37th Int. Conf. on Machine Learning, Vol.119, pp. 1597-1607, 2020.
- [7] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020. https://doi.org/10.1109/cvpr42600.2020.00975
- [8] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Proc. of Advances in Neural Information Processing Systems, pp. 9912-9924, 2020.
- [9] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” Proc. of Advances in Neural Information Processing Systems, pp. 21271-21284, 2020.
- [10] X. Chen and K. He, “Exploring simple siamese representation learning,” 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021. https://doi.org/10.1109/cvpr46437.2021.01549
- [11] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” Proc. of Int. Conf. on Machine Learning, pp. 12310-12320, 2021.
- [12] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, and R. Girshick, “Masked autoencoders are scalable vision learners,” 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 15979-15988, 2022. https://doi.org/10.1109/cvpr52688.2022.01553
- [13] Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, “SimMIM: a simple framework for masked image modeling,” 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 9643-9653, 2022. https://doi.org/10.1109/cvpr52688.2022.00943
- [14] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” Proc. of the National Academy of Sciences, Vol.114, No.13, pp. 3521-3526, 2017. https://doi.org/10.1073/pnas.1611835114
- [15] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” Proc. of the 34th Int. Conf. on Machine Learning, pp. 3987-3995, 2017.
- [16] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive neural networks,” arXiv:1606.04671, 2016. https://doi.org/10.48550/arXiv.1606.04671
- [17] A. Mallya and S. Lazebnik, “PackNET: Adding multiple tasks to a single network by iterative pruning,” 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7765-7773, 2018. https://doi.org/10.1109/cvpr.2018.00810
- [18] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: A strong, simple baseline,” Proc. of Advances in Neural Information Processing Systems, pp. 15920-15930, 2020.
- [19] D. Lopez-Paz and M. A. Ranzato, “Gradient episodic memory for continual learning,” Proc. of Advances in Neural Information Processing Systems, 2017.
- [20] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” Proc. of Int. Conf. on Learning Representations, 2018.
- [21] Z. Shen, Z. Liu, Z. Liu, M. Savvides, T. Darrell, and E. Xing, “Un-mix: Rethinking image mixtures for unsupervised visual representation learning,” Proc. of the AAAI Conf. on Artificial Intelligence, Vol.36, No.2, pp. 2216-2224, 2022. https://doi.org/10.1609/aaai.v36i2.20119
- [22] D. Reynolds, “Gaussian mixture models,” S. Z. Li and A. K. Jain (Eds.), “Encyclopedia of Biometrics,” Springer, pp. 827-832, 2015. https://doi.org/10.1007/978-1-4899-7488-4_196
- [23] S. Lloyd, “Least squares quantization in PCM,” IEEE Trans. on Information Theory, Vol.28, No.2, pp. 129-137, 1982. https://doi.org/10.1109/tit.1982.1056489
- [24] B. Fritzke, “A growing neural gas network learns topologies,” Proc. of Advances in Neural Information Processing Systems, 1994.
- [25] G. Carpenter and S. Grossberg, “The art of adaptive pattern recognition by a self-organizing neural network,” Computer, Vol.21, No.3, pp. 77-88, 1988. https://doi.org/10.1109/2.33
- [26] F. Shen and O. Hasegawa, “A fast nearest neighbor classifier based on self-organizing incremental neural network,” Neural Networks, Vol.21, No.10, pp. 1537-1547, 2008. https://doi.org/10.1016/j.neunet.2008.07.001
- [27] C. Wiwatcharakoses and D. Berrar, “SOINN+, a self-organizing incremental neural network for unsupervised learning from noisy data streams,” Expert Systems with Applications, Vol.143, Article No.113069, 2020. https://doi.org/10.1016/j.eswa.2019.113069
- [28] S. Marsland, J. Shapiro, and U. Nehmzow, “A self-organising network that grows when required,” Neural Networks, Vol.15, No. 8-9, pp. 1041-1058, 2002. https://doi.org/10.1016/s0893-6080(02)00078-3
- [29] G. A. Carpenter, S. Grossberg, and D. B. Rosen, “Fuzzy ART: Fast stable learning and categorization of analog patterns by an adaptive resonance system,” Neural Networks, Vol.4, No.6, pp. 759-771, 1991. https://doi.org/10.1016/0893-6080(91)90056-b
- [30] B. Vigdor and B. Lerner, “The Bayesian ARTMAP,” IEEE Trans. on Neural Networks, Vol.18, No.6, pp. 1628-1644, 2007. https://doi.org/10.1109/tnn.2007.900234
- [31] L. Wang, H. Zhu, J. Meng, and W. He, “Incremental local distribution-based clustering using Bayesian adaptive resonance theory,” IEEE Trans. on Neural Networks and Learning Systems, Vol.30, No.11, pp. 3496-3504, 2019. https://doi.org/10.1109/tnnls.2019.2919723
- [32] L. E. B. da Silva, I. Elnabarawy, and D. C. Wunsch, “Distributed dual vigilance fuzzy adaptive resonance theory learns online, retrieves arbitrarily-shaped clusters, and mitigates order dependence,” Neural Networks, Vol.121, pp. 208-228, 2020. https://doi.org/10.1016/j.neunet.2019.08.033
- [33] W. Liu, P. P. Pokharel, and J. C. Principe, “Correntropy: Properties and applications in non-gaussian signal processing,” IEEE Trans. on Signal Processing, Vol.55, No.11, pp. 5286-5298, 2007. https://doi.org/10.1109/tsp.2007.896065
- [34] N. Masuyama, C. K. Loo, and F. Dawood, “Kernel Bayesian ART and ARTMAP,” Neural Networks, Vol.98, pp. 76-86, 2018. https://doi.org/10.1016/j.neunet.2017.11.003
- [35] N. Masuyama, C. K. Loo, and S. Wermter, “A Kernel Bayesian adaptive resonance theory with a topological structure,” Int. J. of Neural Systems, Vol.29, No.5, Article No.1850052, 2019, pMID: 30764724. https://doi.org/10.1142/s0129065718500521
- [36] N. Masuyama, C. K. Loo, H. Ishibuchi, N. Kubota, Y. Nojima, and Y. Liu, “Topological clustering via adaptive resonance theory with information theoretic learning,” IEEE Access, Vol.7, pp. 76920-76936, 2019. https://doi.org/10.1109/access.2019.2921832
- [37] N. Masuyama, N. Amako, Y. Nojima, Y. Liu, C. K. Loo, and H. Ishibuchi, “Fast topological adaptive resonance theory based on correntropy induced metric,” 2019 IEEE Symp. Series on Computational Intelligence (SSCI), pp. 2215-2221, 2019. https://doi.org/10.1109/ssci44817.2019.9003098
- [38] L. E. B. da Silva, I. Elnabarawy, and D. C. Wunsch, “Dual vigilance fuzzy adaptive resonance theory,” Neural Networks, Vol.109, pp. 1-5, 2019. https://doi.org/10.1016/j.neunet.2018.09.015
- [39] L. Meng, A.-H. Tan, and D. C. Wunsch, “Adaptive scaling of cluster boundaries for large-scale social media data clustering,” IEEE Trans. on Neural Networks and Learning Systems, Vol.27, No.12, pp. 2656-2669, 2016. https://doi.org/10.1109/tnnls.2015.2498625
- [40] N. Masuyama, Y. Nojima, Y. Toda, C. K. Loo, H. Ishibuchi, and N. Kubota, “Privacy-preserving continual federated clustering via adaptive resonance theory,” IEEE Access, Vol.12, pp. 139692-139710, 2024. https://doi.org/10.1109/access.2024.3467114
- [41] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. https://doi.org/10.1109/cvpr.2016.90
- [42] A. Krizhevsky, “Learning Multiple Layers of Features From Tiny Images,” University of Toronto, 2012.
- [43] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A large-scale hierarchical image database,” 2009 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pp. 248-255, 2009. https://doi.org/10.1109/cvpr.2009.5206848
- [44] L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol.46, No.8, pp. 5362-5383, 2024. https://doi.org/10.1109/tpami.2024.3367329
- [45] L. Kaufman and P. J. Rousseeuw, “Partitioning around medoids (program PAM),” L. Kaufman and P. J. Rousseeuw, “Finding Groups in Data: An Introduction to Cluster Analysis,” John Wiley & Sons, Inc., pp. 68-125, 1990.
- [46] A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: Analysis and an algorithm,” Proc. of the 15th Int. Conf. on Neural Information Processing Systems: Natural and Synthetic, pp. 849-856, 2001.
- [47] T. Kohonen, “The self-organizing map,” Proc. of the IEEE, Vol.78, No.9, pp. 1464-1480, 1990. https://doi.org/10.1109/5.58325
- [48] Y. Zhang, L. Charlin, R. Zemel, and M. Ren, “Integrating present and past in unsupervised continual learning,” arXiv:2404.19132, 2024. https://doi.org/10.48550/arXiv.2404.19132
This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.