Distributions are ubiquitous across scientific disciplines, extending well beyond probability and statistics. In machine learning, finite probability distributions arise naturally as the softmax output layers of convolutional neural networks and large language models, where they encode class probabilities in image classification and token probabilities in language generation. When such distributions are high dimensional, however, storage and computational costs become significant. In previous work, we introduced two families of generalized entropies derived from f-divergences, using majorization as a reference framework for comparing distributional homogeneity. In this paper, we extend that framework in several directions. First, we study majorization relationships between subcompositions of a distribution. Second, we introduce generalized negentropies derived from f-divergences and analyze their role alongside entropies in dimensionality reduction. Third, we embed both entropies and negentropies into the setting of Shannon’s information channel and show that the classical channel identities are satisfied exclusively by Shannon entropy. These results provide a unified information-theoretic framework for dimensionality reduction of finite distributions, clarifying the structural role of f-divergence-based entropies and negentropies and their relation to classical information measures.

1. [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Proc. 26th Int. Conf. Neural Inf. Process. Syst. (NIPS), Vol.1, pp. 1097-1105, 2012.
2. [2] T. B. Brown et al., “Language models are few-shot learners,” arXiv:2005.14165, 2020. https://doi.org/10.48550/arXiv.2005.14165
3. [3] F. A. Cowell, “Measuring Inequality (3rd Edition),” Oxford University Press, 2011.
4. [4] L. Martino, V. Elvira, and F. Louzada, “Effective sample size for importance sampling based on discrepancy measures,” Signal Process., Vol.131, pp. 386-401, 2017. https://doi.org/10.1016/j.sigpro.2016.08.025
5. [5] T. Leinster and C. A. Cobbold, “Measuring diversity: The importance of species similarity,” Ecology, Vol.93, No.3, pp. 477-489, 2012. https://doi.org/10.1890/10-2402.1
6. [6] A. Rényi, “On measures of entropy and information,” Proc. 4th Berkeley Symp. Math. Stat. Probab., Vol.1, pp. 547-561, 1961.
7. [7] C. Tsallis, “Possible generalization of Boltzmann-Gibbs statistics,” J. Stat. Phys., Vol.52, No.1, pp. 479-487, 1988. https://doi.org/10.1007/BF01016429
8. [8] T. M. Cover and J. A. Thomas, “Elements of Information Theory,” John Wiley & Sons, 2006.
9. [9] M. Sbert, M. Chen, J. Poch, M. Feixas, and S. Chen, “Entropies from f-divergences,” Proc. 21st Int. Conf. Model. Decis. Artif. Intell. (MDAI 2024), pp. 47-60, 2024. https://doi.org/10.1007/978-3-031-68208-7_5
10. [10] G. Hardy, J. Littlewood, and G. Pólya, “Inequalities,” 2nd Edition, Cambridge University Press, 1952.
11. [11] A. W. Marshall, I. Olkin, and B. C. Arnold, “Inequalities: Theory of Majorization and Its Applications,” 2nd Edition, Springer, 2011. https://doi.org/10.1007/978-0-387-68276-1
12. [12] B. C. Arnold, “Majorization and the Lorenz Order: A Brief Introduction,” Springer, 1987. https://doi.org/10.1007/978-1-4615-7379-1
13. [13] P. Castelló, M. Sbert, M. Chover, and M. Feixas, “Viewpoint-based simplification using -divergences,” Inf. Sci., Vol.178, No.11, pp. 2375-2388, 2008. https://doi.org/10.1016/j.ins.2008.01.011
14. [14] J. Rigau, M. Feixas, and M. Sbert, “Refinement criteria based on -divergences,” Proc. 14th Eurogr. Workshop Render., pp. 260-269, 2003.
15. [15] M. Feixas and M. Sbert, “The role of the information channel in visual computing,” M. Chen, J. M. Dunn, A. Golan, and A. Ullah (Eds.), “Advances in Info-Metrics: Information and Information Processing Across Disciplines,” pp. 464-490, Oxford University Press, 2020. https://doi.org/10.1093/oso/9780190636685.003.0017
16. [16] M. Sbert, M. Feixas, J. Rigau, M. Chover, and I. Viola, “Information Theory Tools for Computer Graphics,” Springer, 2023. https://doi.org/10.1007/978-3-031-79546-6
17. [17] M. Sbert, M. Chen, J. Poch, M. Feixas, S. Chen, and V. Elvira, “Dimensionality reduction with entropies from f-divergences,” Proc. 22nd Int. Conf. Model. Decis. Artif. Intell. (MDAI), pp. 27-41, 2026. https://doi.org/10.1007/978-3-032-00891-6_3
18. [18] I. Csiszár and P. C. Shields, “Information theory and statistics: A tutorial,” Found. Trends Commun. Inf. Theory, Vol.1, No.4, pp. 417-528, 2004. https://doi.org/10.1561/0100000004
19. [19] I. Sason, “On data-processing and majorization inequalities for -divergences with applications,” Entropy, Vol.21, No.10, Article No.1022, 2019. https://doi.org/10.3390/e21101022
20. [20] Y. Polyanskiy and Y. Wu, “Information Theory: From Coding to Learning,” Cambridge University Press, 2025. https://doi.org/10.1017/9781108966351
21. [21] A. Cichocki and S. Amari, “Families of Alpha- Beta- and Gamma-divergences: Flexible and robust measures of similarities,” Entropy, Vol.12, No.6, pp. 1532-1568, 2010. https://doi.org/10.3390/e12061532
22. [22] F. Topsøe, “Entropy and equilibrium via games of complexity,” Phys. A Stat. Mech. Appl., Vol.340, Nos.1-3, pp. 11-31, 2004. https://doi.org/10.1016/j.physa.2004.03.073
23. [23] O. Rioul, “The interplay between error, total variation, alpha-entropy and guessing: Fano and Pinsker direct and reverse inequalities,” Entropy, Vol.25, No.7, Article No.978, 2023. https://doi.org/10.3390/e25070978
24. [24] J.-J. Egozcue and V. Pawlowsky-Glahn, “Evidence functions: A compositional approach to information,” SORT-Stat. Oper. Res. Trans., Vol.42, No.2, pp. 101-124, 2018.
25. [25] J. Aczél and Z. Daróczy, “On Measures of Information and Their Characterizations,” Academic Press, 1975.
26. [26] F. Cowell, “Measuring Inequality,” Oxford University Press, 2011. https://doi.org/10.1093/acprof:osobl/9780199594030.001.0001
27. [27] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., Vol.27, No.3, pp. 379-423 and Vol.27, No.4, pp. 623-656, 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x, https://doi.org/10.1002/j.1538-7305.1948.tb00917.x
28. [28] T. W. Chaundy and J. B. McLeod, “On a functional equation,” Edinb. Math. Notes, Vol.43, pp. 7-8, 1960. https://doi.org/10.1017/S0950184300003244
29. [29] N. Slonim and N. Tishby, “Agglomerative information bottleneck,” Proc. 13th Int. Conf. Neural Inf. Process. Syst. (NIPS’99), pp. 617-623, 1999.
30. [30] A. Bardera, J. Rigau, I. Boada, M. Feixas, and M. Sbert, “Image segmentation using information bottleneck method,” IEEE Trans. Image Process., Vol.18, No.7, pp. 1601-1612, 2009. https://doi.org/10.1109/TIP.2009.2017823
31. [31] M. Sato, “Presentation of a new formulation of negentropy: 1st report, basic concepts,” Bull. JSME, Vol.25, No.202, pp. 599-605, 1982. https://doi.org/10.1299/JSME1958.25.599
32. [32] F. Nielsen and R. Nock, “On the chi square and higher-order chi distances for approximating -divergences,” IEEE Signal Process. Lett., Vol.21, No.1, pp. 10-13, 2014. https://doi.org/10.1109/LSP.2013.2288355