Abstract

This is the first paper in a sequence of studies including also [#!llhm2022!#] and [#!llhm2022_1!#] in which we introduce a new type of neural networks (NNs) – wavelet-based neural networks (WBNNs) – and study their properties and potential for applications. We begin this study with a comparison to the currently existing type of wavelet neural networks (WNNs) and show that WBNNs vastly outperform WNNs. One reason for the vast superiority of WBNNs is their advanced hierarchical tree structure based on biorthonormal multiresolution analysis (MRA). Another reason for this is the implementation of our new idea to incorporate the wavelet tree depth into the neural width of the NN. The separation of the roles of wavelet depth and neural depth provides a conceptually and algorithmically simple but very highly efficient methodology for sharp increase in functionality of swarm and deep WBNNs and rapid acceleration of the machine learning process.

Citation details of the article

Journal: International Journal of Applied Mathematics
Journal ISSN (Print): ISSN 1311-1728
Journal ISSN (Electronic): ISSN 1314-8060
Volume: 36
Issue: 2
Year: 2023

DOI: 10.12732/ijam.v36i2.5

Download Section

Download the full text of article from here.

You will need Adobe Acrobat reader. For more information and free download of the reader, please follow this link.

References

1. [1] I. Daubechies, Ten Lectures on Wavelets, Society for Industrial and Applied Mathematics, USA (1992).
2. [2] W. Dahmen, Wavelet and multiscale methods for operator equations, Acta Numerica, 6 (1997), 55-228.
3. [3] Q. Zhang, A. Benveniste, Wavelet networks, IEEE Trans Neural Networks, 3, No 6 (1992), 889-898.
4. [4] Z. Lu, H. Pu, F. Wang, Z. Hu, L. Wang, The expressive power of neural networks: A view from the width, In: Advances in Neural Information Processing Systems, 30 (2017).
5. [5] A.K. Alexandridis, A.D. Zapranis, Wavelet neural networks: A practical guide, Neural Networks, 42 (2013), 1-27.
6. [6] S. Chen, C.F.N. Cowan, P.M. Grant, Orthogonal least squares learning algorithm for radial basis function networks, IEEE Transactions on Neural Networks, 2, No 2 (1991), 302-309.
7. [7] Y. LeCun, Y. Bengio, G. Hinton, Deep learning, Nature, 521, No 7553 (2015), 436-444.
8. [8] W. Dahmen, Wavelet and multiscale methods for operator equations, Acta Numerica, 6 (1997), 55-228.
9. [9] J. Bergh, J. L¨ofstr¨om, Interpolation Spaces. An Introduction, Grundlehren der Mathematischen Wissenschaften, No. 223, Springer-Verlag, Berlin-New York (1976), x+207.
10. [10] L. Dechevsky, S. Penev, On shape-preserving probabilistic wavelet approximators, Stochastic Analysis and Applications, 15, No 2 (1997), 187-215.
11. [11] A. Cohen, I. Daubechies, P. Vial, Multiresolution analysis, wavelets and fast algorithms on an interval, C.R. Acad. Sci. Paris, Ser. I Math., 316, No 5 (1993), 417-421.
12. [12] A. Cohen, I. Daubechies, P. Vial, Wavelets on the interval and fast wavelet transforms, Appl. Comp. Harmonic Anal., 1, No 1 (1993), 54-81.
13. [13] G. Cybenko, Approximation by superpositions of a sigmoidal function, Math. Control Signal Systems, 2, No 4 (1989), 303-314.
14. [14] G. Cybenko, Correction: “Approximation by superpositions of a sigmoidal function”, [Math. Control Signal Systems, 2.4 (1989), pp. 303-314], Math. Control Signal Systems, 5, No 4 (1992), 455.
15. [15] M. Reed, B. Simon, Methods of Modern Mathematical Physics, Vol. 1: Functional Analysis, 2nd Ed., Academic Press [Harcourt Brace Jovanovich, Publishers], New York (1980).
16. [16] L. Dechevsky, S. Penev, On shape-preserving wavelet estimators of cumulative distribution functions and densities, Stochastic Analysis and Applications, 16, No 3 (1998), 423-462.
17. [17] H. Triebel, Theory of Function Spaces, Monographs in Mathematics, Vol. 78, Birkh¨auser Verlag, Basel (1983).
18. [18] S.G. Samko, A.A. Kilbas, O.I. Marichev, Fractional Integrals and Derivatives: Theory and Applications, Gordon and Breach Science Publishers, Yverdon (1993).
19. [19] M. Frazier, B. Jawerth, G. Weiss, Littlewood-Paley Theory and the Study of Function Spaces, CBMS Regional Conf. Ser. in Mathematics, Vol. 79, Amer. Math. Soc., Providence, RI (1991).
20. [20] H. Iba, Evolutionary Approach to Machine Learning and Deep Neural Networks. Neuro-evolution and Gene Regulatory Networks, Springer, Singapore (2018).
21. [21] H. Iba, Swarm Intelligence and Deep Evolution. Evolutionary Approach to Artificial Intelligence, Taylor & Francis CRC Press (2022).
22. [22] F. Hoyle, The Black Cloud, William Heinemann Ltd. (1957).
23. [23] S. Lem, Niezwyciezony, Wydawnictwo MON (1964).
24. [24] S. Lem, The Invincible, The MIT Press (2020).
25. [25] F.T. Leighton, Introduction to Parallel Algorithms and Architectures. Arrays, Trees, Hypercubes, Morgan Kaufman Publishers, Inc., San Mateo, CA (1992).
26. [26] L.T. Dechevsky, J.O. Ramsay, S.I. Penev, Penalized wavelet estimation with Besov regularity constraints, Math. Balkanica (N.S.), 13, No 3-4 (1999), 257-376.
27. [27] L.T. Dechevsky, Atomic decomposition of function spaces and fractional integral and differential operators, Fract. Calc. Appl. Anal., 2, No 4 (1999), 367-381.
28. [28] R. Okuta, Y. Unno, D. Nishino, S. Hido, C. Loomis, CuPy: A NumPyCompatible Library for NVIDIA GPU Calculations, In: Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirtyfirst Annual Conference on Neural Information Processing Systems (NIPS) (2017), http://learningsys.org/nips17/assets/papers/paper16.pdf.
29. [29] P. Brenner, V. Thom´ee, L.B. Wahlbin, Besov Spaces and Applications to Difference Methods for Initial Value Problems, Lect. Notes in Math., No. 434, Springer-Verlag, Berlin-New York (1975).
30. [30] J. Schmidhuber, Deep learning in neural networks: An overview, Neural Networks, 61 (2015), 85-117.
31. [31] L.T. Dechevsky, L.M. Gulliksen, A multirigid dynamical programming algorithm for discrete dynamical systems and its applications to numerical computation of global geodesics, Int. J. Pure Appl. Math., 33, No 2 (2006), 257–286.
32. [32] L.T. Dechevsky, L.M. Gulliksen, Application of a multirigid dynamical programming algorithm to optimal parametrization, and a model solution of an industrial problem, Int. J. Pure Appl. Math., 33, No 3 (2006), 381- 406.
33. [33] L.T. Dechevsky, L.-E. Person, H. Singh, K.M. Tangrand, Learning nonparametric regression-functions and densities by univariate wavelet-based neural networks (2022).
34. [34] L.T. Dechevsky, L.-E. Person, H. Singh, K.M. Tangrand, Learning of multidimensional geometric manifolds with wavelet-based neural networks (2022).
35. [35] K. Jetten, S. Smale, D.-X. Zhou, Learning theory and approximation, In: Mathematisches Forschungsintitut Oberwolfach Workshop, June 29th - July 5th. Oberfolfach Report 30/2008 (2008), 1655-1705.
36. [36] P. Binev, A. Cohen, W. Dahmen, R. DeVore, Classification algorithms using adaptive partitioning, Ann. Statist., 42, No 6 (2014), 2141-2163.
37. [37] P.P. Petrushev, V.A. Popov, Rational Approximation of Real Functions, Encyclopedia of Mathematics and its Applications, No. 28, Cambridge University Press, Cambridge (1987).
38. [38] L.T. Dechevsky, J. Bratlie, B. Bang, A. Laks˚a, J. Gundersen, Waveletbased lossless one- and two-dimensional representation of multidimensional geometric data, In: AIP Conf. Proc., 1410, Amer. Inst. Phys., Melville, NY (2011), 83-97.
39. [39] L.T. Dechevsky, J. Bratlie, J. Gundersen, Index mapping between tensorproduct wavelet bases of different number of variables, and computing multivariate orthogonal discrete wavelet transforms on graphics processing units, Lecture Notes in Comput. Sci., 7116, Springer, Heidelberg (2012), 402-410.
40. [40] P. Binev, A. Cohen, W. Dahmen, R. DeVore, Universal algorithms for learning theory. II. Piecewise polynomial functions, Constr. Approx., 26, No 2 (2007), 127-152.
41. [41] M. Hairer, C. Labb´e, The reconstruction theorem in Besov spaces, J. of Functional Analysis, 273, No 8 (2017), 2578-2618.