No Access

Generalized support vector regression: Duality and tensor-kernel representation

Saverio Salzo

CSML, Istituto Italiano di Tecnologia, Via E. Melen 83, 16152 Genova, Italy

E-mail Address: [email protected]

Corresponding author.

Search for more papers by this author

and

Johan A. K. Suykens

KU Leuven, ESAT-STADIUS, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium

E-mail Address: [email protected]

Search for more papers by this author

https://doi.org/10.1142/S0219530519410069Cited by:2 (Source: Crossref)

This article is part of the issue:

Special Issue on Mathematics of Data Science

Abstract

In this paper, we study the variational problem associated to support vector regression in Banach function spaces. Using the Fenchel–Rockafellar duality theory, we give an explicit formulation of the dual problem as well as of the related optimality conditions. Moreover, we provide a new computational framework for solving the problem which relies on a tensor-kernel representation. This analysis overcomes the typical difficulties connected to learning in Banach spaces. We finally present a large class of tensor-kernels to which our theory fully applies: power series tensor kernels. This type of kernels describes Banach spaces of analytic functions and includes generalizations of the exponential and polynomial kernels as well as, in the complex case, generalizations of the Szegö and Bergman kernels.

Keywords:

References

1. H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces (Springer, New York, 2011). Google Scholar
2. V. I. Bogachev, Measure Theory (Springer, Berlin, 2007). Google Scholar
3. M. Bóna, A Walk Through Combinatorics, 3rd edn. (World Scientific, Singapore, 2011). Link, Google Scholar
4. A. Christmann and D. X. Zhou, Learning rates for the risk of kernel-based quantile regression estimators in additive models, Anal. Appl. 14 (2016) 449–477. Link, Web of Science, Google Scholar
5. I. Cioranescu, Geometry of Banach Spaces, Duality Mappings and Nonlinear Problems (Kluwer, Dordrecht, 1990). Google Scholar
6. P. L. Combettes, S. Salzo and S. Villa, Consistency of regularized learning schemes in Banach spaces, Anal. Appl. 16 (2018) 1–54. Link, Web of Science, Google Scholar
7. P. L. Combettes, S. Salzo and S. Villa, Consistent learning by composite proximal thresholding, Math. Program. Ser. B 167 (2018) 99–127. Web of Science, Google Scholar
8. N. Cristianini and J. Shawe-Taylor, An Introduction to Support Vector Machines (Cambridge University Press, Cambridge, 2000). Google Scholar
9. C. De Mol, E. De Vito and L. Rosasco, Elastic-net regularization in learning theory, J. Complex. 25 (2009) 201–230. Web of Science, Google Scholar
10. R. Der and D. Lee, Large-margin classification in Banach spaces, in Proc. 11th Int. Conf. Artificial Intelligence and Statistics, Vol. 2, eds. M. Meila and X. Shen (PMLR, 2007), pp. 91–98. Google Scholar
11. E. De Vito, L. Rosasco, A. Caponnetto, M. Piana and A. Verri, Some properties of regularized kernel methods, J. Mach. Learn. Res. 5 (2004) 1363–1390. Web of Science, Google Scholar
12. G. E. Fasshauer, F. J. Hickernell and Q. Ye, Solving support vector machines in reproducing kernel Banach spaces with positive definite functions, Appl. Comput. Harmon. Anal. 38 (2015) 115–139. Web of Science, Google Scholar
13. Y. Feng, S. Lv, H. Hang and J. A. K. Suykens, Kernelized elastic net regularization: Generalization bounds and sparse recovery, Neural Comput. 28 (2016) 525–562. Web of Science, Google Scholar
14. W. Fu, Penalized regressions: The bridge versus the lasso, J. Comput. Graph. Stat. 7 (1998) 397–416. Crossref, Web of Science, Google Scholar
15. F. Girosi, An equivalence between sparse approximation and support vector machines, Neural Comput. 10 (1998) 1455–1480. Web of Science, Google Scholar
16. J.-B. Hiriart-Urruty and C. Lemaréchal, Convex Analysis and Minimization Algorithms II (Springer, Berlin, 1996). Google Scholar
17. T. Hofmann, B. Schölkopf and A. J. Smola, Kernel methods in machine learning, Ann. Statist. 36 (2008) 1171–1220. Web of Science, Google Scholar
18. V. Koltchinskii, Sparsity in penalized empirical risk minimization, Ann. Inst. Henri Poincaré Probab. Stat. 45 (2009) 7–57. Web of Science, Google Scholar
19. B. S. Mendelson and J. Neeman, Regularization in kernel learning, Ann. Statist. 38 (2010) 526–565. Web of Science, Google Scholar
20. C. A. Micchelli, M. Pontil and D. X. Zhou, Error bounds for learning the kernel, Anal. Appl. 14 (2016) 849–868. Link, Web of Science, Google Scholar
21. V. I. Paulsen and M. Raghupathi, An Introduction to the Theory of Reproducing Kernel Hilbert Spaces (Cambridge University Press, Cambridge, 2016). Google Scholar
22. R. T. Rockafellar, Conjugate Duality and Optimization (SIAM, Philadelphia, PA, 1974). Google Scholar
23. S. Salzo, The variable metric forward-backward splitting algorithm under mild differentiability assumptions, SIAM J. Optim. 27 (2017) 2153–2181. Web of Science, Google Scholar
24. S. Salzo, L. Rosasco and J. A. K. Suykens, Solving $ℓ^{p}$ -norm regularization with tensor kernels, in Proc. 21st Int. Conf. Artificial Intelligence and Statistics, Vol. 84, eds. A. Storkey and F. Perez-Cruz (PMLR, 2018), pp. 1655–1663. Google Scholar
25. B. Schölkopf, R. Herbrich and A. J. Smola, A generalized representer theorem, in Computational Learning Theory: 14th Annual Conf. Computational Learning Theory, COLT 2001 (Springer, Berlin, 2001). Google Scholar
26. B. K. Sriperumbudur, K. Fukumizu and G. R. G. Lanckriet, Learning in Hilbert vs. Banach spaces: A measure embedding viewpoint, in Advances in Neural Information Processing Systems, Vol. 24 (Curran Associates, 2011). Google Scholar
27. I. Steinwart and A. Christmann, Support Vector Machines (Springer, New York, 2008). Google Scholar
28. I. Steinwart and A. Christmann, Sparsity of SVMs that use the $𝜀$ -insensitive loss, in Advances in Neural Information Processing Systems, Vol. 21 (Curran Associates, 2009). Google Scholar
29. H. Sun and Q. Wu, Indefinite kernel network with dependent sampling, Anal. Appl. 11 (2013) 1350020. Link, Web of Science, Google Scholar
30. J. A. K. Suykens, T. Van Gestel, J. De Brabanter, B. De Moor and J. Vandewalle, Least Squares Support Vector Machines (World Scientific, Singapore, 2002). Link, Google Scholar
31. V. N. Vapnik, Statistical Learning Theory (Wiley, New York, 1998). Google Scholar
32. Y. Xu and Q. Ye, Generalized Mercer kernels and reproducing kernel Banach spaces, Mem. Amer. Math. Soc. 258 (2019) vi+122. Web of Science, Google Scholar
33. R. M. Young, An Introduction to Nonharmonic Fourier Series (Academic Press, San Diego, 2001). Google Scholar
34. C. Zălinescu, Convex Analysis in General Vector Spaces (World Scientific, River Edge, NJ, 2002). Link, Google Scholar
35. H. Zhang, Y. Xu and J. Zhang, Reproducing kernel Banach spaces for machine learning, J. Mach. Learn. Res. 10 (2009) 2741–2775. Web of Science, Google Scholar
36. H. Zhang and J. Zhang, Regularized learning in Banach spaces as an optimization problem: Representer theorems, J. Global Optim. 54 (2012) 235–250. Web of Science, Google Scholar
37. B. Zwicknagl, Power series kernels, Constr. Approx. 29 (2009) 61–84. Web of Science, Google Scholar

Remember to check out the Most Cited Articles!
Check out our Differential Equations and Mathematical Analysis books in our Mathematics 2021 catalogue Featuring authors such as Ronen Peretz, Antonio Martínez-Abejón & Martin Schechter