Machine learning with kernel methods, Spring 2018

Julien Mairal and Jean-Philippe Vert

MSc Mathematics, Vision, Learning (MVA) (ENS Cachan)

MSc Data science (Ecole Polytechnique, ENSAE ParisTech, Telecom ParisTech, University Paris Saclay)

MSc Mathematics for Life Sciences (MathSV) (University Paris South, Ecole Polytechnique, ENS Cachan)

MSc Mathematiques, Apprentissage et Sciences Humaines (MASH) (PSL Research University, Paris-Dauphine University, ENS Paris)



Slides are frequently updated. Please let us know if you spot typos!


Many problems in real-world applications of machine learning can be formalized as classical statistical problems, e.g., pattern recognition, regression or dimension reduction, with the caveat that the data are often not vectors of numbers. For example, protein sequences and structures in computational biology, text and XML documents in web mining, segmented pictures in image processing, or time series in speech recognition and finance, have particular structures which contain relevant information for the statistical problem but can hardly be encoded into finite-dimensional vector representations.

Kernel methods are a class of algorithms well suited for such problems. Indeed they extend the applicability of many statistical methods initially designed for vectors to virtually any type of data, without the need for explicit vectorization of the data. The price to pay for this extension to non-vectors is the need to define a so-called positive definite kernel function between the objects, formally equivalent to an implicit vectorization of the data. The "art" of kernel design for various objects have witnessed important advances in recent years, resulting in many state-of-the-art algorithms and successful applications in many domains.

The goal of this course is to present the mathematical foundations of kernel methods, as well as the main approaches that have emerged so far in kernel design. We will start with a presentation of the theory of positive definite kernels and reproducing kernel Hilbert spaces, which will allow us to introduce several kernel methods including kernel principal component analysis and support vector machines. Then we will come back to the problem of defining the kernel. We will present the main results about Mercer kernels and semigroup kernels, as well as a few examples of kernel for strings and graphs, taken from applications in computational biology, text processing and image analysis. Finally we will touch upon topics of active research, such as large-scale kernel methods and deep kernel machines.



Lecture (in english) usually take place at ENS Paris, 29 rue d'Ulm, 75005 Paris, in amphi Jaures, 1:30-4pm.

On Feb 14, the lecture will take place at Institut Curie, 12 rue Lhomond, 75005 Paris, in amphi Burg, 1:30-4pm.

WARNING: DATE CHANGE FOR THE EXAM: we had to move the exam from March 21 to March 28 as no large room was available March 21. If you can not attend the March 28 exam, let us know ASAP.

DatePlaceLecturerTopicSlidesMore material
Jan 17ENS Ulm, amphi JauresJPVPositive definite kernel, RKHS, Aronszajn's theorem1-38Uniqueness of the RKHS Aronszajn's theorem
Jan 24ENS Ulm, amphi JauresJMKernel trick, Representer theorem, kernel ridge regression39-97
Jan 31ENS Ulm, amphi JauresJPVSupervised classification, Kernel logistic regression, large margin classifiers, SVM98-159
Feb 7ENS Ulm, amphi JauresJMUnsupervised analysis, kernel PCA, kernel CCA, kernel K-means162-198
Feb 14Institut Curie, amphi BurgJPVGreen, Mercer, Herglotz and Bochner kernels199-273
Feb 21ENS Ulm, amphi JauresJMKernels from generative models, string kernels294-370review paper on string kernels
Feb 28Break
Mar 7ENS Ulm, amphi JauresJMLarge-scale kernel machines, deep kernel learning538-639
Mar 14ENS Ulm, amphi JauresJPVGraph kernels, kernels on graphs
Mar 21MINES ParisTech, Room V1121:30pm-3:30pm: Final exam, only if you can not attend the March 28 exam. In that case, please contact JP Vert to explain why you can not attend March 28's exam. You can only take one exam.
Mar 28MINES ParisTech, Room L1082pm-4pm: Final exam


The final note will be an average of a project (40%) and a final written exam (60%). You can solve work on the project alone or with up to two friends.
The data challenge is now online. Please use the link given during the class.
No material (slides, book...) is allowed for the final written exam. To practice you may want to have a look at the homeworks given in the previous years, e.g., here or here.


Link to the summer school on AI mentioned on Feb 28th (price for students may go slightly down). Appications will be considered until April 4th.
Back to homepage