falkon VS keops

As a satisfied customer (thanks!), was about to recommend KeOps as well. It might also be worth looking into falkon which builds on KeOps and leverages Nystrom approximation and conjugate gradient optimisation to further scale kernel operations.

On the computational side, it is also important to note that kernel methods are now 100-1,000 faster than they were just three years ago. You may be interested by the KeOps library, which is to kernels and geometric ML what cuDNN is to convolutions. You could also have a look at GPyTorch and the Falkon solvers: the software bottlenecks that were holding back kernel methods are progressively being lifted. Million-scale datasets are now routinely processed in minutes/hours and billion-scale problems are starting to become tractable.

On this note, you may be interested by the KeOps library (which is to kernels/geometric ML what cuDNN is to CNNs) and the Falkon solvers: the software bottlenecks that were holding back kernel methods are progressively being lifted. Million-scale datasets are now routinely processed in minutes/hours and billion-scale problems are starting to become tractable. This opens up quite a few possibilities :-)

From direct discussions with the sklearn team, note that this may change relatively soon: a GPU engineer funded by Intel was recently added to the core development team. Last time I met with the team in person (6 months ago), the project was to factor some of the most GPU friendly computations out of the sklearn code base, such as K-Nearest Neighbor search or kernel-related computations, and to document an internal API to let external developers easily develop accelerated backends. As shown by e.g. our KeOps library, GPUs are extremely well suited to classical ML and sklearn is the perfect platform to let users fully take advantage of their hardware. Let’s hope that OP’s question will become redundant at some point in 2023-24 :-)

There has been major progress on the representation of kernel matrices over the last five years. Notably, the KeOps library is an extension for PyTorch/NumPy/etc. that allows you to perform the operations you're thinking of very quickly (10-100 faster than a standard GPU implementation with PyTorch), with low memory usage.

For references on easy-to-use software, you may be interested by e.g. the Falkon and KeOps libraries that were presented as oral/spotlight at last year’s NeurIPS, GPyTorch that you may already know, etc.

You're very welcome! As of today it is still mostly useful when you have less than 50-100 features per point (as detailed here or there), but it's very versatile. We are actively working on making it as useful as possible for the community: if you encounter any issue with it, feel free to let us know!