scikit-learn: a machine learning toolbox in Python

Authors: Nelle Varoquaux, Gael Varoquaux, Andreas Mueller, etc...

scikit-learn

Machine Learning in Python

  • Simple and efficient tools for data mining and data analysis
  • Accessible to everybody, and reusable in various contexts
  • Built on NumPy, SciPy, and matplotlib
  • Open source, commercially usable - BSD license

Machine learning is the process to automatically extract knowledge from data, usually with the goal of making predictions on new, unseen data. A classical example is a spam filter, for which the user keeps labeling incoming mails as either spam or not spam. A machine learning algorithm then “learns” what distinguishes spam from normal emails, and can predict for new emails whether they are spam or not.

Central to machine learning is the concept of making decision automatically from data, without the user specifying explicit rules how this decision should be made.

For the case of emails, the user doesn’t provide a list of words or characteristics that make an email spam. Instead, the user provides examples of spam and non-spam emails.

The second central concept is generalization. The goal of a machine learning algorithm is to predict on new, previously unseen data. We are not interested in marking an email as spam or not, that the human already labeled. Instead, we want to make the users life easier by making an automatic decision for new incoming mail.

The data is presented to the algorithm usually as an array of numbers. Each data point (also known as sample) that we want to either learn from or make a decision on is represented as a list of numbers, called features, that reflect properties of this point.

There are two kinds of machine learning we will talk about today: Supervised learning and unsupervised learning

Supervised Learning: Classification and regression

In Supervised Learning, we have a dataset consisting of both input features and a desired output, such as in the spam / no-spam example. The task is to construct a model (or program) which is able to predict the desired output of an unseen object given the set of features.

Some more complicated examples are:

  • given a multicolor image of an object through a telescope, determine whether that object is a star, a quasar, or a galaxy.
  • given a photograph of a person, identify the person in the photo.
  • given a list of movies a person has watched and their personal rating of the movie, recommend a list of movies they would like.
  • given a persons age, education and position, infer their salary

What these tasks have in common is that there is one or more unknown quantities associated with the object which needs to be determined from other observed quantities.

Supervised learning is further broken down into two categories, classification and regression. In classification, the label is discrete, such as “spam” or “no spam”. In other words, it provides a clear-cut distinction between categories. In regression, the label is continuous, that is a float output. For example, in astronomy, the task of determining whether an object is a star, a galaxy, or a quasar is a classification problem: the label is from three distinct categories. On the other hand, we might wish to estimate the age of an object based on such observations: this would be a regression problem, because the label (age) is a continuous quantity.

In supervised learning, there is always a distinction between a training set for which the desired outcome is given, and a test set for which the desired outcome needs to be inferred. More about that later.

Unsupervised Learning

In Unsupervised Learning there is no desired output associated with the data. Instead, we are interested in extracting some form of knowledge or model from the given data. In a sense, you can think of unsupervised learning as a means of discovering labels from the data itself. Unsupervised learning is often harder to understand and to evaluate.

Unsupervised learning comprises tasks such as dimensionality reduction, clustering, and density estimation, e.g.

sklearn estimator API

Scikit-learn strives to have a uniform interface across all objects. Given a scikit-learn estimator named model, the following methods are available:

  • Available in all Estimators

    • model.fit() : fit training data. For supervised learning applications, this accepts two arguments: the data X and the labels y (e.g. model.fit(X, y)). For unsupervised learning applications, fit takes only a single argument, the data X (e.g. model.fit(X)).

  • Available in supervised estimators

    • model.predict() : given a trained model, predict the label of a new set of data. This method accepts one argument, the new data X_new (e.g. model.predict(X_new)), and returns the learned label for each object in the array.
    • model.predict_proba() : For classification problems, some estimators also provide this method, which returns the probability that a new observation has each categorical label. In this case, the label with the highest probability is returned by model.predict().
    • model.score() : An indication of how well the model fits the training data. Scores are between 0 and 1, with a larger score indicating a better fit.

Data in scikit-learn

Data in scikit-learn, with very few exceptions, is assumed to be stored as a two-dimensional array, of size [n_samples, n_features]. Many algorithms also accept scipy.sparse matrices of the same shape.

  • n_samples: The number of samples: each sample is an item to process (e.g. classify). A sample can be a document, a picture, a sound, a video, an astronomical object, a row in database or CSV file, or whatever you can describe with a fixed set of quantitative traits.
  • n_features: The number of features or distinct traits that can be used to describe each item in a quantitative manner. Features are generally real-valued, but may be boolean or discrete-valued in some cases. The number of features must be fixed in advance. However it can be very high dimensional (e.g. millions of features) with most of them being zeros for a given sample. This is a case where scipy.sparse matrices can be useful, in that they are much more memory-efficient than numpy arrays.

Each sample (data point) is a row in the data array, and each feature is a column.

An unsupervised learning example

import numpy as np

import matplotlib.pyplot as plt



from sklearn.cluster import KMeans

from sklearn.datasets.samples_generator import make_blobs



# Generate sample data



batch_size = 45

centers = [[1, 1], [-1, -1], [1, -1]]

n_clusters = len(centers)

X, labels_true = make_blobs(n_samples=3000, centers=centers, cluster_std=0.7)



# Compute clustering with Means



k_means = KMeans(init='k-means++', n_clusters=3, n_init=10)

k_means.fit(X)

k_means_labels = k_means.labels_

k_means_cluster_centers = k_means.cluster_centers_

k_means_labels_unique = np.unique(k_means_labels)



  # Plot result



fig, ax  = plt.subplots()

colors = ['#4EACC5', '#FF9C34', '#4E9A06']



# KMeans

for k, col in zip(range(n_clusters), colors):

    my_members = k_means_labels == k

    cluster_center = k_means_cluster_centers[k]

    ax.plot(X[my_members, 0], X[my_members, 1], 'w',

            markerfacecolor=col, marker='.')

    ax.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,

            markeredgecolor='k', markersize=6)

ax.set_title('KMeans')

ax.set_xticks(())

ax.set_yticks(())

plt.text(-3.5, 1.8,  'inertia: %f' % (

    k_means.inertia_))



plt.show()

A supervised learning example

import numpy as np

import matplotlib.pyplot as plt

from sklearn.linear_model import SGDClassifier

from sklearn.datasets.samples_generator import make_blobs



# we create 50 separable points

X, Y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.60)



# fit the model

clf = SGDClassifier(loss="hinge", alpha=0.01, n_iter=200, fit_intercept=True)

clf.fit(X, Y)



# plot the line, the points, and the nearest vectors to the plane

xx = np.linspace(-1, 5, 10)

yy = np.linspace(-1, 5, 10)



X1, X2 = np.meshgrid(xx, yy)

Z = np.empty(X1.shape)

for (i, j), val in np.ndenumerate(X1):

    x1 = val

    x2 = X2[i, j]

    p = clf.decision_function([[x1, x2]])

    Z[i, j] = p[0]

levels = [-1.0, 0.0, 1.0]

linestyles = ['dashed', 'solid', 'dashed']

colors = 'k'

plt.contour(X1, X2, Z, levels, colors=colors, linewidth=1, linestyles=linestyles)

plt.scatter(X[:, 0], X[:, 1], c=Y, s=40, cmap=plt.cm.Paired)



plt.axis('tight')

plt.show()