交叉验证与学习曲线

一、交叉验证

1. 数据集划分：

sklearn.cross_validation.KFold(n,n_folds=3,shuffle=False,random_state=None)

 

参数说明：

 n: 要参与到交叉验证中来的元素个数，一般是全选（如下例中5）

 n_folds = 3: 要分成几堆，也就是K值，默认3，视机器性能进行选择，可选5、7、10等

 shuffle = False: 是否打乱原有顺序呢

 

 代码示例：

 

  输出结果：

X_train: [[ 7 8]，[ 9 10]] y_train: [4 5]

X_test: [[1 2]，[3 4]，[5 6]] ytest: [1 2 3]

X_train: [[1 2]，[3 4]，[5 6]] y_train: [1 2 3]

X_test: [[ 7 8]，[ 9 10]] ytest: [4 5]

也可以使用train_test_split，这里的参数很容易理解。

代码示例及结果：

2. 交叉验证得分：

二、学习曲线

用于判断模型是否过拟合，当模型在训练集上得分很高，但是在交叉验证集上得分很低时，模型过拟合

若模型未过拟合，可以考虑继续挖掘更多特征

import numpy as np
import matplotlib.pyplot as plt
from sklearn.learning_curve import learning_curve

# 用sklearn的learning_curve得到training_score和cv_score，使用matplotlib画出learning curve
def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None, n_jobs=1, 
                        train_sizes=np.linspace(.05, 1., 20), verbose=0, plot=True):
    """
    画出data在某模型上的learning curve.
    参数解释
    ----------
    estimator : 你用的分类器。
    title : 表格的标题。
    X : 输入的feature，numpy类型
    y : 输入的target vector
    ylim : tuple格式的(ymin, ymax), 设定图像中纵坐标的最低点和最高点
    cv : 做cross-validation的时候，数据分成的份数，其中一份作为cv集，其余n-1份作为training(默认为3份)
    n_jobs : 并行的的任务数(默认1)
    """
    train_sizes, train_scores, test_scores = learning_curve(
        estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes, verbose=verbose)

    train_scores_mean = np.mean(train_scores, axis=1)
    train_scores_std = np.std(train_scores, axis=1)
    test_scores_mean = np.mean(test_scores, axis=1)
    test_scores_std = np.std(test_scores, axis=1)

    if plot:
        plt.figure()
        plt.title(title)
        if ylim is not None:
            plt.ylim(*ylim)
        plt.xlabel(u"训练样本数")
        plt.ylabel(u"得分")
        plt.gca().invert_yaxis()
        plt.grid()

        plt.fill_between(train_sizes, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, 
                         alpha=0.1, color="b")
        plt.fill_between(train_sizes, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, 
                         alpha=0.1, color="r")
        plt.plot(train_sizes, train_scores_mean, 'o-', color="b", label=u"训练集上得分")
        plt.plot(train_sizes, test_scores_mean, 'o-', color="r", label=u"交叉验证集上得分")

        plt.legend(loc="best")

        plt.draw()
        plt.show()
        plt.gca().invert_yaxis()

    midpoint = ((train_scores_mean[-1] + train_scores_std[-1]) + (test_scores_mean[-1] - test_scores_std[-1])) / 2
    diff = (train_scores_mean[-1] + train_scores_std[-1]) - (test_scores_mean[-1] - test_scores_std[-1])
    return midpoint, diff

plot_learning_curve(clf, u"学习曲线", X, y)