A Study on the Impact of Partition-Induced Dataset Shift on k-fold Cross-Validation - Complementary Material

This Website contains additional material to the SCI2S research paper: A Study on the Impact of Partition-Induced Dataset Shift on k-fold Cross-Validation

J.G. Moreno-Torres, J.A. Sáez, and F. Herrera, A Study on the Impact of Partition-Induced Dataset Shift on k-fold Cross-Validation. IEEE Transactions on Neural Networks 23(8): 1304-1312 (2012) PDF Icon

Summary:

  1. Abstract
  2. Single-experiment classifier performance analysis results
  3. Convergence to stable classifier performance estimate results

Abstract

J.G. Moreno-Torres, J.A. Sáez, and F. Herrera, A Study on the Impact of Partition-Induced Dataset Shift on k-fold Cross-Validation.

Cross-validation is a very commonly employed technique to evaluate classifier performance. However, it can potentially introduce dataset shift, a harmful factor that is often not taken into account, and which can result in inaccurate performance estimation. This works analyzes both the prevalence and impact of partition-induced covariate shift on different k-fold cross-validation schemes.

From the experimental results obtained we conclude that the degree of partition-induced covariate shift depends on the cross-validation scheme considered. In this way, worse schemes may harm the correctness of a single classifier performance estimation and also increase the needed number of repetitions of cross-validation to reach a stable performance estimation.

Single-experiment classifier performance analysis results

This section includes the results for the single-experiment classifier performance analysis experiment. There are 4 files, one for each type of partitioning studied (DOB-SCV, DB-SCV, SCV and MS-SCV). Inside each file, you can find the results divided in sheets. Each sheet corresponds to a different partition granularity: 10x1, 5x2 and 2x5. On each sheet, you can then find the test AUC obtained by each method on each dataset.

We also include here the results of the partitions created to obtain "true" classifier estimations. These are presented in a single file, since we use the same data as a reference when studiying all 4 methods. Remember the presented results are classifier performance measured as ROC AUC in the test set.

  DOB-SCV DB-SCV SCV MS-SCV
 Microsoft Excel format (.xls) Download Download Download Download
 Open Office Calc format (.ods) Download Download Download Download
 Microsoft Excel format (.xls) true estimations Download
 Open Office Calc format (.ods) true estimations Download

You can also download all files compressed into a zip file here.

Convergence to stable classifier performance estimate results

This section includes the results for the experiment regarding the number of repetitions of cross-validation needed to achieve a stable classifier performance. Again, there are 4 files, one for each type of partitioning studied. Inside each file, you can find the results divided in sheets; again, one for each partition granularity: 10x1, 5x2 and 2x5.

On each sheet, you can then find the average number of iterations each classifier needed to converge to a stable estimation on each dataset.

  DOB-SCV DB-SCV SCV MS-SCV
 Microsoft Excel format (.xls) Download Download Download Download
 Open Office Calc format (.ods) Download Download Download Download

You can also download all files compressed into a zip file here.