This Website contains complementary material to the paper:

J. Derrac, S. García and F.Herrera, Fuzzy Nearest Neighbor Algorithms: Taxonomy, Experimental analysis and Prospects. Information Sciences 260 (2014) 98-119, doi: 10.1016/j.ins.2013.10.038

The web is organized according to the following summary:

1. Abstract
2. Survey of Fuzzy Nearest Neighbor Methods
3. Experimental Framework
4. Experimental Study
   1. Stage 1: Comparison of fuzzy nearest neighbor classifiers
   2. Stage 2: Comparison with crisp nearest neighbor approaches

Experimental Study

The full results of the experimental study performed are presented here. This study is composed of two stages: 1) A first one, analyzing the performance of fuzzy nearest neighbor classifiers 2) A second one, comparing the performance of the best performing fuzzy nearest neighbor classifiers with other crisp nearest neighbor based approaches.

Comparison of fuzzy nearest neighbor classifiers

In this first stage, the 18 fuzzy nearest neighbor classifiers are tested. Table 4 summarizes the results obtained, considering the following performance measures: Accuracy in training phase (considering the best K value/configuration for each method), accuracy in test phase (considering the best K value/configuration for each method), accuracy in test phase (considering a fixed K value/configuration for each method), kappa in test phase (considering the best K value/configuration for each method), kappa in test phase (considering a fixed K value/configuration for each method), and running time.

Table 4. Summary results of Stage 1: Fuzzy nearest neighbor classifiers

Note that the values shown in the table are the average results obtained considering the 44 data sets of the experimental study. Table 5 presents XLS sheets containing the detailed results per each data set and configuration, including a XLS sheets for every method. A XLS sheet with the complete results is also provided.

Table 5. Full results of Stage 1: Fuzzy nearest neighbor classifiers

Table 5. Full results of Stage 1: Fuzzy nearest neighbor classifiers

Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file
CFKNN		D-SKNN		FCMKNN		FENN		FRKNNA		FRNN
FRNN-FRS		FRNN-VQRS		FuzzyKNN		FuzzyNPC		GAFuzzyKNN		IF-KNN
IFSKNN		IFV-NP		IT2FKNN		PFKNN		PosIBL		VWFuzzyKNN
Complete results

Table 6. Full results of Friedman and Shaffer tests (Accuracy and Kappa) - Stage 1: Fuzzy nearest neighbor classifiers

Comparison with crisp nearest neighbor approaches

In this second stage, the 7 best performing fuzzy nearest neighbor classifiers are compared with 7 state-of-art crisp nearest neighbor classifier. The 7 fuzzy nearest neighbor classifiers selected are FuzzyKNN and the best performing method of each family (GAFuzzyKNN, IT2FKNN, D-SKNN, IF-KNN, FRNN-FRS and FENN).

Table 7 summarizes the results obtained, considering the following performance measures: Accuracy in training phase (considering the best K value/configuration for each method), accuracy in test phase (considering the best K value/configuration for each method), accuracy in test phase (considering a fixed K value/configuration for each method), kappa in test phase (considering the best K value/configuration for each method), kappa in test phase (considering a fixed K value/configuration for each method), and running time.

Table 7. Summary results of Stage 2: Comparison with crisp nearest neighbor approaches

Note that the values shown in the table are the average results obtained considering the 44 data sets of the experimental study. Table 8 presents XLS sheets containing the detailed results per each data set and configuration, including a XLS sheets for every method. A XLS sheet with the complete results is also provided.

Table 8. Full results of Stage 2: Comparison with crisp nearest neighbor approaches

Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file	Method	XLS file
D-SKNN		ENN		FENN		FRNN-FRS		FuzzyKNN		GAFuzzyKNN		IDIBL
IF-KNN		IT2FKNN		KNN		KNNAdaptive		KSNN		NSC		PW
Complete results

The next step of the study involves the use of the Friedman and Shaffer tests to contrast the results shown in the former tables. For the sake of generality, these statistical analyses have been carried out considering the results presented above, using the accuracy and kappa performance measures with a fixed K value.

Table 9 presents the results of both tests for accuracy and kappa measures. Firstly, a column is presented showing the ranks obtained in the Friedman test, where the lower the rank is, the better behavior the respective algorithm has shown. The p-values obtained by the Friedman test are 1.17E-6 and 5.59E-6 (for accuracy and kappa, respectively), which means that significant differences exists among the algorithms.

Shaffer test is conducted to characterize these differences. 91 pairwise hypotheses can be established, from which in the case of accuracy 10 are significant at a α = 0.1 level of significance (4 at a α = 0.01 level of significance). In the case of kappa, 8 and 3 hypotheses are significant, respectively.

For each method, Table 6 shows the number of algorithms which are significantly improved by it (the "+" column) and the number of algorithms which are significantly improved or equal (the "+=" column).
An XSL sheet containing the Friedman Ranks and the 91 hypotheses with their associated p-values can be dowloaded here.

Table 9. Full results of Friedman and Shaffer tests (Accuracy and Kappa) - Stage 2: Comparison with crisp nearest neighbor approaches