In Machine Learning it is often necessary to evaluate the performances of a classification or regression function that is derived through a learning process. Generally, it means measuring performance indicators over a test dataset. These performances can be then used to decide whether the learning algorithm with its parameterization is good enough for the task of interest. This is a pattern that is repeated every time a new experiment is necessary.<\/p>\n
Performance measures are the same for many different tasks, thus their computation can be easily standardized in order to support many scenarios. In KeLP, we provided the Evaluator<\/a>\u00a0abstract class, which serves as a base class for other performance evaluations classes. The Evaluator<\/a> class contains a public implemented method, whose name is getPerformanceMeasure(String, Object…)<\/em> that will access the internal class methods by means of Java reflection to return a performance measure. So for example, if a specific implementation of an evaluator offers the method to compute the accuracy, whose name is getAccuracy()<\/em>, then the getPerformanceMeasure<\/em> can be invoked as getPerformanceMeasure(“Accuracy”)<\/em>. It serves as a general interface to retrieve the performance measures computed by a specific evaluators. Notice that an evaluator must contain methods whose name is get<\/em><MeasureName>\u00a0to be compliant to the getPerformanceMeasure<\/em> mechanism. This implementation pattern is necessary to support the generic instantiation of an evaluator in the case automatic classes for experiments will be provided.<\/p>\n The Evaluator<\/a>\u00a0class contains 4\u00a0abstract methods that should be implemented by its sub-classes in order to respect the Evaluator<\/a>\u00a0contract. In KeLP\u00a0some implementation of Evaluator<\/a>\u00a0class are available: BinaryClassificationEvaluator<\/a>, MulticlassClassificationEvaluator<\/a>\u00a0and RegressorEvaluator<\/a>. They are intended to satisfy the major needs when dealing with binary classification tasks, multi-class classification tasks and regression tasks.<\/p>\n In ML it is often necessary to tune the classifier parameters or to make more reliable measures in an experiment via cross-validation. These activities are repetitive, and it is easy to extract patterns in code that can be re-used. In KeLP\u00a0we provide several methods\u00a0for automatizing some of these activities, collected in the class ExperimentUtils<\/a>. <\/p>\n Instead, the following code uses the ExperimentUtils.nFoldCrossValidation<\/em>\u00a0method to perform a 5 cross-fold validation:<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Evaluators In Machine Learning it is often necessary to evaluate the performances of a classification or regression function that is derived through a learning process. Generally, it means measuring performance indicators over a test dataset. These performances can be then used to decide whether the learning algorithm with its parameterization is good enough for the Read more about Experiment Utilities<\/span>[…]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":112,"menu_order":16,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/pages\/187"}],"collection":[{"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/pages"}],"about":[{"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=187"}],"version-history":[{"count":11,"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/pages\/187\/revisions"}],"predecessor-version":[{"id":898,"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/pages\/187\/revisions\/898"}],"up":[{"embeddable":true,"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=\/wp\/v2\/pages\/112"}],"wp:attachment":[{"href":"http:\/\/www.kelp-ml.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=187"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nThe four abstract methods of the Evaluator<\/a>\u00a0class are:<\/p>\n\n
Experiment Utils<\/h2>\n
\nFirst, this class contains a method test(PredictionFunction, Evaluator, Dataset)<\/em> the will produce in output a List<<\/strong>Prediction<\/a>><\/strong> and will update the “counters” of the Evaluator<\/a>. It serves to automatize the repetitive operations that are executed for classifying a test set. For example,\u00a0the following code uses the ExperimentUtils.test<\/em> method to evaluate a binary classifier with a fixed train-test split:<\/p>\n\/**\r\n * DATA LOADING\r\n *\/\r\nSimpleDataset tr = new SimpleDataset();\r\ntr.populate(\"<path_to_the_train_dataset_in_kelp_format>\");\r\n\r\nSimpleDataset testSet = new SimpleDataset();\r\ntestSet.populate(\"<path_to_the_test_dataset_in_kelp_format>\");\r\n\r\n\/**\r\n * SETTING THE LEARNING ALGORITHM\r\n *\/\r\nStringLabel positiveLabel = new StringLabel(\"+1\");\r\nLinearPassiveAggressiveClassification pa=new LinearPassiveAggressiveClassification();\r\n\/\/ it is assumed that the data instances have a vector representation called \"REPR1\"\r\npa.setRepresentation(\"REPR1\");\r\n\/\/ indicate to the learner what is the positive class\r\npa.setLabel(positiveLabel);\r\n\/\/ set an aggressiveness parameter for the PA\r\npa.setC(0.01f);\r\n\/\/ Learn the model\r\npa.learn(tr);\r\n\r\n\/**\r\n * TEST THE MODEL\r\n *\/\r\nBinaryClassificationEvaluator ev = new BinaryClassificationEvaluator(positiveLabel);\r\nClassifier f = pa.getPredictionFunction();\r\nList<Prediction> testPredictions = ExperimentUtils.test(f, ev, testSet);\r\n<\/pre>\n
SimpleDataset completeDataset = new SimpleDataset();\r\ncompleteDataset.addExamples(tr);\r\ncompleteDataset.addExamples(testSet);\r\nList<BinaryClassificationEvaluator> nfoldEv = ExperimentUtils.nFoldCrossValidation(nfold, pa, completeDataset, ev)\r\nnfold=5;\r\n\r\n\/**\r\n * RETRIVING AVERAGE ACCURACY AND STANDARD DEVIATION ON THE 5 FOLDS\r\n *\/\r\nfloat[] values = new float[nfoldEv.size()];\r\nfor (int i = 0; i < ret.length; i++) {\r\nvalues[i] = nfoldEv.get(i).getPerfomanceMeasure(\"Accuracy\");\r\n}\r\nfloat mean = it.uniroma2.sag.kelp.utils.Math.getMean(values);\r\ndouble standardDeviation = it.uniroma2.sag.kelp.utils.Math.getStandardDeviation(values);\r\n<\/pre>\n