Examples of weka.classifiers.Evaluation

weka.classifiers.Evaluation
Class for evaluating machine learning models.
-------------------------------------------------------------------
General options when evaluating a learning scheme from the command-line:
-t filename
Name of the file with the training data. (required)
-T filename
Name of the file with the test data. If missing a cross-validation is performed.
-c index
Index of the class attribute (1, 2, ...; default: last).
-x number
The number of folds for the cross-validation (default: 10).
-no-cv
No cross validation. If no test file is provided, no evaluation is done.
-split-percentage percentage
Sets the percentage for the train/test set split, e.g., 66.
-preserve-order
Preserves the order in the percentage split instead of randomizing the data first with the seed value ('-s').
-s seed
Random number seed for the cross-validation and percentage split (default: 1).
-m filename
The name of a file containing a cost matrix.
-l filename
Loads classifier from the given file. In case the filename ends with ".xml", a PMML file is loaded or, if that fails, options are loaded from XML.
-d filename
Saves classifier built from the training data into the given file. In case the filename ends with ".xml" the options are saved XML, not the model.
-v
Outputs no statistics for the training data.
-o
Outputs statistics only, not the classifier.
-i
Outputs information-retrieval statistics per class.
-k
Outputs information-theoretic statistics.
-classifications "weka.classifiers.evaluation.output.prediction.AbstractOutput + options"
Uses the specified class for generating the classification output. E.g.: weka.classifiers.evaluation.output.prediction.PlainText or : weka.classifiers.evaluation.output.prediction.CSV -p range
Outputs predictions for test instances (or the train instances if no test instances provided and -no-cv is used), along with the attributes in the specified range (and nothing else). Use '-p 0' if no attributes are desired.
Deprecated: use "-classifications ..." instead.
-distribution
Outputs the distribution instead of only the prediction in conjunction with the '-p' option (only nominal classes).
Deprecated: use "-classifications ..." instead.
-r
Outputs cumulative margin distribution (and nothing else).
-g
Only for classifiers that implement "Graphable." Outputs the graph representation of the classifier (and nothing else).
-xml filename | xml-string
Retrieves the options from the XML-data instead of the command line.
-threshold-file file
The file to save the threshold data to. The format is determined by the extensions, e.g., '.arff' for ARFF format or '.csv' for CSV.
-threshold-label label
The class label to determine the threshold data for (default is the first label)
-------------------------------------------------------------------
Example usage as the main of a classifier (called FunkyClassifier):
```
 public static void main(String [] args) { runClassifier(new FunkyClassifier(), args); } 
```
------------------------------------------------------------------
Example usage from within an application:
```
 Instances trainInstances = ... instances got from somewhere Instances testInstances = ... instances got from somewhere Classifier scheme = ... scheme got from somewhere Evaluation evaluation = new Evaluation(trainInstances); evaluation.evaluateModel(scheme, testInstances); System.out.println(evaluation.toSummaryString()); 
```
@author Eibe Frank (eibe@cs.waikato.ac.nz) @author Len Trigg (trigg@cs.waikato.ac.nz) @version $Revision: 7228 $

    double[][] probs = initialProbs(numInstances);
    m_NumGenerated = 0;
    double sumOfWeights = train.sumOfWeights();
    for (int j = 0; j < getNumIterations(); j++) {
      performIteration(trainYs, trainFs, probs, trainN, sumOfWeights);
      Evaluation eval = new Evaluation(train);
      eval.evaluateModel(this, test);
      results[j] += eval.correct();
    }
  }
      }
      
      // Find the number of iterations with the lowest error

View Full Code Here

  protected void buildClassifierUsingResampling(Instances data) 
    throws Exception {


    Instances trainData, sample, training;
    double epsilon, reweight, sumProbs;
    Evaluation evaluation;
    int numInstances = data.numInstances();
    Random randomInstance = new Random(m_Seed);
    int resamplingIterations = 0;


    // Initialize data
    m_Betas = new double [m_Classifiers.length];
    m_NumIterationsPerformed = 0;
    // Create a copy of the data so that when the weights are diddled
    // with it doesn't mess up the weights for anyone else
    training = new Instances(data, 0, numInstances);
    sumProbs = training.sumOfWeights();
    for (int i = 0; i < training.numInstances(); i++) {
      training.instance(i).setWeight(training.instance(i).
              weight() / sumProbs);
    }
    
    // Do boostrap iterations
    for (m_NumIterationsPerformed = 0; m_NumIterationsPerformed < m_Classifiers.length; 
   m_NumIterationsPerformed++) {
      if (m_Debug) {
  System.err.println("Training classifier " + (m_NumIterationsPerformed + 1));
      }


      // Select instances to train the classifier on
      if (m_WeightThreshold < 100) {
  trainData = selectWeightQuantile(training, 
           (double)m_WeightThreshold / 100);
      } else {
  trainData = new Instances(training);
      }
      
      // Resample
      resamplingIterations = 0;
      double[] weights = new double[trainData.numInstances()];
      for (int i = 0; i < weights.length; i++) {
  weights[i] = trainData.instance(i).weight();
      }
      do {
  sample = trainData.resampleWithWeights(randomInstance, weights);


  // Build and evaluate classifier
  m_Classifiers[m_NumIterationsPerformed].buildClassifier(sample);
  evaluation = new Evaluation(data);
  evaluation.evaluateModel(m_Classifiers[m_NumIterationsPerformed], 
         training);
  epsilon = evaluation.errorRate();
  resamplingIterations++;
      } while (Utils.eq(epsilon, 0) && 
        (resamplingIterations < MAX_NUM_RESAMPLING_ITERATIONS));
        
      // Stop if error too big or 0

View Full Code Here

  protected void buildClassifierWithWeights(Instances data) 
    throws Exception {


    Instances trainData, training;
    double epsilon, reweight;
    Evaluation evaluation;
    int numInstances = data.numInstances();
    Random randomInstance = new Random(m_Seed);


    // Initialize data
    m_Betas = new double [m_Classifiers.length];
    m_NumIterationsPerformed = 0;


    // Create a copy of the data so that when the weights are diddled
    // with it doesn't mess up the weights for anyone else
    training = new Instances(data, 0, numInstances);
    
    // Do boostrap iterations
    for (m_NumIterationsPerformed = 0; m_NumIterationsPerformed < m_Classifiers.length; 
   m_NumIterationsPerformed++) {
      if (m_Debug) {
  System.err.println("Training classifier " + (m_NumIterationsPerformed + 1));
      }
      // Select instances to train the classifier on
      if (m_WeightThreshold < 100) {
  trainData = selectWeightQuantile(training, 
           (double)m_WeightThreshold / 100);
      } else {
  trainData = new Instances(training, 0, numInstances);
      }


      // Build the classifier
      if (m_Classifiers[m_NumIterationsPerformed] instanceof Randomizable)
  ((Randomizable) m_Classifiers[m_NumIterationsPerformed]).setSeed(randomInstance.nextInt());
      m_Classifiers[m_NumIterationsPerformed].buildClassifier(trainData);


      // Evaluate the classifier
      evaluation = new Evaluation(data);
      evaluation.evaluateModel(m_Classifiers[m_NumIterationsPerformed], training);
      epsilon = evaluation.errorRate();


      // Stop if error too small or error too big and ignore this model
      if (Utils.grOrEq(epsilon, 0.5) || Utils.eq(epsilon, 0)) {
  if (m_NumIterationsPerformed == 0) {
    m_NumIterationsPerformed = 1; // If we're the first we have to to use it

View Full Code Here

   * @throws Exception if subset can't be evaluated
   */
  protected double estimatePerformance(BitSet feature_set, int num_atts)
  throws Exception {


    m_evaluation = new Evaluation(m_theInstances);
    int i;
    int [] fs = new int [num_atts];


    double [] instA = new double [num_atts];
    int classI = m_theInstances.classIndex();

View Full Code Here

    int bestIndex = -1;
    double bestPerformance = Double.NaN;
    int numClassifiers = m_Classifiers.length;
    for (int i = 0; i < numClassifiers; i++) {
      Classifier currentClassifier = getClassifier(i);
      Evaluation evaluation;
      if (m_NumXValFolds > 1) {
  evaluation = new Evaluation(newData);
  for (int j = 0; j < m_NumXValFolds; j++) {


          // We want to randomize the data the same way for every 
          // learning scheme.
    train = newData.trainCV(m_NumXValFolds, j, new Random (1));
    test = newData.testCV(m_NumXValFolds, j);
    currentClassifier.buildClassifier(train);
    evaluation.setPriors(train);
    evaluation.evaluateModel(currentClassifier, test);
  }
      } else {
  currentClassifier.buildClassifier(train);
  evaluation = new Evaluation(train);
  evaluation.evaluateModel(currentClassifier, test);
      }


      double error = evaluation.errorRate();
      if (m_Debug) {
  System.err.println("Error rate: " + Utils.doubleToString(error, 6, 4)
         + " for classifier "
         + currentClassifier.getClass().getName());
      }

View Full Code Here

    if (classifier instanceof weka.classifiers.misc.InputMappedClassifier) {
      Instances mappedClassifierHeader = 
        ((weka.classifiers.misc.InputMappedClassifier)classifier).
        getModelHeader(new Instances(inst, 0));


      eval = new Evaluation(new Instances(mappedClassifierHeader, 0));


      if (!eval.getHeader().equalHeaders(inst)) {
        // When the InputMappedClassifier is loading a model, 
        // we need to make a new dataset that maps the test instances to
        // the structure expected by the mapped classifier - this is only

View Full Code Here

        }
        if (ce.getTrainSet().getDataSet() == null ||
            ce.getTrainSet().getDataSet().numInstances() == 0) {
          // we have no training set to estimate majority class
          // or mean of target from
          Evaluation eval = new Evaluation(ce.getTestSet().getDataSet());
          m_PlotInstances = ExplorerDefaults.getClassifierErrorsPlotInstances();
          m_PlotInstances.setInstances(ce.getTestSet().getDataSet());
          m_PlotInstances.setClassifier(ce.getClassifier());
          m_PlotInstances.setClassIndex(ce.getTestSet().getDataSet().classIndex());
          m_PlotInstances.setEvaluation(eval);


          eval = adjustForInputMappedClassifier(eval, ce.getClassifier(),
              ce.getTestSet().getDataSet(), m_PlotInstances);
          eval.useNoPriors();
          m_eval = new AggregateableEvaluation(eval);
        } else {
          // we can set up with the training set here
          Evaluation eval = new Evaluation(ce.getTrainSet().getDataSet());
          m_PlotInstances = ExplorerDefaults.getClassifierErrorsPlotInstances();
          m_PlotInstances.setInstances(ce.getTrainSet().getDataSet());
          m_PlotInstances.setClassifier(ce.getClassifier());
          m_PlotInstances.setClassIndex(ce.getTestSet().getDataSet().classIndex());
          m_PlotInstances.setEvaluation(eval);

View Full Code Here

      }
      try {


        ClassifierErrorsPlotInstances plotInstances =
          ExplorerDefaults.getClassifierErrorsPlotInstances();
        Evaluation eval = null;


        if (m_trainData == null || m_trainData.numInstances() == 0) {
          eval = new Evaluation(m_testData);
          plotInstances.setInstances(m_testData);
          plotInstances.setClassifier(m_classifier);
          plotInstances.setClassIndex(m_testData.classIndex());
          plotInstances.setEvaluation(eval);
          eval = adjustForInputMappedClassifier(eval, m_classifier,
              m_testData, plotInstances);
          
          eval.useNoPriors();
        } else {
          eval = new Evaluation(m_trainData);
          plotInstances.setInstances(m_trainData);
          plotInstances.setClassifier(m_classifier);
          plotInstances.setClassIndex(m_trainData.classIndex());
          plotInstances.setEvaluation(eval);
          eval = adjustForInputMappedClassifier(eval, m_classifier,

View Full Code Here

   * This method must be called if pruning is not to be performed.
   *
   * @throws Exception if an error occurs
   */
  public void installLinearModels() throws Exception {
    Evaluation nodeModelEval;
    if (m_isLeaf) {
      buildLinearModel(m_indices);
    } else {
      if (m_left != null) {
  m_left.installLinearModels();
      }


      if (m_right != null) {
  m_right.installLinearModels();
      }
      buildLinearModel(m_indices);
    }
    nodeModelEval = new Evaluation(m_instances);
    nodeModelEval.evaluateModel(m_nodeModel, m_instances);
    m_rootMeanSquaredError = nodeModelEval.rootMeanSquaredError();
    // save space
    if (!m_saveInstances) {
      m_instances = new Instances(m_instances, 0);
    }
  }

View Full Code Here

   * Recursively prune the tree
   *
   * @throws Exception if an error occurs
   */
  public void prune() throws Exception {
    Evaluation nodeModelEval = null;


    if (m_isLeaf) {
      buildLinearModel(m_indices);
      nodeModelEval = new Evaluation(m_instances);


      // count the constant term as a paramter for a leaf
      // Evaluate the model
      nodeModelEval.evaluateModel(m_nodeModel, m_instances);


      m_rootMeanSquaredError = nodeModelEval.rootMeanSquaredError();
    } else {


      // Prune the left and right subtrees
      if (m_left != null) {
  m_left.prune();
      } 


      if (m_right != null) {
  m_right.prune();  
      } 
      
      buildLinearModel(m_indices);
      nodeModelEval = new Evaluation(m_instances);


      double rmsModel;
      double adjustedErrorModel;


      nodeModelEval.evaluateModel(m_nodeModel, m_instances);


      rmsModel = nodeModelEval.rootMeanSquaredError();
      adjustedErrorModel = rmsModel 
  * pruningFactor(m_numInstances, 
      m_nodeModel.numParameters() + 1);


      // Evaluate this node (ie its left and right subtrees)
      Evaluation nodeEval = new Evaluation(m_instances);
      double     rmsSubTree;
      double     adjustedErrorNode;
      int   l_params = 0, r_params = 0;


      nodeEval.evaluateModel(this, m_instances);


      rmsSubTree = nodeEval.rootMeanSquaredError();


      if (m_left != null) {
  l_params = m_left.numParameters();
      }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of weka.classifiers.Evaluation

bnGUI.att_BayesAttSel

bnGUI.att_ev_SingleBayesROCAttributeEval

bnGUI.att_Ranker$EvalThread

bnGUI.gui_WekaExperiment

bnGUI.Thread_competitors

bnGUI.Thread_weka_run

com.deafgoat.ml.prognosticator.AppClassifier

CrossValWeka

de.tudarmstadt.ukp.similarity.experiments.coling2012.util.Evaluator

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.