Examples of com.datasalt.pangool.solr.TupleSolrOutputFormat

com.datasalt.pangool.solr.TupleSolrOutputFormat
Instantiable OutputFormat that can be used in Pangool for indexing {@link ITuple} in SOLR. It behaves similar toSOLR-1301's SolrOutputFormat with the difference that configuration is passed via instance (constructor params). This allows us to easily have multiple {@link TupleSolrOutputFormat} in the same Pangool Job. Also, it is much easier toconfigure: it just needs to be instantiated (no need to call multiple static methods to configure it). Everything will be configured underneath.
Things that can be configured via constructor:
- solrHome: Local folder where schema.xml and such can be found.
- converter: An implementation of {@link TupleDocumentConverter} other than {@link DefaultTupleDocumentConverter}if user wants to do complex things like boosting.
- outputZipFile: If user wants to produce a ZIP with the index.
- batchSize: Number of documents that will go in each indexing batch.
- threadCount: Number of threads in a pool that will be used for indexing.
- queueSize: Maximum number of batches that can be pooled in the batch indexing thread pool.
For a usage example see test class {@link TupleSolrOutputFormatExample}.

      for(FileStatus fileStatus : fileSystem.listStatus(new Path(input + "/" + categoryString))) {
        job.addInput(fileStatus.getPath(), new HadoopInputFormat(TextInputFormat.class),
            new CategoryMapper(category, fileStatus.getPath().getName()));
      }
      // Add a named output for each category
      job.addNamedOutput(categoryString, new TupleSolrOutputFormat(new File(
          "src/test/resources/shakespeare-solr"), conf), ITuple.class, NullWritable.class);
    }
    job.setOutput(new Path(output), new HadoopOutputFormat(NullOutputFormat.class), ITuple.class,
        NullWritable.class);
    // The reducer will just emit the tuple to the corresponding Category output

View Full Code Here

      for(FileStatus fileStatus: fileSystem.listStatus(new Path(input + "/" + categoryString))) {
        job.addInput(fileStatus.getPath(), new HadoopInputFormat(TextInputFormat.class),
            new CategoryMapper(category, fileStatus.getPath().getName()));
      }
      // Add a named output for each category
      job.addNamedOutput(categoryString, new TupleSolrOutputFormat(new File("src/test/resources/shakespeare-solr"),
          conf), ITuple.class, NullWritable.class);
    }
    job.setOutput(new Path(output), new HadoopOutputFormat(NullOutputFormat.class), ITuple.class, NullWritable.class);
    // The reducer will just emit the tuple to the corresponding Category output
    job.setTupleReducer(new TupleReducer<ITuple, NullWritable>() {

View Full Code Here

      for(FileStatus fileStatus : fileSystem.listStatus(new Path(input + "/" + categoryString))) {
        job.addInput(fileStatus.getPath(), new HadoopInputFormat(TextInputFormat.class),
            new CategoryMapper(category, fileStatus.getPath().getName()));
      }
      // Add a named output for each category
      job.addNamedOutput(categoryString, new TupleSolrOutputFormat(new File(
          "src/test/resources/shakespeare-solr"), job.getConf()), ITuple.class, NullWritable.class);
    }
    job.setOutput(new Path(output), new HadoopOutputFormat(NullOutputFormat.class), ITuple.class,
        NullWritable.class);
    // The reducer will just emit the tuple to the corresponding Category output

View Full Code Here

TOP

Related Classes of com.datasalt.pangool.solr.TupleSolrOutputFormat

com.datasalt.pangool.examples.solr.MultiShakespeareIndexer

org.apache.hadoop.fs.FileSystem

org.apache.hadoop.fs.Path

java.io.InputStream

java.util.zip.ZipEntry

java.net.URI

java.io.FileInputStream

java.io.FileOutputStream

java.util.zip.ZipOutputStream

java.io.File

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.