Examples of org.terrier.indexing.Indexer

org.terrier.indexing.Indexer

Properties:

termpipelines - the sequence of TermPipeline stages (e.g. Stopwords removal and PorterStemmer).
termpipelines.skip - a list of tokens which should not be skipped from the term pipeline. If not set or empty, then none will be skipped.
indexing.max.tokens - The maximum number of tokens the indexer will attempt to index in a document. If 0, then all tokens will be indexed (default).
ignore.empty.documents - Assign empty documents documnent Ids. Default true
indexing.max.docs.per.builder - Maximum number of documents in an index before a new index is created, and merged later.
indexing.builder.boundary.docnos - Docnos of documents that force the index being created to be completed, and a new index to be commenced. An alternative to indexing.max.docs.per.builder

@author Craig Macdonald

          logger.error("Aborting indexing process");
          return;
        }
  
  
      Indexer indexer;
      final boolean useSinglePass = Boolean.parseBoolean(ApplicationSetup.getProperty("desktop.indexing.singlepass", "false"));
      indexer = ApplicationSetup.BLOCK_INDEXING
        ? useSinglePass 
          ? new BlockSinglePassIndexer(ApplicationSetup.TERRIER_INDEX_PATH, ApplicationSetup.TERRIER_INDEX_PREFIX)  
          : new BlockIndexer(ApplicationSetup.TERRIER_INDEX_PATH, ApplicationSetup.TERRIER_INDEX_PREFIX)
        : useSinglePass
          ? new BasicSinglePassIndexer(ApplicationSetup.TERRIER_INDEX_PATH, ApplicationSetup.TERRIER_INDEX_PREFIX)
          : new BasicIndexer(ApplicationSetup.TERRIER_INDEX_PATH, ApplicationSetup.TERRIER_INDEX_PREFIX);
  
      SimpleFileCollection sfc = new SimpleFileCollection(folderList, true);
  
      indexer.index(new Collection[] { sfc });
  
      
      //load in the indexes
      if (loadIndices()) {
        //indices loaded

Examples of org.terrier.indexing.Indexer

Related Classes of org.terrier.indexing.Indexer