Examples of it.unimi.dsi.mg4j.index.TermProcessor

it.unimi.dsi.mg4j.index.TermProcessor
A term processor, implementing term/prefix transformation and possibly term/prefix filtering.
Index contruction requires sometimes modifications of the given terms: downcasing, stemming, and so on. The same transformation must be applied to terms in a query. This interface provides a uniform way to perform arbitrary term transformations.
Index construction requires also term filtering: {@link #processTerm(MutableString)} mayreturn false, indicating that the term should not be processed at all (e.g., because it is a stopword).
Additionally, the method {@link #processPrefix(MutableString)} mayprocess analogously a prefix (used for prefix queries).
Implementation are encouraged to expose a singleton, when possible, by means of the static factory method getInstance(). Warning: implementations of this class are not required to be thread-safe, but they provide {@link it.unimi.dsi.lang.FlyweightPrototype flyweight copies}. The {@link #copy()} method is strengthened so to return a instance of this class.
This interface was originally suggested by Fabien Campagne.

    
    // This will remain set if *all* indices to be merged agree
    boolean haveCounts = true, havePositions = true;
    /* This will be set if *all* indices to be merged agree. Moreover, if some
     * indices disagree we will emit a warning. */
    TermProcessor termProcessor = null;
    /* This will be set if *all* indices to be merged agree. Moreover, if some
     * indices disagree we will emit a warning. */
    Payload payload = null;
    String field = null;
    writeGlobCounts = writeSizes = true;
    boolean someGlobCounts = false, someSizes = false, allDataForSizeComputation = true;
    
    for( int i = 0; i < numIndices; i++ ) {
      index[ i ] = (BitStreamIndex)Index.getInstance( inputBasename[ i ], false, requireSizes, false );
      if ( i == 0 ) {
        termProcessor = index[ 0 ].termProcessor.copy();
        payload = index[ 0 ].payload == null ? null : index[ 0 ].payload.copy();
      }
      else {
        if ( ! termProcessor.equals( index[ i ].termProcessor ) ) throw new IllegalStateException( "The term processor of the first index (" + termProcessor + ") is different from the term processor of index " + i + " (" + index[ i ].termProcessor + ")" );
        if ( ( payload == null ) != ( index[ i ].payload == null ) || payload != null && ! payload.compatibleWith( index[ i ].payload ) ) throw new IllegalStateException( "The payload specification of index " + index[ 0 ] + " is not compatible with that of index " + index[ i ] );
      }


      if ( index[ i ].field != null ) {
        if ( field == null ) {

View Full Code Here


    final int[] indexedField = Scan.parseFieldNames( jsapResult.getStringArray( "indexedField" ), factory, jsapResult.getBoolean( "allFields" ) );
    final VirtualDocumentResolver[] virtualDocumentResolver = Scan.parseVirtualDocumentResolver( jsapResult.getStringArray( "virtualDocumentResolver" ), indexedField, factory );
    final int[] virtualDocumentGap = Scan.parseVirtualDocumentGap( jsapResult.getStringArray( "virtualDocumentGap" ), indexedField, factory );


    final TermProcessor termProcessor = jsapResult.getBoolean( "downcase" ) ? DowncaseTermProcessor.getInstance() :
      ObjectParser.fromSpec( jsapResult.getString( "termProcessor" ), TermProcessor.class, MG4JClassParser.PACKAGE, new String[] { "getInstance" } ); 


    final boolean skips = ! jsapResult.getBoolean( "noSkips" );
    final boolean interleaved = jsapResult.getBoolean( "interleaved" );
    if ( ! skips && ( jsapResult.userSpecified( "quantum" ) || jsapResult.userSpecified( "height" ) ) ) throw new IllegalArgumentException( "You specified quantum or height, but you also disabled skips." );

View Full Code Here

import junit.framework.TestCase;


public class TestDowncaseTermProcessor extends TestCase {


  public void testReadResolve() throws IOException, ClassNotFoundException {
    TermProcessor t = DowncaseTermProcessor.getInstance();
    FastByteArrayOutputStream os = new FastByteArrayOutputStream();
    BinIO.storeObject( t, os );
    assertTrue( t == (TermProcessor)BinIO.loadObject( new FastByteArrayInputStream( os.array ) ) );
  }

View Full Code Here

TOP

Related Classes of it.unimi.dsi.mg4j.index.TermProcessor

it.unimi.dsi.mg4j.tool.Combine

it.unimi.dsi.mg4j.tool.IndexBuilder

it.unimi.dsi.mg4j.tool.TestDowncaseTermProcessor

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.