Examples of org.apache.lucene.search.similarities.Similarity

Package org.apache.lucene.search.similarities

Examples of org.apache.lucene.search.similarities.Similarity

org.apache.lucene.search.similarities.Similarity
Similarity defines the components of Lucene scoring.
Expert: Scoring API.
This is a low-level API, you should only extend this API if you want to implement an information retrieval model. If you are instead looking for a convenient way to alter Lucene's scoring, consider extending a higher-level implementation such as {@link TFIDFSimilarity}, which implements the vector space model with this API, or just tweaking the default implementation: {@link DefaultSimilarity}.
Similarity determines how Lucene weights terms, and Lucene interacts with this class at both index-time and query-time.
At indexing time, the indexer calls {@link #computeNorm(FieldInvertState)}, allowing the Similarity implementation to set a per-document value for the field that will be later accessible via {@link AtomicReader#getNormValues(String)}. Lucene makes no assumption about what is in this norm, but it is most useful for encoding length normalization information.
Implementations should carefully consider how the normalization is encoded: while Lucene's classical {@link TFIDFSimilarity} encodes a combination of index-time boostand length normalization information with {@link SmallFloat} into a single byte, this might not be suitable for all purposes.
Many formulas require the use of average document length, which can be computed via a combination of {@link CollectionStatistics#sumTotalTermFreq()} and {@link CollectionStatistics#maxDoc()} or {@link CollectionStatistics#docCount()}, depending upon whether the average should reflect field sparsity.
Additional scoring factors can be stored in named NumericDocValuesFields and accessed at query-time with {@link AtomicReader#getNumericDocValues(String)}.
Finally, using index-time boosts (either via folding into the normalization byte or via DocValues), is an inefficient way to boost the scores of different fields if the boost will be the same for every document, instead the Similarity can simply take a constant boost parameter C, and {@link PerFieldSimilarityWrapper} can return different instances with different boosts depending upon field name.
At query-time, Queries interact with the Similarity via these steps:
When {@link IndexSearcher#explain(org.apache.lucene.search.Query,int)} is called, queries consult the Similarity's DocScorer for an explanation of how it computed its score. The query passes in a the document id and an explanation of how the frequency was computed. @see org.apache.lucene.index.IndexWriterConfig#setSimilarity(Similarity) @see IndexSearcher#setSimilarity(Similarity) @lucene.experimental


    IndexReader reader = DirectoryReader.open(FSDirectory.open(indexLocation));
    IndexSearcher searcher = new IndexSearcher(reader);


    if (similarity.equalsIgnoreCase("BM25")) {
      Similarity simBM25 = new BM25Similarity();
      searcher.setSimilarity(simBM25);
    } else if (similarity.equalsIgnoreCase("LM")) {
      NamedList<Double> paramNamedList = new NamedList<Double>();
      paramNamedList.add("mu", 2500.0);
      SolrParams params = SolrParams.toSolrParams(paramNamedList);
      LMDirichletSimilarityFactory factory = new LMDirichletSimilarityFactory();
      factory.init(params);
      Similarity simLMDir = factory.getSimilarity();
      searcher.setSimilarity(simLMDir);
    }


    QueryParser p = new QueryParser(Version.LUCENE_43, IndexStatuses.StatusField.TEXT.name, IndexStatuses.ANALYZER);
    Query query = p.parse(queryText);

View Full Code Here

    NamedList<Double> paramNamedList = new NamedList<Double>();
    paramNamedList.add("mu", 2500.0);
    SolrParams params = SolrParams.toSolrParams(paramNamedList);
    LMDirichletSimilarityFactory factory = new LMDirichletSimilarityFactory();
    factory.init(params);
    Similarity simLMDir = factory.getSimilarity();
    searcher.setSimilarity(simLMDir);
  }

View Full Code Here

    NamedList<Double> paramNamedList = new NamedList<Double>();
    paramNamedList.add("mu", 2500.0);
    SolrParams params = SolrParams.toSolrParams(paramNamedList);
    LMDirichletSimilarityFactory factory = new LMDirichletSimilarityFactory();
    factory.init(params);
    Similarity simLMDir = factory.getSimilarity();
    searcher.setSimilarity(simLMDir);
  }

View Full Code Here

  public void runQueries() throws IOException {
    IndexReader reader = DirectoryReader.open(FSDirectory.open(new File(pathToIndex)));
    searcher = new IndexSearcher(reader);


    if (similarity.equalsIgnoreCase("bm25")) {
      Similarity simBM25 = new BM25Similarity();
      searcher.setSimilarity(simBM25);
    } else if (similarity.equalsIgnoreCase("lm")) {
      NamedList<Double> paramNamedList = new NamedList<Double>();
      paramNamedList.add("mu", 2500.0);
      SolrParams params = SolrParams.toSolrParams(paramNamedList);
      LMDirichletSimilarityFactory factory = new LMDirichletSimilarityFactory();
      factory.init(params);
      Similarity simLMDir = factory.getSimilarity();
      searcher.setSimilarity(simLMDir);
    }


    Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_41);

View Full Code Here


    IndexReader reader = DirectoryReader.open(FSDirectory.open(indexLocation));
    IndexSearcher searcher = new IndexSearcher(reader);


    if (similarity.equalsIgnoreCase("BM25")) {
      Similarity simBM25 = new BM25Similarity();
      searcher.setSimilarity(simBM25);
    } else if (similarity.equalsIgnoreCase("LM")) {
      NamedList<Double> paramNamedList = new NamedList<Double>();
      paramNamedList.add("mu", 2500.0);
      SolrParams params = SolrParams.toSolrParams(paramNamedList);
      LMDirichletSimilarityFactory factory = new LMDirichletSimilarityFactory();
      factory.init(params);
      Similarity simLMDir = factory.getSimilarity();
      searcher.setSimilarity(simLMDir);
    }


    out.println("Using similarity: " + searcher.getSimilarity().toString());

View Full Code Here

      SearchConfiguration cfg,
      WorkerBuildContext context
  ) {
    String indexName = getIndexName( entity, cfg );
    Properties[] indexProperties = getIndexProperties( cfg, indexName );
    Similarity similarity = createSimilarity( indexName, cfg, indexProperties[0], entity );
    boolean isDynamicSharding = isShardingDynamic( indexProperties[0] );


    IndexManager[] indexManagers = new IndexManager[0];
    if ( !isDynamicSharding ) {
      indexManagers = createIndexManagers(

View Full Code Here

    return interceptor;
  }


  private Similarity createSimilarity(String directoryProviderName, SearchConfiguration cfg, Properties indexProperties, XClass clazz) {
    // first check on class level
    Similarity classLevelSimilarity = null;


    // TODO - the processing of the @Similarity annotation is temporary here. The annotation should be removed in Search 5 (HF)
    List<XClass> hierarchyClasses = ReflectionHelper.createXClassHierarchy( clazz );
    Class<?> similarityClass = null;
    for ( XClass hierarchyClass : hierarchyClasses ) {
      org.hibernate.search.annotations.Similarity similarityAnnotation = hierarchyClass.getAnnotation( org.hibernate.search.annotations.Similarity.class );
      if ( similarityAnnotation != null ) {
        Class<?> tmpSimilarityClass = similarityAnnotation.impl();
        if ( similarityClass != null && !similarityClass.equals( tmpSimilarityClass ) ) {
          throw log.getMultipleInconsistentSimilaritiesInClassHierarchyException( clazz.getName() );
        }
        else {
          similarityClass = tmpSimilarityClass;
        }
        classLevelSimilarity = ClassLoaderHelper.instanceFromClass(
            Similarity.class,
            similarityClass,
            "Similarity class for index " + directoryProviderName
        );
      }
    }


    // now we check the config
    Similarity configLevelSimilarity = null;
    String similarityClassName = indexProperties.getProperty( Environment.SIMILARITY_CLASS_PER_INDEX );
    if ( similarityClassName != null ) {
      configLevelSimilarity = ClassLoaderHelper.instanceFromName(
          Similarity.class,
          similarityClassName,
          DirectoryProviderFactory.class.getClassLoader(),
          "Similarity class for index " + directoryProviderName
      );
    }


    if ( classLevelSimilarity != null && configLevelSimilarity != null ) {
      throw log.getInconsistentSimilaritySettingBetweenAnnotationsAndConfigPropertiesException(
          classLevelSimilarity.getClass().getName(),
          configLevelSimilarity.getClass().getCanonicalName()
      );
    }
    else if ( classLevelSimilarity != null ) {
      return classLevelSimilarity;
    }

View Full Code Here

  private LazyQueryState buildSearcher(SearchFactoryImplementor searchFactoryImplementor, Boolean forceScoring) {
    Map<Class<?>, EntityIndexBinding> builders = searchFactoryImplementor.getIndexBindings();
    List<IndexManager> targetedIndexes = new ArrayList<IndexManager>();
    Set<String> idFieldNames = new HashSet<String>();


    Similarity searcherSimilarity = null;
    //TODO check if caching this work for the last n list of indexedTargetedEntities makes a perf boost
    if ( indexedTargetedEntities.size() == 0 ) {
      // empty indexedTargetedEntities array means search over all indexed entities,
      // but we have to make sure there is at least one
      if ( builders.isEmpty() ) {

View Full Code Here

    this.indexName = indexManager.getIndexName();
    LuceneIndexingParameters luceneParameters = indexManager.getIndexingParameters();
    this.indexParameters = luceneParameters.getIndexParameters();
    this.directoryProvider = indexManager.getDirectoryProvider();
    luceneParameters.applyToWriter( writerConfig );
    Similarity similarity = indexManager.getSimilarity();
    if ( similarity != null ) {
      writerConfig.setSimilarity( similarity );
    }
    writerConfig.setOpenMode( OpenMode.APPEND ); //More efficient to open
  }

View Full Code Here

      SearchConfiguration cfg,
      WorkerBuildContext buildContext
  ) {
    String indexName = getIndexName( entity, cfg );
    Properties[] indexProperties = getIndexProperties( cfg, indexName );
    Similarity similarity = createSimilarity( indexName, cfg, indexProperties[0], entity, buildContext );
    boolean isDynamicSharding = isShardingDynamic( indexProperties[0], buildContext );


    IndexManager[] indexManagers = new IndexManager[0];
    if ( !isDynamicSharding ) {
      indexManagers = createIndexManagers(

View Full Code Here

0 1 2 3 4 5 6 7 8

TOP

Related Classes of org.apache.lucene.search.similarities.Similarity

cc.twittertools.search.api.TrecSearchHandler

cc.twittertools.search.indexing.SearchStatuses

cc.twittertools.search.local.RunQueries

cc.twittertools.search.local.SearchStatuses

cc.twittertools.search.retrieval.QueryEnvironment

cc.twittertools.search.retrieval.TrecSearchHandler

cc.wikitools.lucene.WikipediaSearcher

org.apache.blur.manager.clusterstatus.ZookeeperClusterStatus

org.apache.blur.manager.indexserver.DistributedIndexServer

org.apache.lucene.index.memory.MemoryIndex$MemoryIndexReader

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.