Examples of org.apache.lucene.index.IndexWriter

org.apache.lucene.index.IndexWriter
An IndexWriter creates and maintains an index.
The create argument to the {@link #IndexWriter(Directory,Analyzer,boolean,MaxFieldLength) constructor} determines whether a new index is created, or whether an existing index is opened. Note that you can open an index with create=true even while readers are using the index. The old readers will continue to search the "point in time" snapshot they had opened, and won't see the newly created index until they re-open. There are also {@link #IndexWriter(Directory,Analyzer,MaxFieldLength) constructors}with no create argument which will create a new index if there is not already an index at the provided path and otherwise open the existing index.

In either case, documents are added with {@link #addDocument(Document) addDocument} and removed with {@link #deleteDocuments(Term)} or {@link #deleteDocuments(Query)}. A document can be updated with {@link #updateDocument(Term,Document) updateDocument} (which just deletesand then adds the entire document). When finished adding, deleting and updating documents, {@link #close() close} should be called.

These changes are buffered in memory and periodically flushed to the {@link Directory} (during the above methodcalls). A flush is triggered when there are enough buffered deletes (see {@link #setMaxBufferedDeleteTerms}) or enough added documents since the last flush, whichever is sooner. For the added documents, flushing is triggered either by RAM usage of the documents (see {@link #setRAMBufferSizeMB}) or the number of added documents. The default is to flush when RAM usage hits 16 MB. For best indexing speed you should flush by RAM usage with a large RAM buffer. Note that flushing just moves the internal buffered state in IndexWriter into the index, but these changes are not visible to IndexReader until either {@link #commit()} or {@link #close} is called. A flush mayalso trigger one or more segment merges which by default run with a background thread so as not to block the addDocument calls (see below for changing the {@link MergeScheduler}).

If an index will not have more documents added for a while and optimal search performance is desired, then either the full {@link #optimize() optimize}method or partial {@link #optimize(int)} method should becalled before the index is closed.

Opening an IndexWriter creates a lock file for the directory in use. Trying to open another IndexWriter on the same directory will lead to a {@link LockObtainFailedException}. The {@link LockObtainFailedException}is also thrown if an IndexReader on the same directory is used to delete documents from the index.

Expert: IndexWriter allows an optional {@link IndexDeletionPolicy} implementation to bespecified. You can use this to control when prior commits are deleted from the index. The default policy is {@link KeepOnlyLastCommitDeletionPolicy} which removes all priorcommits as soon as a new commit is done (this matches behavior before 2.2). Creating your own policy can allow you to explicitly keep previous "point in time" commits alive in the index for some time, to allow readers to refresh to the new commit without having the old commit deleted out from under them. This is necessary on filesystems like NFS that do not support "delete on last close" semantics, which Lucene's "point in time" search normally relies on.

Expert: IndexWriter allows you to separately change the {@link MergePolicy} and the {@link MergeScheduler}. The {@link MergePolicy} is invoked whenever there arechanges to the segments in the index. Its role is to select which merges to do, if any, and return a {@link MergePolicy.MergeSpecification} describing the merges. Italso selects merges to do for optimize(). (The default is {@link LogByteSizeMergePolicy}. Then, the {@link MergeScheduler} is invoked with the requested merges andit decides when and how to run the merges. The default is {@link ConcurrentMergeScheduler}.

NOTE: if you hit an OutOfMemoryError then IndexWriter will quietly record this fact and block all future segment commits. This is a defensive measure in case any internal state (buffered documents and deletions) were corrupted. Any subsequent calls to {@link #commit()} will throw anIllegalStateException. The only course of action is to call {@link #close()}, which internally will call {@link #rollback()}, to undo any changes to the index since the last commit. You can also just call {@link #rollback()}directly.

NOTE: {@link IndexWriter} instances are completely threadsafe, meaning multiple threads can call any of its methods, concurrently. If your application requires external synchronization, you should not synchronize on the IndexWriter instance as this may cause deadlock; use your own (non-Lucene) objects instead.

NOTE: If you call Thread.interrupt() on a thread that's within IndexWriter, IndexWriter will try to catch this (eg, if it's in a wait() or Thread.sleep()), and will then throw the unchecked exception {@link ThreadInterruptedException}and clear the interrupt status on the thread.

  }


  private static RAMDirectory makeEmptyIndex(final int numDeletedDocs) 
    throws IOException {
      RAMDirectory d = new RAMDirectory();
      IndexWriter w = new IndexWriter(d, new WhitespaceAnalyzer(), true,
                                      MaxFieldLength.LIMITED);
      for (int i = 0; i < numDeletedDocs; i++) {
        w.addDocument(new Document());
      }
      w.commit();
      w.deleteDocuments( new MatchAllDocsQuery() );
      w.commit();


      if (0 < numDeletedDocs)
        Assert.assertTrue("writer has no deletions", w.hasDeletions());


      Assert.assertEquals("writer is missing some deleted docs", 
                          numDeletedDocs, w.maxDoc());
      Assert.assertEquals("writer has non-deleted docs", 
                          0, w.numDocs());
      w.close();
      IndexReader r = IndexReader.open(d, true);
      Assert.assertEquals("reader has wrong number of deleted docs", 
                          numDeletedDocs, r.numDeletedDocs());
      r.close();
      return d;

View Full Code Here


    // plan to add a set of useful stopwords, consider changing some of the
    // interior filters.
    StandardAnalyzer analyzer = new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_CURRENT, Collections.emptySet());
    // TODO: something about lock timeouts and leftover locks.
    IndexWriter writer = new IndexWriter(storeDirectory, analyzer, true, IndexWriter.MaxFieldLength.LIMITED);
    IndexSearcher searcher = new IndexSearcher(storeDirectory, true);
    
    for(int dx = 0; dx < 1000; dx ++) {
      String f = randomField();
      Document doc = new Document();
      doc.add(new Field("data", f, Field.Store.YES, Field.Index.ANALYZED));  
      writer.addDocument(doc);
    }
    
    searcher.close();
    writer.close();
                rmDir(new File(storePathname));
  }

View Full Code Here

import org.apache.lucene.util.OpenBitSetDISI;


public class TestCachingWrapperFilter extends LuceneTestCase {
  public void testCachingWorks() throws Exception {
    Directory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new KeywordAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
    writer.close();


    IndexReader reader = IndexReader.open(dir, true);


    MockFilter filter = new MockFilter();
    CachingWrapperFilter cacher = new CachingWrapperFilter(filter);

View Full Code Here

    reader.close();
  }
  
  public void testNullDocIdSet() throws Exception {
    Directory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new KeywordAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
    writer.close();


    IndexReader reader = IndexReader.open(dir, true);


    final Filter filter = new Filter() {
      @Override

View Full Code Here

    reader.close();
  }
  
  public void testNullDocIdSetIterator() throws Exception {
    Directory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new KeywordAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
    writer.close();


    IndexReader reader = IndexReader.open(dir, true);


    final Filter filter = new Filter() {
      @Override

View Full Code Here

    }
  }
  
  public void testIsCacheAble() throws Exception {
    Directory dir = new RAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new KeywordAnalyzer(), true, IndexWriter.MaxFieldLength.LIMITED);
    writer.close();


    IndexReader reader = IndexReader.open(dir, true);


    // not cacheable:
    assertDocIdSetCacheable(reader, new QueryWrapperFilter(new TermQuery(new Term("test","value"))), false);

View Full Code Here

    reader.close();
  }


  public void testEnforceDeletions() throws Exception {
    Directory dir = new MockRAMDirectory();
    IndexWriter writer = new IndexWriter(dir, new WhitespaceAnalyzer(), IndexWriter.MaxFieldLength.UNLIMITED);
    IndexReader reader = writer.getReader();
    IndexSearcher searcher = new IndexSearcher(reader);


    // add a doc, refresh the reader, and check that its there
    Document doc = new Document();
    doc.add(new Field("id", "1", Field.Store.YES, Field.Index.NOT_ANALYZED));
    writer.addDocument(doc);


    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);


    TopDocs docs = searcher.search(new MatchAllDocsQuery(), 1);
    assertEquals("Should find a hit...", 1, docs.totalHits);


    final Filter startFilter = new QueryWrapperFilter(new TermQuery(new Term("id", "1")));


    // ignore deletions
    CachingWrapperFilter filter = new CachingWrapperFilter(startFilter, CachingWrapperFilter.DeletesMode.IGNORE);
        
    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);
    assertEquals("[query + filter] Should find a hit...", 1, docs.totalHits);
    ConstantScoreQuery constantScore = new ConstantScoreQuery(filter);
    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should find a hit...", 1, docs.totalHits);


    // now delete the doc, refresh the reader, and see that it's not there
    writer.deleteDocuments(new Term("id", "1"));


    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);


    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);
    assertEquals("[query + filter] Should *not* find a hit...", 0, docs.totalHits);


    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should find a hit...", 1, docs.totalHits);




    // force cache to regenerate:
    filter = new CachingWrapperFilter(startFilter, CachingWrapperFilter.DeletesMode.RECACHE);


    writer.addDocument(doc);
    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);
        
    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);
    assertEquals("[query + filter] Should find a hit...", 1, docs.totalHits);


    constantScore = new ConstantScoreQuery(filter);
    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should find a hit...", 1, docs.totalHits);


    // make sure we get a cache hit when we reopen reader
    // that had no change to deletions
    IndexReader newReader = refreshReader(reader);
    assertTrue(reader != newReader);
    reader = newReader;
    searcher = new IndexSearcher(reader);
    int missCount = filter.missCount;
    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should find a hit...", 1, docs.totalHits);
    assertEquals(missCount, filter.missCount);


    // now delete the doc, refresh the reader, and see that it's not there
    writer.deleteDocuments(new Term("id", "1"));


    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);


    missCount = filter.missCount;
    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);
    assertEquals(missCount+1, filter.missCount);
    assertEquals("[query + filter] Should *not* find a hit...", 0, docs.totalHits);
    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should *not* find a hit...", 0, docs.totalHits);




    // apply deletions dynamically
    filter = new CachingWrapperFilter(startFilter, CachingWrapperFilter.DeletesMode.DYNAMIC);


    writer.addDocument(doc);
    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);
        
    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);
    assertEquals("[query + filter] Should find a hit...", 1, docs.totalHits);
    constantScore = new ConstantScoreQuery(filter);
    docs = searcher.search(constantScore, 1);
    assertEquals("[just filter] Should find a hit...", 1, docs.totalHits);


    // now delete the doc, refresh the reader, and see that it's not there
    writer.deleteDocuments(new Term("id", "1"));


    reader = refreshReader(reader);
    searcher = new IndexSearcher(reader);


    docs = searcher.search(new MatchAllDocsQuery(), filter, 1);

View Full Code Here


  public void testLoadIndexReader() throws Exception {
    RAMDirectory dir = new RAMDirectory();


    // create dir data
    IndexWriter indexWriter = new IndexWriter(dir, new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.UNLIMITED);
    for (int i = 0; i < 20; i++) {
      Document document = new Document();
      assembleDocument(document, i);
      indexWriter.addDocument(document);
    }
    indexWriter.close();


    // test load ii from index reader
    IndexReader ir = IndexReader.open(dir, false);
    InstantiatedIndex ii = new InstantiatedIndex(ir);
    ir.close();

View Full Code Here


    RAMDirectory dir = new RAMDirectory();
    InstantiatedIndex ii = new InstantiatedIndex();


    // create dir data
    IndexWriter indexWriter = new IndexWriter(dir, new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.UNLIMITED);
    for (int i = 0; i < 500; i++) {
      Document document = new Document();
      assembleDocument(document, i);
      indexWriter.addDocument(document);
    }
    indexWriter.close();


    // test ii writer
    InstantiatedIndexWriter instantiatedIndexWriter = ii.indexWriterFactory(new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_CURRENT), true);
    for (int i = 0; i < 500; i++) {
      Document document = new Document();

View Full Code Here

        dir.setLockFactory(lf);


        // Lock prefix should have been set:
        assertTrue("lock prefix was not set by the RAMDirectory", lf.lockPrefixSet);


        IndexWriter writer = new IndexWriter(dir, new WhitespaceAnalyzer(), true,
                                             IndexWriter.MaxFieldLength.LIMITED);


        // add 100 documents (so that commit lock is used)
        for (int i = 0; i < 100; i++) {
            addDoc(writer);
        }


        // Both write lock and commit lock should have been created:
        assertEquals("# of unique locks created (after instantiating IndexWriter)",
                     1, lf.locksCreated.size());
        assertTrue("# calls to makeLock is 0 (after instantiating IndexWriter)",
                   lf.makeLockCount >= 1);
        
        for(Iterator e = lf.locksCreated.keySet().iterator(); e.hasNext();) {
            String lockName = (String) e.next();
            MockLockFactory.MockLock lock = (MockLockFactory.MockLock) lf.locksCreated.get(lockName);
            assertTrue("# calls to Lock.obtain is 0 (after instantiating IndexWriter)",
                       lock.lockAttempts > 0);
        }
        
        writer.close();
    }

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.lucene.index.IndexWriter

com.alimama.mdrill.solr.hbaserealtime.realtime.RealTimeDirectoryData

com.alimama.mdrill.solr.realtime.realtime.RealTimeDirectoryData

com.browseengine.bobo.facets.attribute.AttributesFacetHandlerTest

com.browseengine.bobo.test.BoboTestCase

com.gentics.cr.lucene.indexaccessor.DefaultIndexAccessor

fi.foyt.hibernate.gae.search.GaeInstantWorkspace

index.IndexBuilder

it.eng.spagobi.commons.utilities.indexing.LuceneIndexer

org.apache.derby.optional.lucene.LuceneSupport

org.apache.jackrabbit.core.query.lucene.AbstractIndex

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.