Examples of org.apache.lucene.store.FSDirectory

org.apache.lucene.store.FSDirectory
sun.com/bugdatabase/view_bug.do?bug_id=6265734">Sun JRE bug this is a poor choice for Windows, but on all other platforms this is the preferred choice. Applications using {@link Thread#interrupt()} or{@link Future#cancel(boolean)} should use{@link SimpleFSDirectory} instead. See {@link NIOFSDirectory} java docfor details.
{@link MMapDirectory} uses memory-mapped IO whenreading. This is a good choice if you have plenty of virtual memory relative to your index size, eg if you are running on a 64 bit JRE, or you are running on a 32 bit JRE but your index sizes are small enough to fit into the virtual memory space. Java has currently the limitation of not being able to unmap files from user code. The files are unmapped, when GC releases the byte buffers. Due to this bug in Sun's JRE, MMapDirectory's {@link IndexInput#close}is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of. This class supplies a (possibly dangerous) workaround mentioned in the bug report, which may fail on non-Sun JVMs. Applications using {@link Thread#interrupt()} or{@link Future#cancel(boolean)} should use{@link SimpleFSDirectory} instead. See {@link MMapDirectory}java doc for details.

Unfortunately, because of system peculiarities, there is no single overall best implementation. Therefore, we've added the {@link #open} method, to allow Lucene to choosethe best FSDirectory implementation given your environment, and the known limitations of each implementation. For users who have no reason to prefer a specific implementation, it's best to simply use {@link #open}. For all others, you should instantiate the desired implementation directly.

The locking implementation is by default {@link NativeFSLockFactory}, but can be changed by passing in a custom {@link LockFactory} instance. @see Directory

    {
        File indexDir = File.createTempFile( remoteIndexFile, ".dir" );
        indexDir.delete();
        indexDir.mkdirs();


        FSDirectory directory = FSDirectory.open( indexDir );


        BufferedInputStream is = null;


        try
        {
            is = new BufferedInputStream( fetcher.retrieve( remoteIndexFile ) );


            Date timestamp = null;


            if ( remoteIndexFile.endsWith( ".gz" ) )
            {
                timestamp = unpackIndexData( is, directory, //
                    updateRequest.getIndexingContext() );
            }
            else
            {
                // legacy transfer format
                timestamp = unpackIndexArchive( is, directory, //
                    updateRequest.getIndexingContext() );
            }


            if ( updateRequest.getDocumentFilter() != null )
            {
                filterDirectory( directory, updateRequest.getDocumentFilter() );
            }


            if ( merge )
            {
                updateRequest.getIndexingContext().merge( directory );
            }
            else
            {
                updateRequest.getIndexingContext().replace( directory );
            }
            if ( sideEffects != null && sideEffects.size() > 0 )
            {
                getLogger().info( IndexUpdateSideEffect.class.getName() + " extensions found: " + sideEffects.size() );
                for ( IndexUpdateSideEffect sideeffect : sideEffects )
                {
                    sideeffect.updateIndex( directory, updateRequest.getIndexingContext(), merge );
                }
            }


            return timestamp;
        }
        finally
        {
            IOUtil.close( is );


            if ( directory != null )
            {
                directory.close();
            }


            try
            {
                FileUtils.deleteDirectory( indexDir );

View Full Code Here


        File indexDir = new File( indexArchive.getAbsoluteFile().getParentFile(), indexArchive.getName() + ".dir" );


        indexDir.mkdirs();


        FSDirectory fdir = FSDirectory.open( indexDir );


        try
        {
            unpackDirectory( fdir, is );
            copyUpdatedDocuments( fdir, directory, context );

View Full Code Here


        File indexDir = new File( indexArchive.getAbsoluteFile().getParentFile(), indexArchive.getName() + ".dir" );


        indexDir.mkdirs();


        FSDirectory fdir = FSDirectory.open( indexDir );


        try
        {
            // force the timestamp update
            IndexUtils.updateTimestamp( context.getIndexDirectory(), context.getTimestamp() );

View Full Code Here

   @Test
   public void profileTestFSDirectory() throws InterruptedException, IOException {
      File indexDir = new File(new File("."), indexName);
      boolean directoriesCreated = indexDir.mkdirs();
      assert directoriesCreated : "couldn't create directory for FSDirectory test";
      FSDirectory dir = FSDirectory.open(indexDir);
      testDirectory(dir, "FSDirectory");
   }

View Full Code Here

   @Test
   public void profileTestFSDirectory() throws InterruptedException, IOException {
      File indexDir = new File(new File("."), indexName);
      boolean directoriesCreated = indexDir.mkdirs();
      assert directoriesCreated : "couldn't create directory for FSDirectory test";
      FSDirectory dir = FSDirectory.open(indexDir);
      stressTestDirectory(dir, "FSDirectory");
   }

View Full Code Here

      File subDir = new File(rootDir, indexName);
      boolean directoriesCreated = subDir.mkdir();
      assert directoriesCreated : "couldn't create directory for test";


      //We need at least one Directory to exist on filesystem to trigger the problem
      FSDirectory luceneDirectory = FSDirectory.open(subDir);
      luceneDirectory.close();
      ConfigurationBuilder builder = new ConfigurationBuilder();
      builder.persistence()
            .addStore(LuceneStoreConfigurationBuilder.class)
               .autoChunkSize(110)
               .location(rootDir.getAbsolutePath());

View Full Code Here

    * @param invert           flag which identifies which terms should be inserted which not.
    * @throws IOException
    */
   public static void createIndex(File rootDir, String indexName, int termsToAdd, boolean invert) throws IOException {
      File indexDir = new File(rootDir, indexName);
      FSDirectory directory = FSDirectory.open(indexDir);
      try {
         CacheTestSupport.initializeDirectory(directory);
         IndexWriter iwriter = LuceneSettings.openWriter(directory, 100000);
         try {
            for (int i = 0; i <= termsToAdd; i++) {
               Document doc = new Document();
               String term = String.valueOf(i);
               //For even values of i we add to "main" field
               if (i % 2 == 0 ^ invert) {
                  doc.add(new Field("main", term, Field.Store.NO, Field.Index.NOT_ANALYZED));
               }
               else {
                  doc.add(new Field("secondaryField", term, Field.Store.YES, Field.Index.NOT_ANALYZED));
               }
               iwriter.addDocument(doc);
            }
            iwriter.commit();
         }
         finally {
            iwriter.close();
         }
      }
      finally {
         directory.close();
      }
   }

View Full Code Here

      if (adapter == null) {
         synchronized (openDirectories) {
            adapter = openDirectories.get(indexName);
            if (adapter == null) {
               final File path = new File(this.rootDirectory, indexName);
               final FSDirectory directory = openLuceneDirectory(path);
               final InternalDirectoryContract wrapped = ContractAdaptorFactory.wrapNativeDirectory(directory);
               adapter = new DirectoryLoaderAdaptor(wrapped, indexName, autoChunkSize);
               openDirectories.put(indexName, adapter);
            }
         }

View Full Code Here

   * @param indexDir The directory where to write a new index
   * @return the created FSDirectory
   * @throws IOException
   */
  public static FSDirectory createFSIndex(File indexDir) throws IOException {
    FSDirectory fsDirectory = FSDirectory.getDirectory( indexDir );
    if ( ! IndexReader.indexExists( fsDirectory ) ) {
      log.debug( "Initialize index: '{}'", indexDir.getAbsolutePath() );
      IndexWriter iw = new IndexWriter( fsDirectory, new StandardAnalyzer(), true );
      iw.close();
    }

View Full Code Here

   * @return the created FSDirectory
   * @throws IOException
   */
  public static FSDirectory createFSIndex(File indexDir, Properties dirConfiguration) throws IOException {
    LockFactory lockFactory = createLockFactory(indexDir, dirConfiguration);
    FSDirectory fsDirectory = FSDirectory.open( indexDir, null );
    // must use the setter (instead of using the constructor) to set the lockFactory, or Lucene will
    // throw an exception if it's different than a previous setting.
    fsDirectory.setLockFactory( lockFactory );
    if ( ! IndexReader.indexExists( fsDirectory ) ) {
      log.debug( "Initialize index: '{}'", indexDir.getAbsolutePath() );
      IndexWriter.MaxFieldLength fieldLength = new IndexWriter.MaxFieldLength( IndexWriter.DEFAULT_MAX_FIELD_LENGTH );
      IndexWriter iw = new IndexWriter( fsDirectory, new SimpleAnalyzer(), true, fieldLength );
      iw.close();

View Full Code Here

0 1 2 3 4 5 6 7 8 9

TOP

Related Classes of org.apache.lucene.store.FSDirectory

com.alimama.mdrill.buffer.CacheKeyBuffer

com.bericotech.clavin.index.IndexDirectoryBuilder

com.browseengine.bobo.api.BoboIndexReader

de.arago.lucene.api.Index

larq.cmdline.ModLARQindex

org.apache.blur.store.blockcache.BlockDirectoryTest

org.apache.cocoon.components.search.LuceneCocoonHelper

org.apache.ctakes.core.resource.LuceneIndexReaderResourceImpl

org.apache.lucene.ant.IndexTask

org.apache.lucene.benchmark.quality.trec.QueryDriver

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.