Examples of ivory.core.data.stat.DocLengthTable4B

ivory.core.data.stat.DocLengthTable4B
Object that keeps track of the length of each document in the collection as a four-byte integers (ints). Document lengths are measured in number of terms.

Document length data is stored in a serialized data file, in the following format:
- An integer that specifies the docno offset d, where d + 1 is the first docno in the collection.
- An integer that specifies the number of documents in the collection n.
- Exactly n ints, one for each document in the collection.
Since the documents are numbered sequentially starting at d + 1, each short corresponds unambiguously to a particular document.
@author Jimmy Lin

      
      try {
        if(shortDocLengths)
          mDLTable = new DocLengthTable2B(localFiles[2], FileSystem.getLocal(conf));
        else 
          mDLTable = new DocLengthTable4B(localFiles[2], FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from "+localFiles[2]);
      }  
      try {
        mScoreFn = (ScoringModel) Class.forName(conf.get("Ivory.ScoringModel")).newInstance();

View Full Code Here


    conf.set("Ivory.IndexPath", indexPath);
    conf.set("Ivory.ScoringModel", scoringModel);
    DocLengthTable mDLTable;
    try {
      mDLTable = new DocLengthTable4B(env.getDoclengthsData(), fs);
    } catch (IOException e1) {
      throw new RuntimeException("Error initializing Doclengths file");
    }
    LOG.info(mDLTable.getAvgDocLength()+" is average doc len.");
    LOG.info(mDLTable.getDocCount()+" is num docs.");

View Full Code Here


      try {
        if(shortDocLengths)
          mDLTable = new DocLengthTable2B(localFiles[2], FileSystem.getLocal(conf));
        else 
          mDLTable = new DocLengthTable4B(localFiles[2], FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from "+localFiles[4]);
      }  
      try {
        mScoreFn = (ScoringModel) Class.forName(conf.get("Ivory.ScoringModel")).newInstance();

View Full Code Here

    LOG.info("Target-language stemmed stopwords: " + eStemmedStopwords);
    LOG.info("Target-language tokenizer model: " + eTokenizerModel);


    DocLengthTable mDLTable;
    try {
      mDLTable = new DocLengthTable4B(env.getDoclengthsData(), fs);
    } catch (IOException e1) {
      throw new RuntimeException("Error initializing Doclengths file");
    }
    LOG.info(mDLTable.getAvgDocLength()+" is average source-language document length.");
    LOG.info(targetEnv.readCollectionDocumentCount()+" is number of target-language docs. We use the target-side DF table so we set #docs to this value in our scoring model.");

View Full Code Here

    LOG.info("Target-language stemmed stopwords: " + eStemmedStopwords);
    LOG.info("Target-language tokenizer model: " + eTokenizerModel);


    DocLengthTable mDLTable;
    try {
      mDLTable = new DocLengthTable4B(env.getDoclengthsData(), fs);
    } catch (IOException e1) {
      throw new RuntimeException("Error initializing Doclengths file");
    }
    LOG.info(mDLTable.getAvgDocLength()+" is average source-language document length.");
    LOG.info(targetEnv.readCollectionDocumentCount()+" is number of target-language docs. We use the target-side DF table so we set #docs to this value in our scoring model.");

View Full Code Here


      try {
        if (shortDocLengths)
          mDLTable = new DocLengthTable2B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
        else
          mDLTable = new DocLengthTable4B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from " + localFiles[2]);
      }
      try {
        mScoreFn = (ScoringModel) Class.forName(conf.get("Ivory.ScoringModel")).newInstance();

View Full Code Here


      try {
        if(shortDocLengths)
          mDLTable = new DocLengthTable2B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
        else 
          mDLTable = new DocLengthTable4B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from "+localFiles[4]);
      }


      try {

View Full Code Here

    LOG.info("CollectionName: " + collectionName);
    LOG.info("Input path: " + inputPath);


    DocLengthTable mDLTable;
    try {
      mDLTable = new DocLengthTable4B(env.getDoclengthsData(), fs);
    } catch (IOException e1) {
      throw new RuntimeException("Error initializing Doclengths file");
    }
    LOG.info(mDLTable.getAvgDocLength()+" is average source-language document length.");
    LOG.info(targetEnv.readCollectionDocumentCount()+" is number of target-language docs. We use the target-side DF table so we set #docs to this value in our scoring model.");

View Full Code Here


      try {
        if (shortDocLengths)
          mDLTable = new DocLengthTable2B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
        else
          mDLTable = new DocLengthTable4B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from " + localFiles[2]);
      }
      try {
        mScoreFn = (ScoringModel) Class.forName(conf.get("Ivory.ScoringModel")).newInstance();

View Full Code Here


      try {
        if(shortDocLengths)
          mDLTable = new DocLengthTable2B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
        else 
          mDLTable = new DocLengthTable4B(pathMapping.get(dlFile), FileSystem.getLocal(conf));
      } catch (IOException e1) {
        throw new RuntimeException("Error loading dl table from "+localFiles[4]);
      }


      try {

View Full Code Here

0 1

TOP

Related Classes of ivory.core.data.stat.DocLengthTable4B

ivory.core.preprocess.BuildTranslatedTermDocVectors

ivory.core.preprocess.BuildWeightedIntDocVectors$MyMapper

ivory.core.preprocess.BuildWeightedTermDocVectors$MyMapper

org.apache.hadoop.conf.Configuration

org.apache.hadoop.fs.FSDataInputStream

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.