Examples of ivory.core.data.stat.DocLengthTable4B

ivory.core.data.stat.DocLengthTable4B

Object that keeps track of the length of each document in the collection as a four-byte integers (ints). Document lengths are measured in number of terms.

Document length data is stored in a serialized data file, in the following format:

An integer that specifies the docno offset d, where d + 1 is the first docno in the collection.
An integer that specifies the number of documents in the collection n.
Exactly n ints, one for each document in the collection.

Since the documents are numbered sequentially starting at d + 1, each short corresponds unambiguously to a particular document.

@author Jimmy Lin

    LOG.info("CollectionName: " + collectionName);
    LOG.info("Input path: " + inputPath);


    DocLengthTable mDLTable;
    try {
      mDLTable = new DocLengthTable4B(env.getDoclengthsData(), fs);
    } catch (IOException e1) {
      throw new RuntimeException("Error initializing Doclengths file");
    }
    LOG.info(mDLTable.getAvgDocLength()+" is average source-language document length.");
    LOG.info(targetEnv.readCollectionDocumentCount()+" is number of target-language docs. We use the target-side DF table so we set #docs to this value in our scoring model.");

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.

Examples of ivory.core.data.stat.DocLengthTable4B

Related Classes of ivory.core.data.stat.DocLengthTable4B