Examples of org.terrier.utility.LookAheadStream

org.terrier.utility.LookAheadStream

Implements an InputStream, that encapsulates another stream, but only upto the point that a pre-defined end marker in the stream is identified. The Reader will then become endOfFile, and refuse to return any more bytes from the stream. Suppose that we create an instance of a LookAheadStream with the end marker END. For the following input: a b c d END e f g... the LookAheadStream, will stop after reading the string END. Note that the end marker will be missing from the parent stream.

LookAheadStream allows the encoding to be changed between markers. Handy for collections of webpages, which may use different encodings. However, the end marker must be obtainable using the default encoding. @author Craig Macdonald, Vassilis Plachouras @see org.terrier.utility.LookAheadReader

   * @return Document the object of the current document to process.
   */
  public Document getDocument() {
    try{
      InputStream is = tags_CaseSensitive
            ? new LookAheadStream(br, desiredEncoding == null ? end_docTag.getBytes() : end_docTag.getBytes(desiredEncoding))
            : new LookAheadStreamCaseInsensitive(br, end_docTag);
        return documentClass.getConstructor(InputStream.class, Map.class, Tokeniser.class).newInstance(is, DocProperties, tokeniser);
      } catch (Exception e) {
        throw new RuntimeException(e);
      }

Examples of org.terrier.utility.LookAheadStream

Related Classes of org.terrier.utility.LookAheadStream