Examples of org.terrier.utility.TagSet

org.terrier.utility.TagSet
A class that models a set of tags to process (white list), a set of tags to skip (black list), a tag that is used as a document delimiter, and a tag the contents of which are used as a unique identifier. The text within any tag encountered within the scope of a tag from the white list, is processed by default, unless it is explicitly black listed.
For example, in order to index all the text within the DOC tag of a document from a typical TREC collection, without indexing the contents of the DOCHDR tag, we could define in the properties file the following properties:
TrecDocTags.doctag=DOC
TrecDocTags.idtag=DOCNO
TrecDocTags.process=
TrecDocTags.skip=DOCHDR
TrecDocTags.casesensitive=false

In the source code, we would create an instance of the class as follows:
TagSet TrecIndexToProcess = new TagSet("TrecDocTags");
All the tags are converted to uppercase, in order to check whether they belong to the specified set of tags. @author Vassilis Plachouras, Craig Macdonald

        logger.error("The topics file " + queryfilename + " does not exist, or it cannot be read.");
        return false;
      } else {
        br = Files.openFileReader(queryfilename,desiredEncoding);
        TRECFullTokenizer queryTokenizer = new TRECFullTokenizer(
              new TagSet(TagSet.TREC_QUERY_TAGS),
              new TagSet(TagSet.EMPTY_TAGS),
              br);
        queryTokenizer.setIgnoreMissingClosingTags(true);
        while (!queryTokenizer.isEndOfFile()) {
          String docnoToken = null;
          StringBuilder query = new StringBuilder();

View Full Code Here

   * opening and closing document and document number
   * tags.
   */
  protected void setTags(String TagSet)
  {
    TagSet tagSet = new TagSet(TagSet);
    tags_CaseSensitive = tagSet.isCaseSensitive();
    docnotag = tagSet.getDocTag();
    String tmpDocTag = "<" + tagSet.getDocTag() + ">";
    String tmpEndDocTag = "</" + tagSet.getDocTag() + ">";
    String tmpDocnoTag = "<" + tagSet.getIdTag() + ">";
    String tmpEndDocnoTag = "</" + tagSet.getIdTag() + ">";
    start_docTag = tmpDocTag.toCharArray();
    start_docTagLength = start_docTag.length;
    start_docnoTag = tmpDocnoTag.toCharArray();
    start_docnoTagLength = start_docnoTag.length;
    end_docTag = tmpEndDocTag;

View Full Code Here

    } catch (UnsupportedEncodingException uee) {
      //logger.warn("Desired encoding ("+charset+") unsupported. Resorting to platform default.", uee);
      this.br = new BufferedReader(new InputStreamReader(docStream));
    }
    this.properties = docProperties;  
    this._tags = new TagSet(TagSet.TREC_DOC_TAGS);
    this._exact = new TagSet(TagSet.TREC_EXACT_DOC_TAGS);
    this._fields = new TagSet(TagSet.FIELD_TAGS);
    this.tokeniser = _tokeniser;
    this.currentTokenStream = Tokeniser.EMPTY_STREAM;
    for(int i=0;i<abstractCount;i++)
    {
      abstracts[i] = new StringBuilder(abstractlengths[i]);

View Full Code Here

   */
  public TaggedDocument(Reader docReader, Map<String, String> docProperties, Tokeniser _tokeniser)
  {
    this.br = docReader;
    properties = docProperties;  
    this._tags = new TagSet(TagSet.TREC_DOC_TAGS);
    this._exact = new TagSet(TagSet.TREC_EXACT_DOC_TAGS);
    this._fields = new TagSet(TagSet.FIELD_TAGS);
    this.tokeniser = _tokeniser;
    this.currentTokenStream = Tokeniser.EMPTY_STREAM;
  }

View Full Code Here

   */
  public TRECFullTokenizer() {
    inTagToProcess = false;
    inTagToSkip = false;
    inDocnoTag = false;
    tagSet = new TagSet(TagSet.TREC_DOC_TAGS);
    exactTagSet = new TagSet(TagSet.TREC_EXACT_DOC_TAGS);
    EOD = false;
    EOF = false;
  }

View Full Code Here

  public TRECFullTokenizer(BufferedReader _br) {
    inTagToProcess = false;
    inTagToSkip = false;
    inDocnoTag = false;
    this.br = _br;
    tagSet = new TagSet(TagSet.TREC_DOC_TAGS);
    exactTagSet = new TagSet(TagSet.TREC_EXACT_DOC_TAGS);
    EOD = false;
    EOF = false;
  }

View Full Code Here

TOP

Related Classes of org.terrier.utility.TagSet

org.terrier.indexing.TaggedDocument

org.terrier.indexing.TRECCollection

org.terrier.indexing.TRECFullTokenizer

org.terrier.structures.TRECQuery

All source code are property of their respective owners. Java is a trademark of Sun Microsystems, Inc and owned by ORACLE Inc. Contact coftware#gmail.com.